When we were reading the byte files from "gs://" url, it worked fine locally but gave stackoverflow error while doing the same on large dataset. This is why I changed the reading path of the file to http path.
Please note that you can still use the local file path like:
'''spark-submit main.py
'''
But for byte files, we will have to use the http path.
When we were reading the byte files from "gs://" url, it worked fine locally but gave stackoverflow error while doing the same on large dataset. This is why I changed the reading path of the file to http path.
Please note that you can still use the local file path like: '''spark-submit main.py ''' But for byte files, we will have to use the http path.