Open bfleming-ciena opened 7 years ago
These don't seem like questions about the spark-ec2 tool itself. I think this question and #102 probably belong better on the Spark Users mailing list or on Stack Overflow.
Issues here should be restricted to issues with the spark-ec2 tool itself, not with Spark in general or with generic Spark interactions with AWS.
Add this to spark-defaults.conf on the master.
spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4 spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4
I am told that if I can set fs.s3a.endpoint=s3.ca-central-1.amazonaws.com I can support the S3 V4 signature in the new regions so saveAsText file won't fail.
However, could someone please chime in as to where I set this value in spark-ec2 and what I need to do for it to take effect?
Would be a big help! Thanks.