search

amplab / spark-ec2

Scripts used to setup a Spark cluster on EC2
Apache License 2.0
392 stars 299 forks source link

Quick question on setting fs.s3a.endpoint #103

Open bfleming-ciena opened 7 years ago

bfleming-ciena commented 7 years ago

I am told that if I can set fs.s3a.endpoint=s3.ca-central-1.amazonaws.com I can support the S3 V4 signature in the new regions so saveAsText file won't fail.

However, could someone please chime in as to where I set this value in spark-ec2 and what I need to do for it to take effect?

Would be a big help! Thanks.

nchammas commented 7 years ago

These don't seem like questions about the spark-ec2 tool itself. I think this question and #102 probably belong better on the Spark Users mailing list or on Stack Overflow.

Issues here should be restricted to issues with the spark-ec2 tool itself, not with Spark in general or with generic Spark interactions with AWS.

bfleming-ciena commented 7 years ago

Add this to spark-defaults.conf on the master.

Support Canada Region which uses the V4 S3 signature. Otherwise you will see "bad request" messages from saveAsTextFile

spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4 spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4