Closed hlmiao closed 2 years ago
Please try to add the following option:
-Dspark.hadoop.fs.s3a.endpoint=s3.cn-north-1.amazonaws.com.cn
If you use Ningxia region, use cn-northwest-1 in the endpoint.
Yes, as you can see in the README, you need to set spark.hadoop.fs.s3a.endpoint
.
Based on xmubeta's comment, I added the examples to README. Feel free to reopen this issue if the issue still occurs.
Do we need to include Dspark.hadoop.fs.s3a.endpoint for all regions?
You only need to use this option for China region (Beijing & Ningxia).
I am also getting this issue. https://github.com/aws-samples/aws-glue-samples/issues/140
So far it is only for China Region.
Regards HanLin
At 2023-02-23 07:06:05, "purnima1612" @.***> wrote:
Do we need to include Dspark.hadoop.fs.s3a.endpoint for all regions?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
Hi, I am sure the AK/SK is available to access S3 buckets at a prefix level, but the error trace always followed "The AWS Access Key Id you provided does not exist in our records" , maybe less aws-cn in some places?
docker run -it -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID -Dspark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY" -p 18080:18080 glue/sparkui "/opt/spark/bin/spark-classorg.apache.spark.deploy.history.HistoryServer"