In class S3FS, the endpoint_url is always set when creating s3 client. The default value is s3.amazonaws.com. However, when caller uses s3 access point as notebooks storage (instead of S3 bucket), the bucket is set to a S3 access point ARN. In this scenario, an endpoint_url is not allowed to set in S3 Client - it will result in validation failure in botocore.
The solution is not passing endpoint_url, or pass None, when bucket is in format of S3 access point ARN.
danielfrg
commented
3 years ago
In class S3FS, the endpoint_url is always set when creating s3 client. The default value is
s3.amazonaws.com. However, when caller uses s3 access point as notebooks storage (instead of S3 bucket), the bucket is set to a S3 access point ARN. In this scenario, an endpoint_url is not allowed to set in S3 Client - it will result in validation failure in botocore.The solution is not passing endpoint_url, or pass None, when bucket is in format of S3 access point ARN.