copying 100MiB file errors with "error reading <file>: Input/output error"

Mountpoint for Amazon S3 version

mount-s3 1.10.0

AWS Region

us-east-1

Describe the running environment

Running in EC2 RedHat 8 Linux Connected to the EC2 instance via Session Manager

sh-4.4$ whoami ssm-user

Copying the file with

$ aws s3 cp s3://<bucket>/big_file.zip .

failed with:

An error occurred (SignatureDoesNotMatch) when calling the GetObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I then issued

$ aws configure set default.s3.multipart_threshold 1000MB

and repeated the aws s3 cp command, which now succeeded.

I am mounting the bucket with

$ mount <bucket_name> ./mnt

I can copy small files with no issues.

When trying to copy a 100MiB file instead the log shows:

mountpoint_s3::fuse: read failed: get request failed: get object request failed: Client error: Forbidden: The request signature we calculated does not match the signature you provided. Check your key and signing method

My hypothesis

Setting the aws cli multipart threshold to a high value means that the whole file was transferred in one go instead of using multi-part. That solved the issue for the aws cli cp.

Mount-s3 instead uses multipart read with apparently no possibility to disable this feature.

note I uploaded the file to S3 from a different machine, using different credentials than those on the EC2 instance where mount-s3 is running.

Mountpoint options

mount-s3 <bucket> mnt

What happened?

$ cp mnt/big_file.zip .

fails with:

cp: error reading 'mnt/big_file.zip': Input/output error

Relevant log output

mount-s3[16724]: [ERROR] mountpoint_s3::prefetch::part_stream: GetObject body part failed key=big_file.zip error=ClientError(Forbidden("The request signature we calculated does not match the signature you provided. Check your key and signing method.", ClientErrorMetadata { http_code: Some(403), error_code: Some("SignatureDoesNotMatch"), error_message: Some("The request signature we calculated does not match the signature you provided. Check your key and signing method.") }))
mount-s3[16724]: [WARN] read{req=24 ino=2 fh=2 offset=0 size=4096 name=big_file.zip}: mountpoint_s3::fuse: read failed: get request failed: get object request failed: Client error: Forbidden: The request signature we calculated does not match the signature you provided. Check your key and signing method.

awslabs / mountpoint-s3