mc trying to read file before uploading

[scherepanov]

NOTE

If this case is urgent, please subscribe to Subnet so that our 24/7 support team may help you faster.

I am implementing write-only bucket, when clients can write file but cannot read or list dir etc. It works on concept of blackhole - you put file there and losing any access to it.

mc works fine, but backend reports permission denied.

Looking at mc -- debug, I cannot see it is getting metadata of file by issuing HEAD command.

This is part of --debug output of mc cp myfile blackhole/sink/dir/newfile:

mc: HEAD /sink/onemoredir/tryfile1.txt HTTP/1.1 Host: s3feast.mycompany.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.66 mc/RELEASE.2023-12-23T08-47-21Z Authorization: AWS4-HMAC-SHA256 Credential=*REMOVED X-Amz-Content-Sha256: REMOVED X-Amz-Date: 20241001T194913Z

mc: HTTP/1.1 403 Forbidden Connection: close Content-Length: 110 Content-Type: application/xml Date: Tue, 01 Oct 2024 19:49:13 GMT

Despite getting permission error, mc successfully complete command. That show that this command is not necessary and can be removed.

Expected Behavior

mc is uploading file and not issuing read file commands - anything else except PUT will cause permission error.

Current Behavior

mc is issuing HEAD command, to return file metadata.

Possible Solution

Remove getting file metadata altogether. Or, add switch not to issue any extra S3 API calls except PUT to upload file.

Steps to Reproduce (for bugs)

Run command: mc cp myfile myalias/mybycket/newfile --debug

You will see request: mc: HEAD /myalias/mybycket/newfile HTTP/1.1

Read, list, stats etc of file is not permitted, and it returns error: mc: HTTP/1.1 403 Forbidden

At same time, upload works just fine and mc return OK.

1. 2. 3. 4.

Context

With uploading of millions of files, permission errors from each upload spread into error reporting system, chocking it.

Regression

Your Environment

• Version used (minio --version):minio-go/v7.0.66 mc/RELEASE.2023-12-23T08-47-21Z
• Server setup and configuration:
• Operating System and version (uname -a):

[harshavardhana]

Try using mc put instead

[scherepanov]

@harshavardhana Big thanks for taking time to answer. Yes, since 2024-02, mc has command put, and it works perfectly well. It requires only privileges getBucketLocation and PutObject. Out of curiosity, why mc always start with getting bucket location? Is it a way to make sure that S3 is operational?

[harshavardhana]

Yes you can set MC_REGION to avoid bucket region lookup.

[scherepanov]

Thanks for a hint! MC_REGION=X mc put myfile blackhole/mybucket Debug does show - it is only calling Put API.