AWS mb command is failing with access denied error . native_fs_utils: create_config_file error [Error: Permission denied] { code: 'EACCES' }

Environment info

NooBaa Version: VERSION noobaa-core-5.15.0-20231207.el9.x86_6
Platform: Standalone RHEL 9.2

Actual behavior

AWS mb command is failing with access denied error. From the noobaa.log it is evident that the create config file for bucket is failing with permission issue (see below for more details). However node command is working with out any issue.

Expected behavior

AWS mb command should create the config files as well as bucket directory successfully.

Steps to reproduce

Create account with uid and did

[root@an2node-x-worker1 certificates]# /usr/local/noobaa-core/bin/node /usr/local/noobaa-core/src/cmd/manage_nsfs account add --name an-test1 --email an-test1@gmail.com --new_buckets_path /mnt/fs1/s3-user234 --uid 234 --gid 234 --access_key a-antest1 --secret_key s-antest1

Create alias for that account [root@an2node-x-worker1 certificates]# alias aws-antest1="AWS_ACCESS_KEY_ID=a-antest1 AWS_SECRET_ACCESS_KEY=s-antest1 aws --endpoint https://localhost:6443 --no-verify-ssl s3"

Now create a bucket using aws mb command --> FAILING

[root@an2node-x-worker1 certificates]# aws-antest1 mb s3://newbucket-anandhu
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
make_bucket failed: s3://newbucket-anandhu An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

From the noobaa.log. check for time stamp 2023-12-13T03:08:09.562411-08:00

2023-12-13T03:08:09.553051-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /newbucket-anandhu search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.0 Python/3.11.6 Linux/5.14.0-284.30.1.el9_2.x86_64 exe/x86_64.rhel.9 prompt/off command/s3.mb', 'X-Amz-Date': '20231213T110809Z', 'X-Amz-Content-Sha256': '61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1', Authorization: 'AWS4-HMAC-SHA256 Credential=a-antest1/20231213/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=815c8312c93da125e17f2e32725d27b0a53642d55941e77a3444437e96d9a94d', 'Content-Length': '154' } region us-east-1 canonical_str PUT/newbucket-anandhuhost:localhost:6443x-amz-content-sha256:61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1x-amz-date:20231213T110809Zhost;x-amz-content-sha256;x-amz-date61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1 string_to_sign AWS4-HMAC-SHA25620231213T110809Z20231213/us-east-1/s3/aws4_request1bf0e7ae126e6eca4251e29b653bc5e3bbf7037102f7d0e9f8feb661bfeb894b
2023-12-13T03:08:09.554892-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /newbucket-anandhu search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.0 Python/3.11.6 Linux/5.14.0-284.30.1.el9_2.x86_64 exe/x86_64.rhel.9 prompt/off command/s3.mb', 'X-Amz-Date': '20231213T110809Z', 'X-Amz-Content-Sha256': '61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1', Authorization: 'AWS4-HMAC-SHA256 Credential=a-antest1/20231213/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=815c8312c93da125e17f2e32725d27b0a53642d55941e77a3444437e96d9a94d', 'Content-Length': '154' } region us-east-1 canonical_str PUT/newbucket-anandhuhost:localhost:6443x-amz-content-sha256:61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1x-amz-date:20231213T110809Zhost;x-amz-content-sha256;x-amz-date61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1 string_to_sign AWS4-HMAC-SHA25620231213T110809Z20231213/us-east-1/s3/aws4_request1bf0e7ae126e6eca4251e29b653bc5e3bbf7037102f7d0e9f8feb661bfeb894b
2023-12-13T03:08:09.556797-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294]    [L1] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /newbucket-anandhu op put_bucket request_id lq3o45am-96ovrl-fgx { host: 'localhost:6443', 'accept-encoding': 'identity', 'user-agent': 'aws-cli/2.15.0 Python/3.11.6 Linux/5.14.0-284.30.1.el9_2.x86_64 exe/x86_64.rhel.9 prompt/off command/s3.mb', 'x-amz-date': '20231213T110809Z', 'x-amz-content-sha256': '61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1', authorization: 'AWS4-HMAC-SHA256 Credential=a-antest1/20231213/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=815c8312c93da125e17f2e32725d27b0a53642d55941e77a3444437e96d9a94d', 'content-length': '154' }
2023-12-13T03:08:09.559366-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.create_bucket                 requesting_account={ name: SENSITIVE-a29894a871e8fa36, email: SENSITIVE-21b82afecdffd667, creation_date: '2023-12-13T09:36:19.028Z', access_keys: [ { access_key: SENSITIVE-12111d0945c2c835, secret_key: SENSITIVE-5e3c4baf4cfc5ec1 } ], nsfs_account_config: { uid: 234, gid: 234, new_buckets_path: '/mnt/fs1/s3-user234', warn_threshold_ms: 100 } },                bucket_config_path=/gpfs/fs-ces-shared/ces/s3-config/buckets/newbucket-anandhu.json,                bucket_storage_path=/mnt/fs1/s3-user234/newbucket-anandhu
2023-12-13T03:08:09.562411-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294]    [L1] core.util.native_fs_utils:: native_fs_utils: create_config_file config_path: /gpfs/fs-ces-shared/ces/s3-config/buckets/newbucket-anandhu.json config_data: {"name":"newbucket-anandhu","tag":"","system_owner":"an-test1@gmail.com","bucket_owner":"an-test1@gmail.com","versioning":"DISABLED","creation_date":"2023-12-13T11:08:09.559Z","path":"/mnt/fs1/s3-user234/newbucket-anandhu","should_create_underlying_storage":true} is_gpfs: w
2023-12-13T03:08:09.575741-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294] [ERROR] core.util.native_fs_utils:: native_fs_utils: create_config_file error [Error: Permission denied] { code: 'EACCES' }
2023-12-13T03:08:09.576780-08:00 an2node-x-worker1 node[3627294]: [nsfs/3627294] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/newbucket-anandhu</Resource><RequestId>lq3o45am-96ovrl-fgx</RequestId></Error> PUT /newbucket-anandhu {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.15.0 Python/3.11.6 Linux/5.14.0-284.30.1.el9_2.x86_64 exe/x86_64.rhel.9 prompt/off command/s3.mb","x-amz-date":"20231213T110809Z","x-amz-content-sha256":"61d056dc66f1882c0f4053be381523a7a28d384abde04fcf5b0021c716bb0ea1","authorization":"AWS4-HMAC-SHA256 Credential=a-antest1/20231213/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=815c8312c93da125e17f2e32725d27b0a53642d55941e77a3444437e96d9a94d","content-length":"154"} Error: Permission denied

Possible root cause is due to uid and gid I specified during account creation. That uid does not have access to create config file /gpfs/fs-ces-shared/ces/s3-config/buckets/newbucket-anandhu.json

[root@an2node-x-worker1 certificates]# ls -ld /gpfs/fs-ces-shared/ces/s3-config/buckets/
drwxr-xr-x 3 root root 4096 Dec 13 02:01 /gpfs/fs-ces-shared/ces/s3-config/buckets/

noobaa / noobaa-core