NC | NSFS | Versioning | Delete of partial directory of nested key results in `AccessDeniedError`

[shirady]

Environment info

• NooBaa Version: 5.18.0 (current master)
• Platform: NC

Actual behavior

1. When a bucket with versioning enabled and there is a nested key inside and the client tries to delete the parent directory of the file without the last slash '/' (let's say it was by accident) will result in an error AccessDenied and in FS layer it is Error: Operation not permitted, code: 'EPERM'

Expected behavior

1. Create a delete marker without an error.

Steps to reproduce

1. Create an account with the CLI: sudo node src/cmd/manage_nsfs account add --name <account-name> --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid> Note: before creating the account need to give permission to the new_buckets_path: chmod 777 /tmp/nsfs_root1, chmod 777 /tmp/nsfs_root2.
2. Start the NSFS server with: sudo node src/cmd/nsfs --debug 5 Notes:
   • I Change the config.NSFS_CHECK_BUCKET_BOUNDARIES = false; //SDSD because I’m using the /tmp/ and not /private/tmp/.
3. Create the alias for S3 service:alias nc-user-1-s3=‘AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443’.
4. Check the connection to the endpoint and try to list the buckets (should be empty): nc-user-1-s3 s3 ls; echo $?
5. Add bucket to the account using AWS CLI: nc-user-1-s3 s3 mb s3://bucket-v (bucket-v is the bucket name in this example)
6. Enable versioning: nc-user-1-s3 s3api put-bucket-versioning --bucket bucket-v --versioning-configuration Status=Enabled
7. Put a nested object: nc-user-1-s3 s3api put-object --bucket bucket-v --key /a/b/c/lala.txt
8. Delete the key /a/b/c (without the last slash): nc-user-1-s3 s3api delete-object --bucket bucket-v --key /a/b/c Note: deleting the directory creates the delete marker as expected (nc-user-1-s3 s3api delete-object --bucket bucket-v --key /a/b/c/).

More information - Screenshots / Logs / Other output

Might be that this issue is with the same root cause of the issue - #8320

Oct-10 16:13:38.960 [nsfs/79233]  [WARN] core.sdk.namespace_fs:: NamespaceFS._delete_latest_version error: retries=10 latest_ver_path=/tmp/nsfs_root1/bucket-v/a/b/c [Error: Operation not permitted] { code: 'EPERM', context: 'SafeLink _link_from.c_str()=/tmp/nsfs_root1/bucket-v/a/b/c _link_to.c_str()=/tmp/nsfs_root1/bucket-v/a/b/.versions/c_null _link_expected_mtime=1728566008914826496 _link_expected_inode=152582414 ' }
Oct-10 16:13:38.961 [nsfs/79233] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/bucket-v//a/b/c</Resource><RequestId>m23bksck-csss5c-135r</RequestId></Error> DELETE /bucket-v//a/b/c {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.17.11 md/awscrt#0.20.11 ua/2.0 os/macos#24.0.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.delete-object","x-amz-date":"20241010T131338Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=<>/20241010/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4dc8ae840909447ab17ef08ce5c6d25c62a422ed1ba7fa79900a210ac1cfa3b5","content-length":"0"} Error: Operation not permitted - context: SafeLink _link_from.c_str()=/tmp/nsfs_root1/bucket-v/a/b/c _link_to.c_str()=/tmp/nsfs_root1/bucket-v/a/b/.versions/c_null _link_expected_mtime=1728566008914826496 _link_expected_inode=152582414

[naveenpaul1]

@romayalon same behaiour is tested in version enabled bucket and its working with out creating any issue

1. put key to object

   s3-api put-object --bucket naveen-noobaa-bucket --key x/y/z/lala.txt
   {
   "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
   "ServerSideEncryption": "AES256",
   "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb"
   }

2. listing item

   3-api list-object-versions --bucket naveen-noobaa-bucket
   {
   "Versions": [
       {
           "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
           "Size": 0,
           "StorageClass": "STANDARD",
           "Key": "x/y/z/lala.txt",
           "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb",
           "IsLatest": true,
           "LastModified": "2024-11-04T10:10:26+00:00",
           "Owner": {
               "DisplayName": "pnt-aws-root+255735097469",
               "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
           }
       }
   ],
   "RequestCharged": null
   }

3. delete parial dir key and its retun delete marker

   s3-api delete-object --bucket naveen-noobaa-bucket --key  x/y/z
   {
   "DeleteMarker": true,
   "VersionId": "5OKNiYVfBRTsyZzQhEV9WMQcpq3H8Gn0"
   }

4. list bucket after deleting partial dir

   naveenpaul:noobaa-core$ s3-api list-object-versions --bucket naveen-noobaa-bucket
   {
   "Versions": [
       {
           "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
           "Size": 0,
           "StorageClass": "STANDARD",
           "Key": "x/y/z/lala.txt",
           "VersionId": "xoWDrLSfwBFc4ByeDT1hD3VIPambJblb",
           "IsLatest": true,
           "LastModified": "2024-11-04T10:10:26+00:00",
           "Owner": {
               "DisplayName": "pnt-aws-root+255735097469",
               "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
           }
       }
   ],
   "DeleteMarkers": [
       {
           "Owner": {
               "DisplayName": "pnt-aws-root+255735097469",
               "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
           },
           "Key": "x/y/z",
           "VersionId": "5OKNiYVfBRTsyZzQhEV9WMQcpq3H8Gn0",
           "IsLatest": true,
           "LastModified": "2024-11-04T10:11:33+00:00"
       }
   ],
   "RequestCharged": null
   }

cc : @shirady

[naveenpaul1]

@romayalon both noobaa and aws behaving similer, When I try to get object using dir path such as (a/b/c or a/b/c/) its returning The specified key does not exist. error for both noobaa and AWS

Local noobaa behavior

1. put object

   s3-local put-object --bucket bucket4 --key a/b/c/lala.txt
   urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
   {
   "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
   "VersionId": "mtime-d5ddydsepvcw-ino-byd1o"
   }

2. try to get using dir path both failed

   
   s3-local get-object --bucket bucket4 --key a/b/c ./src/cmd/text1.txt
   urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-local get-object --bucket bucket4 --key a/b/c/ ./src/cmd/text1.txt urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

3. try with object key path and its working

naveenpaul:noobaa-core$ s3-local get-object --bucket bucket4 --key a/b/c/lala.txt ./src/cmd/text1.txt urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings { "AcceptRanges": "bytes", "LastModified": "2024-11-04T12:16:30+00:00", "ContentLength": 0, "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"", "VersionId": "mtime-d5ddydsepvcw-ino-byd1o", "ContentType": "text/plain", "Metadata": {} }

**AWS Behavior** 
1. list object items

naveenpaul:noobaa-core$ s3-aws list-objects --bucket naveen-noobaa-bucket { "Contents": [ { "Key": "i/j/k/lala.txt", "LastModified": "2024-11-04T11:50:33+00:00", "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"", "Size": 0, "StorageClass": "STANDARD", "Owner": { "DisplayName": "pnt-aws-root+255735097469", "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902" } }, { "Key": "x/y/z/lala.txt", "LastModified": "2024-11-04T10:10:26+00:00", "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"", "Size": 0, "StorageClass": "STANDARD", "Owner": { "DisplayName": "pnt-aws-root+255735097469", "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902" } } ], "RequestCharged": null }

2. try to get using dir path (i/j/k or i/j/k/) both failed.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k ./src/cmd/text1.txt An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/ ./src/cmd/text1.txt\ An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

3. get object with object complete path and its working

noobaa-core$ s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/lala.txt ./src/cmd/text1.txt { "AcceptRanges": "bytes", "LastModified": "2024-11-04T11:50:33+00:00", "ContentLength": 0, "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"", "VersionId": "MnMY1mGNXquvYLicPaEeqgFb8gmNJ_Ff", "ContentType": "binary/octet-stream", "ServerSideEncryption": "AES256", "Metadata": {} }

[romayalon]

@romayalon both noobaa and aws behaving similer, When I try to get object using dir path such as (a/b/c or a/b/c/) its returning The specified key does not exist. error for both noobaa and AWS

Local noobaa behavior

1. put object

s3-local put-object --bucket bucket4 --key a/b/c/lala.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o"
}

2. try to get using dir path both failed

s3-local get-object --bucket bucket4 --key a/b/c ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-local get-object --bucket bucket4 --key a/b/c/ ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

3. try with object key path and its working

naveenpaul:noobaa-core$ s3-local get-object --bucket bucket4 --key a/b/c/lala.txt ./src/cmd/text1.txt
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T12:16:30+00:00",
    "ContentLength": 0,
    "ETag": "\"mtime-d5ddydsepvcw-ino-byd1o\"",
    "VersionId": "mtime-d5ddydsepvcw-ino-byd1o",
    "ContentType": "text/plain",
    "Metadata": {}
}

AWS Behavior

1. list object items

naveenpaul:noobaa-core$  s3-aws list-objects --bucket naveen-noobaa-bucket
{
    "Contents": [
        {
            "Key": "i/j/k/lala.txt",
            "LastModified": "2024-11-04T11:50:33+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        },
        {
            "Key": "x/y/z/lala.txt",
            "LastModified": "2024-11-04T10:10:26+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "pnt-aws-root+255735097469",
                "ID": "2e055fd7dc2c9eb01fe8d089e32b84c4b9e8a66b85e370793b4c7259ea477902"
            }
        }
    ],
    "RequestCharged": null
}

2. try to get using dir path (i/j/k or i/j/k/) both failed.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k ./src/cmd/text1.txt
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/ ./src/cmd/text1.txt\
An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

3. get object with object complete path and its working

noobaa-core$ s3-aws get-object --bucket naveen-noobaa-bucket --key i/j/k/lala.txt ./src/cmd/text1.txt
{
    "AcceptRanges": "bytes",
    "LastModified": "2024-11-04T11:50:33+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "MnMY1mGNXquvYLicPaEeqgFb8gmNJ_Ff",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

@naveenpaul1 On NooBaa you tested it on the branch that contains your fix https://github.com/noobaa/noobaa-core/pull/8480 ?

[naveenpaul1]

yes @romayalon , I did the test with my changes in branch delete_partial_dir