search

noobaa / noobaa-core

High-performance S3 application gateway to any backend - file / s3-compatible / multi-clouds / caching / replication ...
https://www.noobaa.io
Apache License 2.0
265 stars 78 forks source link

Unable to reuse buckets names from buckets previously deleted #8180

Closed ramya-c3 closed 1 month ago

ramya-c3 commented 2 months ago

Environment info

Actual behavior

  1. Create a bucket bucket1 with filesystempath /mnt/gpfs1/tes/test1
  2. Delete the filesystempath /mnt/gpfs1/tes/test1
  3. Now try to delete the bucket bucket1. It fails but bucket list displays it.

noobaa-cli bucket delete bucket1 2>/dev/null

{ "error": { "code": "NoSuchBucket", "message": "Bucket does not exist" } }

noobaa-cli bucket list 2>/dev/null

{ "response": { "code": "BucketList", "reply": [ { "name": "bucket1" } ] } }

Expected behavior

  1. It should error out properly instead of saying bucket doesnot exist it should say bucket path missing

Steps to reproduce

1.

More information - Screenshots / Logs / Other output

romayalon commented 2 months ago

@shirady PTAL, looks like a small fix - we need to allow deleting the bucket when the underlying path was already deleted.

romayalon commented 1 month ago

@shirady Do you see this issue also on S3 flow?

romayalon commented 1 month ago

@ramya-c3 can you please validate? @shirady fixed it via #8187

shirady commented 1 month ago

@romayalon I see an issue on S3 flow, and also we would need to fix the error code. Do you prefer to open an issue for this or continue on this issue?

Partial output:

2024-07-16 14:57:29.395996 [PID-79847/TID-259] [L1] FS::FSWorker::OnError: Stat _path=/tmp/nsfs_root1/bucket-1001-16-07  error.Message()=No such file or directory
Jul-16 14:57:29.398 [nsfs/79847] [ERROR] core.sdk.bucketspace_fs:: BucketSpaceFS: delete_bucket: error [Error: No such file or directory] { code: 'ENOENT', rpc_code: 'NO_SUCH_OBJECT' }
Jul-16 14:57:29.398 [nsfs/79847] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/bucket-1001-16-07</Resource><RequestId>lyoczl4b-db24b2-oza</RequestId></Error> DELETE /bucket-1001-16-07 {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.17.11 md/awscrt#0.20.11 ua/2.0 os/macos#23.4.0 md/arch#arm64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3.rb","x-amz-date":"20240716T115729Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=<>/20240716/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=163e27c0cf9f108363fef1f54034a505f6f4f43bae6d52d604fa16026ee356a2","content-length":"0"} Error: No such file or directory

In the flow, it fails on: delete_bucket (bucketspace_fs) -> ns.delete_uls (namespace_fs) -> fails when it tries to list -> error translated from ENOENT to err.rpc_code = 'NO_SUCH_OBJECT' in namespace_fs and then we don't translate it again in bucketspace_fs again.

Steps to reproduce:

  1. Create the root user account with the CLI: sudo node src/cmd/manage_nsfs account add --name shira-1001 --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid> Note: before creating the account need to give permission to the new_buckets_path: chmod 777 /tmp/nsfs_root1.
  2. Start the NSFS server with: sudo node src/cmd/nsfs --debug 5 Note: before starting the server please add this line: process.env.NOOBAA_LOG_LEVEL = 'nsfs'; in the endpoint.js (before the condition if (process.env.NOOBAA_LOG_LEVEL) {)
  3. Create the alias for S3 service: alias s3-nc-user-1='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443'.
  4. Create the bucket using S3: s3-nc-user-1-s3 s3 mb s3://bucket-1001-16-07
  5. Remove the directory that was created for the bucket: sudo rmdir /tmp/nsfs_root1/bucket-1001-16-07
  6. Try to delete the bucket: s3-nc-user-1-s3 s3 rb s3://bucket-1001-16-07, the output: remove_bucket failed: s3://bucket-1001-16-07 An error occurred (NoSuchKey) when calling the DeleteBucket operation: The specified key does not exist.
romayalon commented 1 month ago

Thanks @shirady ! No need for another issue, i'll reopen this one just for S3 flow, CLI (the original bug) already fixed via #8187