search

noobaa / noobaa-core

High-performance S3 application gateway to any backend - file / s3-compatible / multi-clouds / caching / replication ...
https://www.noobaa.io
Apache License 2.0
271 stars 80 forks source link

Nc | NSFS | Creating a Bucket with Name That Is the Same as a Internal Directory Throws `BucketAlreadyExists` #8193

Closed shirady closed 3 months ago

shirady commented 4 months ago

Environment info

Actual behavior

  1. Creating a bucket with name that is the same as a internal directory throws BucketAlreadyExists, when there is not config file with this name.

Expected behavior

  1. I think we should differentiate between a bucket that has a config file (and therefore BucketAlreadyExists explains it) and a check that we do and there is a file that already exists (the bucket doesn't have a config file).

Steps to reproduce

  1. Create a path: sudo mkdir -p /tmp/nsfs_root1/my-bucket (the /tmp/nsfs_root1 will be used as new_buckets_path the sub directory my-bucket will create the issue) and give permission chmod 777 /tmp/nsfs_root1/..
  2. Create an account: sudo node src/cmd/manage_nsfs account add --name <account-name> --uid <uid> --gid <gid> --new_buckets_path <new_buckets_path> (use those credentials in the alias step below).
  3. Start the NSFS server with: sudo node src/cmd/nsfs --debug 5 Create the alias for the created account in the S3 service: alias s3-nc-user-1='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443/' Note: I Change the config.NSFS_CHECK_BUCKET_BOUNDARIES = false; //SDSD because I'm using the /tmp/ and not /private/tmp/.
  4. Create a bucket using S3 API with the name of the created subdirectory: s3-nc-user-1 s3 mb s3://my-bucket

To show the difference that we have in the logs between the cases:

  1. Delete the sub directory: sudo rmdir /tmp/nsfs_root1/my-bucket
  2. Create a bucket using S3 API: s3-nc-user-1 s3 mb s3://my-bucket
  3. Create the bucket again: s3-nc-user-1 s3 mb s3://my-bucket (expect BucketAlreadyExists).

More information - Screenshots / Logs / Other output

Output of s3-nc-user-1 s3 mb s3://my-bucket:

make_bucket failed: s3://my-bucket An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Partial logs (udo node src/cmd/nsfs --debug 5):

Jul-10 13:58:47.855 [nsfs/19489]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /my-bucket search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb', 'X-Amz-Date': '20240710T105847Z', 'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', Authorization: 'AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3aa47d133d544823e492cdadcf2a4d5dbffe77f980d63a64c64d49a7f47794e5', 'Content-Length': '0' } region us-east-1 canonical_str
PUT
/my-bucket

host:localhost:6443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240710T105847Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 string_to_sign
AWS4-HMAC-SHA256
20240710T105847Z
20240710/us-east-1/s3/aws4_request
f2f3296ca1f6b1405e03c71579e78bb515211efb953440902a2c8c35a9716284

Jul-10 13:58:47.856 [nsfs/19489]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /my-bucket search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb', 'X-Amz-Date': '20240710T105847Z', 'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', Authorization: 'AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3aa47d133d544823e492cdadcf2a4d5dbffe77f980d63a64c64d49a7f47794e5', 'Content-Length': '0' } region us-east-1 canonical_str
PUT
/my-bucket

host:localhost:6443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240710T105847Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 string_to_sign
AWS4-HMAC-SHA256
20240710T105847Z
20240710/us-east-1/s3/aws4_request
f2f3296ca1f6b1405e03c71579e78bb515211efb953440902a2c8c35a9716284

Jul-10 13:58:47.859 [nsfs/19489]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.create_bucket
                requesting_account={ _id: '668e67a37162b7b11fe8c2e6', name: SENSITIVE-34617e85c8ee3957, email: SENSITIVE-34617e85c8ee3957, creation_date: '2024-07-10T10:51:15.038Z', access_keys: [ { access_key: SENSITIVE-d52ff789eb6f71e8, secret_key: SENSITIVE-c60ed951c4165069 } ], nsfs_account_config: { uid: 1001, gid: 1001, new_buckets_path: '/tmp/nsfs_root1', warn_threshold_ms: 100, backend: undefined }, allow_bucket_creation: true, master_key_id: '668e67a3f692d84ab061636b' },
                bucket_config_path=/etc/noobaa.conf.d/buckets/my-bucket.json,
                bucket_storage_path=/tmp/nsfs_root1/my-bucket
Jul-10 13:58:47.860 [nsfs/19489]    [L2] core.sdk.bucketspace_fs:: create_bucket: bucket properties before validate_bucket_schema { _id: '668e69674b8369cb2ebb5845', name: 'my-bucket', owner_account: '668e67a37162b7b11fe8c2e6', system_owner: 'shira-1001', bucket_owner: 'shira-1001', versioning: 'DISABLED', creation_date: '2024-07-10T10:58:47.860Z', path: '/tmp/nsfs_root1/my-bucket', should_create_underlying_storage: true }
2024-07-10 13:58:47.861315 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json
2024-07-10 13:58:47.861379 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.861405 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.861465 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json  took: 0.031709 ms
2024-07-10 13:58:47.861514 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json  error.Message()=No such file or directory
Jul-10 13:58:47.861 [nsfs/19489]    [L1] core.util.native_fs_utils:: create_config_file:: config_path: /etc/noobaa.conf.d/buckets/my-bucket.json config_data: {"_id":"668e69674b8369cb2ebb5845","name":"my-bucket","owner_account":"668e67a37162b7b11fe8c2e6","system_owner":"shira-1001","bucket_owner":"shira-1001","versioning":"DISABLED","creation_date":"2024-07-10T10:58:47.860Z","path":"/tmp/nsfs_root1/my-bucket","should_create_underlying_storage":true} is_gpfs: w
2024-07-10 13:58:47.861868 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/ _mode=448
2024-07-10 13:58:47.861899 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=448 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.861919 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=448 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.861950 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=448  took: 0.0035 ms
2024-07-10 13:58:47.861972 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/ _mode=448  error.Message()=File exists
2024-07-10 13:58:47.862007 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc _mode=448
2024-07-10 13:58:47.862025 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=448 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.862047 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=448 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.862076 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=448  took: 0.002792 ms
2024-07-10 13:58:47.862103 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc _mode=448  error.Message()=File exists
2024-07-10 13:58:47.862147 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d _mode=448
2024-07-10 13:58:47.862163 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=448 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.862186 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=448 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.862224 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=448  took: 0.009708 ms
2024-07-10 13:58:47.862246 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d _mode=448  error.Message()=File exists
2024-07-10 13:58:47.862287 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=448
2024-07-10 13:58:47.862305 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=448 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.862324 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=448 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.862354 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=448  took: 0.004208 ms
2024-07-10 13:58:47.862379 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=448  error.Message()=File exists
2024-07-10 13:58:47.862414 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Fsync _path=/etc/noobaa.conf.d/buckets
2024-07-10 13:58:47.862431 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.862449 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.862530 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets  took: 0.048167 ms
2024-07-10 13:58:47.862551 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined Fsync _path=/etc/noobaa.conf.d/buckets
2024-07-10 13:58:47.862853 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/ _mode=504
2024-07-10 13:58:47.862874 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.862896 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.862925 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504  took: 0.001083 ms
2024-07-10 13:58:47.862946 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/ _mode=504  error.Message()=File exists
2024-07-10 13:58:47.862976 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc _mode=504
2024-07-10 13:58:47.862993 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863010 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863044 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504  took: 0.001375 ms
2024-07-10 13:58:47.863062 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863092 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d _mode=504
2024-07-10 13:58:47.863107 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863128 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863162 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504  took: 0.004125 ms
2024-07-10 13:58:47.863180 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863208 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504
2024-07-10 13:58:47.863224 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863243 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863274 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  took: 0.003375 ms
2024-07-10 13:58:47.863291 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863320 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504
2024-07-10 13:58:47.863337 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863357 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863388 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  took: 0.004042 ms
2024-07-10 13:58:47.863406 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863436 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504
2024-07-10 13:58:47.863452 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863469 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863504 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  took: 0.004416 ms
2024-07-10 13:58:47.863522 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863548 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
2024-07-10 13:58:47.863563 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863582 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863656 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found  took: 0.043042 ms
2024-07-10 13:58:47.863673 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
Jul-10 13:58:47.863 [nsfs/19489]    [L1] core.util.native_fs_utils:: NamespaceFS._open_file: mode=w creating dirs /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 /etc/noobaa.conf.d/buckets
2024-07-10 13:58:47.863885 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/ _mode=504
2024-07-10 13:58:47.863902 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.863920 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.863949 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504  took: 0.001208 ms
2024-07-10 13:58:47.863967 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/ _mode=504  error.Message()=File exists
2024-07-10 13:58:47.863995 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc _mode=504
2024-07-10 13:58:47.864011 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864030 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864057 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504  took: 0.001125 ms
2024-07-10 13:58:47.864074 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc _mode=504  error.Message()=File exists
2024-07-10 13:58:47.864106 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d _mode=504
2024-07-10 13:58:47.864120 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864138 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864168 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504  took: 0.003458 ms
2024-07-10 13:58:47.864188 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d _mode=504  error.Message()=File exists
2024-07-10 13:58:47.864216 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504
2024-07-10 13:58:47.864231 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864249 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864286 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  took: 0.003542 ms
2024-07-10 13:58:47.864303 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  error.Message()=File exists
2024-07-10 13:58:47.864333 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504
2024-07-10 13:58:47.864348 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864369 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864400 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  took: 0.00325 ms
2024-07-10 13:58:47.864417 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  error.Message()=File exists
2024-07-10 13:58:47.864444 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504
2024-07-10 13:58:47.864460 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864480 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864511 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  took: 0.002959 ms
2024-07-10 13:58:47.864528 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  error.Message()=File exists
2024-07-10 13:58:47.864552 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
2024-07-10 13:58:47.864570 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864589 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.864637 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found  took: 0.022 ms
2024-07-10 13:58:47.864658 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
Jul-10 13:58:47.864 [nsfs/19489]    [L1] core.util.native_fs_utils:: NamespaceFS._open_file: mode=w /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32
2024-07-10 13:58:47.864849 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: FileOpen _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _flags=1537 _mode=384
2024-07-10 13:58:47.864879 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileOpen _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _flags=1537 _mode=384 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.864898 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileOpen _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _flags=1537 _mode=384 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.865012 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileOpen _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _flags=1537 _mode=384  took: 0.084125 ms
2024-07-10 13:58:47.865032 [PID-19489/TID-259] [L1] FS::FileOpen::OnOK: _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32
2024-07-10 13:58:47.865092 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: FileWritev _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _total_len=294 buffers_len=1 _offset=0
2024-07-10 13:58:47.865107 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: FileWritev _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _total_len=294 buffers_len=1 _offset=0 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.865131 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: FileWritev _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _total_len=294 buffers_len=1 _offset=0 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.865202 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: FileWritev _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _total_len=294 buffers_len=1 _offset=0  took: 0.042916 ms
2024-07-10 13:58:47.865222 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined FileWritev _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _total_len=294 buffers_len=1 _offset=0
2024-07-10 13:58:47.865249 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Stat _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32
2024-07-10 13:58:47.865264 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.865289 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.865354 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32  took: 0.036208 ms
2024-07-10 13:58:47.865378 [PID-19489/TID-259] [L1] FS::Stat::OnOK: _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _stat_res.st_ino=136918623 _stat_res.st_size=294
Jul-10 13:58:47.865 [nsfs/19489]    [L1] core.util.native_fs_utils:: create_config_file:: moving from: /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 to: /etc/noobaa.conf.d/buckets/my-bucket.json is_gpfs= false
Jul-10 13:58:47.865 [nsfs/19489]    [L1] core.util.native_fs_utils:: Namespace_fs.safe_move_posix /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 /etc/noobaa.conf.d/buckets/my-bucket.json { dev: 16777233, ino: 136918623, mode: 33152, nlink: 1, uid: 0, gid: 0, rdev: 0, size: 294, blksize: 4096, blocks: 8, atimeMs: 1720609127864.9707, ctimeMs: 1720609127865.2002, mtimeMs: 1720609127865.2002, birthtimeMs: 1720609127864.9707, atime: 2024-07-10T10:58:47.865Z, mtime: 2024-07-10T10:58:47.865Z, ctime: 2024-07-10T10:58:47.865Z, birthtime: 2024-07-10T10:58:47.865Z, atimeNsBigint: 1720609127864970752n, ctimeNsBigint: 1720609127865200384n, mtimeNsBigint: 1720609127865200384n, xattr: {} }
Jul-10 13:58:47.865 [nsfs/19489]    [L1] core.util.native_fs_utils:: Namespace_fs.safe_link_posix: /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 /etc/noobaa.conf.d/buckets/my-bucket.json { dev: 16777233, ino: 136918623, mode: 33152, nlink: 1, uid: 0, gid: 0, rdev: 0, size: 294, blksize: 4096, blocks: 8, atimeMs: 1720609127864.9707, ctimeMs: 1720609127865.2002, mtimeMs: 1720609127865.2002, birthtimeMs: 1720609127864.9707, atime: 2024-07-10T10:58:47.865Z, mtime: 2024-07-10T10:58:47.865Z, ctime: 2024-07-10T10:58:47.865Z, birthtime: 2024-07-10T10:58:47.865Z, atimeNsBigint: 1720609127864970752n, ctimeNsBigint: 1720609127865200384n, mtimeNsBigint: 1720609127865200384n, xattr: {} }
2024-07-10 13:58:47.865785 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: SafeLink _link_from.c_str()=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _link_to.c_str()=/etc/noobaa.conf.d/buckets/my-bucket.json _link_expected_mtime=1720609127865200384 _link_expected_inode=136918623
2024-07-10 13:58:47.865910 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeLink _link_from.c_str()=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _link_to.c_str()=/etc/noobaa.conf.d/buckets/my-bucket.json _link_expected_mtime=1720609127865200384 _link_expected_inode=136918623 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.865930 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeLink _link_from.c_str()=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _link_to.c_str()=/etc/noobaa.conf.d/buckets/my-bucket.json _link_expected_mtime=1720609127865200384 _link_expected_inode=136918623 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.866239 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeLink _link_from.c_str()=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _link_to.c_str()=/etc/noobaa.conf.d/buckets/my-bucket.json _link_expected_mtime=1720609127865200384 _link_expected_inode=136918623  took: 0.281083 ms
2024-07-10 13:58:47.866260 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined SafeLink _link_from.c_str()=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _link_to.c_str()=/etc/noobaa.conf.d/buckets/my-bucket.json _link_expected_mtime=1720609127865200384 _link_expected_inode=136918623
Jul-10 13:58:47.866 [nsfs/19489]    [L1] core.util.native_fs_utils:: Namespace_fs.safe_unlink_posix: /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 { dev: 16777233, ino: 136918623, mode: 33152, nlink: 1, uid: 0, gid: 0, rdev: 0, size: 294, blksize: 4096, blocks: 8, atimeMs: 1720609127864.9707, ctimeMs: 1720609127865.2002, mtimeMs: 1720609127865.2002, birthtimeMs: 1720609127864.9707, atime: 2024-07-10T10:58:47.865Z, mtime: 2024-07-10T10:58:47.865Z, ctime: 2024-07-10T10:58:47.865Z, birthtime: 2024-07-10T10:58:47.865Z, atimeNsBigint: 1720609127864970752n, ctimeNsBigint: 1720609127865200384n, mtimeNsBigint: 1720609127865200384n, xattr: {} } /etc/noobaa.conf.d/buckets/.noobaa-config-nsfs
2024-07-10 13:58:47.866875 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/ _mode=504
2024-07-10 13:58:47.866895 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.866916 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.866948 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/ _mode=504  took: 0.001333 ms
2024-07-10 13:58:47.866967 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/ _mode=504  error.Message()=File exists
2024-07-10 13:58:47.867000 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc _mode=504
2024-07-10 13:58:47.867037 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.867057 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.868564 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc _mode=504  took: 0.002417 ms
2024-07-10 13:58:47.868816 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc _mode=504  error.Message()=File exists
2024-07-10 13:58:47.869083 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d _mode=504
2024-07-10 13:58:47.869109 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.869130 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.869162 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d _mode=504  took: 0.00475 ms
2024-07-10 13:58:47.869184 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d _mode=504  error.Message()=File exists
2024-07-10 13:58:47.869225 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504
2024-07-10 13:58:47.869245 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.869263 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.870074 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  took: 0.004708 ms
2024-07-10 13:58:47.870099 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets _mode=504  error.Message()=File exists
2024-07-10 13:58:47.870136 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504
2024-07-10 13:58:47.870153 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.870171 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.870201 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  took: 0.003459 ms
2024-07-10 13:58:47.870342 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs _mode=504  error.Message()=File exists
2024-07-10 13:58:47.870377 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504
2024-07-10 13:58:47.870394 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.870412 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.870442 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  took: 0.003583 ms
2024-07-10 13:58:47.870584 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _mode=504  error.Message()=File exists
2024-07-10 13:58:47.870675 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
2024-07-10 13:58:47.870697 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.870749 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.870816 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found  took: 0.032042 ms
2024-07-10 13:58:47.870839 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined Fsync _path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found
2024-07-10 13:58:47.870908 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: SafeUnlink _to_unlink=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _mv_to=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/4d231935-29e7-435f-b8c0-0df965af2721 _unlink_expected_mtime=1720609127865200384 _unlink_expected_inode=136918623
2024-07-10 13:58:47.870928 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeUnlink _to_unlink=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _mv_to=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/4d231935-29e7-435f-b8c0-0df965af2721 _unlink_expected_mtime=1720609127865200384 _unlink_expected_inode=136918623 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.870947 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeUnlink _to_unlink=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _mv_to=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/4d231935-29e7-435f-b8c0-0df965af2721 _unlink_expected_mtime=1720609127865200384 _unlink_expected_inode=136918623 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.872098 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: SafeUnlink _to_unlink=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _mv_to=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/4d231935-29e7-435f-b8c0-0df965af2721 _unlink_expected_mtime=1720609127865200384 _unlink_expected_inode=136918623  took: 1.12333 ms
2024-07-10 13:58:47.872127 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined SafeUnlink _to_unlink=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _mv_to=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/4d231935-29e7-435f-b8c0-0df965af2721 _unlink_expected_mtime=1720609127865200384 _unlink_expected_inode=136918623
Jul-10 13:58:47.872 [nsfs/19489]    [L1] core.util.native_fs_utils:: create_config_file:: done /etc/noobaa.conf.d/buckets/my-bucket.json
2024-07-10 13:58:47.872311 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: FileClose _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _wrap->_fd=24
2024-07-10 13:58:47.872333 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileClose _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _wrap->_fd=24 _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.872357 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileClose _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _wrap->_fd=24 _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.872456 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: FileClose _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _wrap->_fd=24  took: 0.070666 ms
2024-07-10 13:58:47.872485 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined FileClose _wrap->_path=/etc/noobaa.conf.d/buckets/.noobaa-config-nsfs/lost+found/0582f380-09a2-4ffe-9c7c-a22c4c0b8e32 _wrap->_fd=24
2024-07-10 13:58:47.872527 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Mkdir _path=/tmp/nsfs_root1/my-bucket _mode=504
2024-07-10 13:58:47.872546 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/tmp/nsfs_root1/my-bucket _mode=504 _uid=1001 _gid=1001 _backend=
2024-07-10 13:58:47.872573 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/tmp/nsfs_root1/my-bucket _mode=504 _uid=1001 _gid=1001 geteuid()=1001 getegid()=1001 getuid()=1001 getgid()=1001
2024-07-10 13:58:47.872616 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Mkdir _path=/tmp/nsfs_root1/my-bucket _mode=504  took: 0.014708 ms
2024-07-10 13:58:47.872645 [PID-19489/TID-259] [L1] FS::FSWorker::OnError: Mkdir _path=/tmp/nsfs_root1/my-bucket _mode=504  error.Message()=File exists
Jul-10 13:58:47.873 [nsfs/19489] [ERROR] core.sdk.bucketspace_fs:: BucketSpaceFS: create_bucket could not create underlying directory - nsfs, deleting bucket [Error: File exists] { code: 'EEXIST' }
2024-07-10 13:58:47.873686 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Unlink _path=/etc/noobaa.conf.d/buckets/my-bucket.json
2024-07-10 13:58:47.873725 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Unlink _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 _backend=
2024-07-10 13:58:47.873755 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Unlink _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 13:58:47.874237 [PID-19489/TID-8451] [L1] FS::FSWorker::Execute: Unlink _path=/etc/noobaa.conf.d/buckets/my-bucket.json  took: 0.454667 ms
2024-07-10 13:58:47.874276 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: undefined Unlink _path=/etc/noobaa.conf.d/buckets/my-bucket.json
Jul-10 13:58:47.874 [nsfs/19489] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.</Message><Resource>/my-bucket</Resource><RequestId>lyfq8zvx-d871nq-q4w</RequestId></Error> PUT /my-bucket {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb","x-amz-date":"20240710T105847Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3aa47d133d544823e492cdadcf2a4d5dbffe77f980d63a64c64d49a7f47794e5","content-length":"0"} Error: File exists

Run s3-nc-user-1 s3 mb s3://my-bucket after it was created:

Jul-10 14:10:12.164 [nsfs/19489]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /my-bucket search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb', 'X-Amz-Date': '20240710T111012Z', 'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', Authorization: 'AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a573dbd5c184b8dd48ee28eab7faa21509d67c4551c2fc6e53d91e7e96042d0b', 'Content-Length': '0' } region us-east-1 canonical_str
PUT
/my-bucket

host:localhost:6443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240710T111012Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 string_to_sign
AWS4-HMAC-SHA256
20240710T111012Z
20240710/us-east-1/s3/aws4_request
b97298d956a823e54914b1cb7820dcefb4dd175ffd4d96c6f11d9b5a93bd74d3

Jul-10 14:10:12.164 [nsfs/19489]    [L1] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /my-bucket search  headers { Host: 'localhost:6443', 'Accept-Encoding': 'identity', 'User-Agent': 'aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb', 'X-Amz-Date': '20240710T111012Z', 'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', Authorization: 'AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a573dbd5c184b8dd48ee28eab7faa21509d67c4551c2fc6e53d91e7e96042d0b', 'Content-Length': '0' } region us-east-1 canonical_str
PUT
/my-bucket

host:localhost:6443
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240710T111012Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 string_to_sign
AWS4-HMAC-SHA256
20240710T111012Z
20240710/us-east-1/s3/aws4_request
b97298d956a823e54914b1cb7820dcefb4dd175ffd4d96c6f11d9b5a93bd74d3

2024-07-10 14:10:12.165116 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Readfile _path=/etc/noobaa.conf.d/access_keys/Dwertyuiopasdfg11001.symlink
2024-07-10 14:10:12.165155 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Readfile _path=/etc/noobaa.conf.d/access_keys/Dwertyuiopasdfg11001.symlink _uid=0 _gid=0 _backend=
2024-07-10 14:10:12.165179 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Readfile _path=/etc/noobaa.conf.d/access_keys/Dwertyuiopasdfg11001.symlink _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 14:10:12.165428 [PID-19489/TID-10499] [L1] FS::FSWorker::Execute: Readfile _path=/etc/noobaa.conf.d/access_keys/Dwertyuiopasdfg11001.symlink  took: 0.220542 ms
2024-07-10 14:10:12.165457 [PID-19489/TID-259] [L1] FS::FSWorker::OnOK: Readfile _path=/etc/noobaa.conf.d/access_keys/Dwertyuiopasdfg11001.symlink
2024-07-10 14:10:12.165751 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Stat _path=/etc/noobaa.conf.d/master_keys.json
2024-07-10 14:10:12.165781 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/master_keys.json _uid=0 _gid=0 _backend=
2024-07-10 14:10:12.165801 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/master_keys.json _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 14:10:12.165874 [PID-19489/TID-8707] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/master_keys.json  took: 0.046667 ms
2024-07-10 14:10:12.165898 [PID-19489/TID-259] [L1] FS::Stat::OnOK: _path=/etc/noobaa.conf.d/master_keys.json _stat_res.st_ino=136917351 _stat_res.st_size=288
Jul-10 14:10:12.167 [nsfs/19489]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.create_bucket
                requesting_account={ _id: '668e67a37162b7b11fe8c2e6', name: SENSITIVE-34617e85c8ee3957, email: SENSITIVE-34617e85c8ee3957, creation_date: '2024-07-10T10:51:15.038Z', access_keys: [ { access_key: SENSITIVE-d52ff789eb6f71e8, secret_key: SENSITIVE-c60ed951c4165069 } ], nsfs_account_config: { uid: 1001, gid: 1001, new_buckets_path: '/tmp/nsfs_root1', warn_threshold_ms: 100, backend: undefined }, allow_bucket_creation: true, master_key_id: '668e67a3f692d84ab061636b' },
                bucket_config_path=/etc/noobaa.conf.d/buckets/my-bucket.json,
                bucket_storage_path=/tmp/nsfs_root1/my-bucket
Jul-10 14:10:12.167 [nsfs/19489]    [L2] core.sdk.bucketspace_fs:: create_bucket: bucket properties before validate_bucket_schema { _id: '668e6c140f8c9f2955abfd1d', name: 'my-bucket', owner_account: '668e67a37162b7b11fe8c2e6', system_owner: 'shira-1001', bucket_owner: 'shira-1001', versioning: 'DISABLED', creation_date: '2024-07-10T11:10:12.167Z', path: '/tmp/nsfs_root1/my-bucket', should_create_underlying_storage: true }
2024-07-10 14:10:12.169192 [PID-19489/TID-259] [L1] FS::FSWorker::Begin: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json
2024-07-10 14:10:12.169215 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 _backend=
2024-07-10 14:10:12.169237 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json _uid=0 _gid=0 geteuid()=0 getegid()=0 getuid()=0 getgid()=0
2024-07-10 14:10:12.169314 [PID-19489/TID-8195] [L1] FS::FSWorker::Execute: Stat _path=/etc/noobaa.conf.d/buckets/my-bucket.json  took: 0.050791 ms
2024-07-10 14:10:12.169691 [PID-19489/TID-259] [L1] FS::Stat::OnOK: _path=/etc/noobaa.conf.d/buckets/my-bucket.json _stat_res.st_ino=136918888 _stat_res.st_size=294
Jul-10 14:10:12.170 [nsfs/19489] [ERROR] core.util.native_fs_utils:: create_config_file:: error Error: configuration file already exists
    at create_config_file (/Users/shiradymnik/SourceCode/noobaa-core/src/util/native_fs_utils.js:261:25)
    at async /Users/shiradymnik/SourceCode/noobaa-core/src/sdk/bucketspace_fs.js:297:17
    at async Semaphore.surround (/Users/shiradymnik/SourceCode/noobaa-core/src/util/semaphore.js:71:84)
    at async KeysSemaphore.surround_key (/Users/shiradymnik/SourceCode/noobaa-core/src/util/keys_semaphore.js:28:20)
    at async NsfsObjectSDK._call_op_and_update_stats (/Users/shiradymnik/SourceCode/noobaa-core/src/sdk/object_sdk.js:543:27)
    at async Object.put_bucket [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/ops/s3_put_bucket.js:11:5)
    at async handle_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:150:19)
    at async Object.s3_rest [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:65:9) {
  code: 'EEXIST'
}
Jul-10 14:10:12.171 [nsfs/19489] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.</Message><Resource>/my-bucket</Resource><RequestId>lyfqnnwj-1jw1ad-69k</RequestId></Error> PUT /my-bucket {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.15.36 Python/3.11.9 Darwin/23.4.0 source/arm64 prompt/off command/s3.mb","x-amz-date":"20240710T111012Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20240710/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a573dbd5c184b8dd48ee28eab7faa21509d67c4551c2fc6e53d91e7e96042d0b","content-length":"0"} Error: configuration file already exists
    at create_config_file (/Users/shiradymnik/SourceCode/noobaa-core/src/util/native_fs_utils.js:261:25)
    at async /Users/shiradymnik/SourceCode/noobaa-core/src/sdk/bucketspace_fs.js:297:17
    at async Semaphore.surround (/Users/shiradymnik/SourceCode/noobaa-core/src/util/semaphore.js:71:84)
    at async KeysSemaphore.surround_key (/Users/shiradymnik/SourceCode/noobaa-core/src/util/keys_semaphore.js:28:20)
    at async NsfsObjectSDK._call_op_and_update_stats (/Users/shiradymnik/SourceCode/noobaa-core/src/sdk/object_sdk.js:543:27)
    at async Object.put_bucket [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/ops/s3_put_bucket.js:11:5)
    at async handle_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:150:19)
    at async Object.s3_rest [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:65:9)
shirady commented 3 months ago

closing - I don't think we have a better solution for this scenario.

The options that were considered:

  1. Use the path that we see as the bucket and create the bucket (instead of creating the directory) - it might be dangerous if the user did not intend.
  2. Change the error to give more details - the only error to which we can add additional detail is InternalError and it is less user-friendly than the original solution (although we could attach to it additional details). By changing the message for this specific error (BucketAlreadyExists) we will pay for it by being not compatible with the S3 error message.