Lack of application error handling at the fuse level

[kittyshell]

There is lack of application error handling at the fuse level. As a result, there is a mismatch between the data in the bucket and in at the fuse level. How to reproduce.

1. Mount bucket to the directory.
2. Execute the ‘mv’ command to rename the file (object in the mounted bucket).
3. The command is executed instantly and changes occur only in memory. In the basket, the object remains in the same state. It was checked using s3cmd.

In the first screenshot, the 'mv' command is executed and after that the listing is made.

The listing performed by the second alternative client shows that the name changes were not applied at the S3 bucket and notification about error was is not recieved on operation system level.

[vitalif]

Hi, it's explicitly stated in the README and it's implemented this way due to the performance reasons. S3 services are relatively slow (high-latency), so we have to utilize parallelism and asynchrony to achieve high performance of copying small files and so on. Use fsync() if you want to guarantee that changes are flushed to the server.

[crioman]

Hi there. Got some logs for this issue. Trying to mv test.img to test.img2

-----------------------------------------------------
2024/01/11 18:35:40.303966 s3.DEBUG DEBUG: Validate Response s3/UploadPartCopy failed, attempt 0/3, error PreconditionFailed: At least one of the preconditions you specified did not hold.
        status code: 412, request id: txc2e7e4da5d354bee9d500-0065a034fc, host id: txc2e7e4da5d354bee9d500-0065a034fc
2024/01/11 18:35:40.304005 s3.WARNING UploadPartCopy {
  Bucket: "test",
  CopySource: "test/test.img",
  CopySourceIfMatch: "\"a7c5c4fbaa5148f6f527183ebb1ebf7c-410\"",
  CopySourceRange: "bytes=2097152000-2147483647",
  Key: "test.img2",
  PartNumber: 41,
  UploadId: "ZTJhZGZjN2UtNDVlNC00ODBiLWI4MzgtM2Y1YjdlODY2Y2Nk"
} = PreconditionFailed: At least one of the preconditions you specified did not hold.
        status code: 412, request id: txc2e7e4da5d354bee9d500-0065a034fc, host id: txc2e7e4da5d354bee9d500-0065a034fc
2024/01/11 18:35:40.304043 s3.WARNING http=412 At least one of the preconditions you specified did not hold. s3=PreconditionFailed request=tx639dbfee064d4a8795a4e-0065a034fb

2024/01/11 18:35:40.304064 main.WARNING Failed to copy test.img to test.img2 (rename): PreconditionFailed: At least one of the preconditions you specified did not hold.
        status code: 412, request id: tx639dbfee064d4a8795a4e-0065a034fb, host id: tx639dbfee064d4a8795a4e-0065a034fb

[crioman]

Further investigation showed that problem in --multipart-copy-threshold It is 128M by default. So 129M file got the same 412 from swift. Increasing this parameter to 5120M (5G) allowed me to mv files less then 5G

[nixargh]

Further investigation showed that problem in --multipart-copy-threshold It is 128M by default. So 129M file got the same 412 from swift. Increasing this parameter to 5120M (5G) allowed me to mv files less then 5G

Doing this you just avoid calling UploadPartCopy method of S3 API. The problem is between GeeseFS and Openstack swift. Documentation says about condition when s3 server returns 412, in brief it happens with some combination of two headers: x-amz-copy-source-if-none-match & x-amz-copy-source-if-modified-since.

Collegues of mine investigated the issue and made GeeseFS work with modern Openstack Swift by withdrawing X-Amz-Copy-Source-If-Match (I'll double check as it is not the one from the list above) header on the way to API. So the theory is that due to multithread nature of GeeseFS the value of this header is not what swift server expects to get. Regarding other S3 providers, I think they all ignores this header and everything works smooze (we checked about 4 different solutions).