Cannot mount "Input/output error"

[corbinjurgens]

libfuse 3.2.6 fuse-overlayfs 1.8.2 prebuilt x86_64 amazonlinux2

I have been trying to use fuse-overlayfs to merge two goofys s3 folders.

I succeeded using migratefs (a fork of fuse-overlayfs) but had a different issue (copyup permission issue) so I am now trying fuse-overlayfs without copyup feature.

I call the following

sudo fuse-overlayfs -f -d  -o lowerdir=/mnt/html/upload_aws -o upperdir=/mnt/html/upload_wasabi -o workdir=/mnt/html/upload_wasabi /mnt/html/upload

Result

uid=unchanged
uid=unchanged
upperdir=/mnt/html/upload_wasabi
workdir=/mnt/html/upload_wasabi
lowerdir=/mnt/html/upload_aws
mountpoint=/mnt/html/upload
plugins=<none>
fuse-overlayfs: cannot read upper dir: Input/output error

On the upper directories goofys debug, the error seems to be:

2022/07/29 01:45:52.302399 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

Full upper goofys debug log:

2022/07/29 01:45:51.437198 fuse.DEBUG Op 0x000005b8        connection.go:408] <- GetInodeAttributes (inode 1)
2022/07/29 01:45:51.437323 fuse.DEBUG GetAttributes 1  []
2022/07/29 01:45:51.437567 fuse.DEBUG Op 0x000005b8        connection.go:491] -> OK ()
2022/07/29 01:45:51.440671 fuse.DEBUG Op 0x000005ba        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.441009 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.441365 fuse.DEBUG Op 0x000005ba        connection.go:491] -> OK ()
2022/07/29 01:45:51.442442 fuse.DEBUG Op 0x000005bc        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.442570 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.442695 fuse.DEBUG Op 0x000005bc        connection.go:491] -> OK ()
2022/07/29 01:45:51.443307 fuse.DEBUG Op 0x000005be        connection.go:408] <- GetXattr (inode 1, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.444496 fuse.DEBUG GetXattr 1  [trusted.overlay.origin]
2022/07/29 01:45:51.446570 fuse.DEBUG Op 0x000005be        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.447892 fuse.DEBUG GetXattr 1  [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.447745 fuse.DEBUG Op 0x000005c0        connection.go:408] <- GetXattr (inode 1, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.448388 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.449699 s3.DEBUG DEBUG: Build Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.452055 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.453653 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.455393 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.457162 fuse.DEBUG Op 0x000005c0        connection.go:493] -> Error: "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, HeadObjectInput.Key.\n"
2022/07/29 01:45:51.457607 fuse.ERROR *fuseops.GetXattrOp error: InvalidParameter: 1 validation error(s) found.
2022/07/29 01:45:51.458606 fuse.ERROR - minimum field size of 1, HeadObjectInput.Key.
2022/07/29 01:45:51.459532 fuse.DEBUG Op 0x000005c2        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:51.459899 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:51.460563 fuse.DEBUG Op 0x000005c2        connection.go:491] -> OK ()
2022/07/29 01:45:51.464642 fuse.DEBUG Op 0x000005c4        connection.go:408] <- LookUpInode (parent 1, name ".wh.")
2022/07/29 01:45:51.464998 fuse.DEBUG Inode.LookUp 1  [.wh.]
2022/07/29 01:45:51.467044 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh. HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f55c1d0bf44c98cedb147747adcc74cb4034c772ac9cb18aa3297c533cbbfbe3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.467211 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh./ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58ef78e800210967e193c15bba038644c93fb644b9705e3e8a155a02ee8f01f0
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.467077 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58b3660c2661a1aa81fd3a938d0249998512c435046588821a19ab17267539b4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.495414 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: +XcQpncPhfHKBhFtgK81Tv4iL0yjXIQaABZJ/VE4sUYsufy7g1yzY8UZRjYojvubigQkhV9Abn6j
X-Amz-Request-Id: 3833CB436D039451

-----------------------------------------------------
2022/07/29 01:45:51.496274 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.497702 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 8TKDDODdJrJJ6QiHslBt5O8TuQwFo2BtXBpmLDx7hJ+HpU3AKiTOIvjTY9N3afzn1WmUjz1dWPeE
X-Amz-Request-Id: 073B7AF662083608

-----------------------------------------------------
2022/07/29 01:45:51.497810 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 073B7AF662083608, host id: 8TKDDODdJrJJ6QiHslBt5O8TuQwFo2BtXBpmLDx7hJ+HpU3AKiTOIvjTY9N3afzn1WmUjz1dWPeE
2022/07/29 01:45:51.498048 s3.DEBUG HEAD .wh. = no such file or directory
2022/07/29 01:45:51.507085 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: BDH2th3vZStD8FzM403iDd2lRMyJ60PO2O0nL9hiFjVjzeJy/0VrZrpL4K5zHnirr3VLw6qgOzWW
X-Amz-Request-Id: D81533D470412681

-----------------------------------------------------
2022/07/29 01:45:51.507139 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: D81533D470412681, host id: BDH2th3vZStD8FzM403iDd2lRMyJ60PO2O0nL9hiFjVjzeJy/0VrZrpL4K5zHnirr3VLw6qgOzWW
2022/07/29 01:45:51.507442 s3.DEBUG HEAD .wh./ = no such file or directory
2022/07/29 01:45:51.507500 fuse.DEBUG <-- LookUpInode 1 .wh. no such file or directory
2022/07/29 01:45:51.507630 fuse.DEBUG Op 0x000005c4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.508851 fuse.DEBUG Op 0x000005c6        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.509178 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.510257 fuse.DEBUG Op 0x000005c6        connection.go:491] -> OK ()
2022/07/29 01:45:51.510606 fuse.DEBUG Op 0x000005c8        connection.go:408] <- ReadDir (inode 1)
2022/07/29 01:45:51.511130 fuse.DEBUG ReadDir 1  [0]
2022/07/29 01:45:51.512441 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&prefix= HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=dcc9a8aa7a8c2026639c69393556d19bf03ad9badc73dc98da3691d9dc8c9cbc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.534118 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: CNczelyjbQXMnYsF5pYy4JeZFjT9ufgQe320J2oELMhVwvUiIFVorjVXce+6Qlmq2uLLJZWrStu+
X-Amz-Request-Id: D2A924220ED0D423

-----------------------------------------------------
2022/07/29 01:45:51.536087 s3.DEBUG &{[] [.test: 4 haha: 2 s3fs_exist.txt: 0 test.txt: 6 wasabi.txt: 11] <nil> false }
2022/07/29 01:45:51.537058 fuse.DEBUG <-- ReadDir 1  [. 1]
2022/07/29 01:45:51.537148 fuse.DEBUG <-- ReadDir 1  [.. 2]
2022/07/29 01:45:51.537193 fuse.DEBUG <-- ReadDir 1  [.test 3]
2022/07/29 01:45:51.538208 fuse.DEBUG <-- ReadDir 1  [haha 4]
2022/07/29 01:45:51.539033 fuse.DEBUG <-- ReadDir 1  [s3fs_exist.txt 5]
2022/07/29 01:45:51.539277 fuse.DEBUG <-- ReadDir 1  [test.txt 6]
2022/07/29 01:45:51.540223 fuse.DEBUG <-- ReadDir 1  [wasabi.txt 7]
2022/07/29 01:45:51.541040 fuse.DEBUG Op 0x000005c8        connection.go:491] -> OK ()
2022/07/29 01:45:51.541623 fuse.DEBUG Op 0x000005ca        connection.go:408] <- LookUpInode (parent 1, name ".wh..test")
2022/07/29 01:45:51.542481 fuse.DEBUG Inode.LookUp 1  [.wh..test]
2022/07/29 01:45:51.546471 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh..test HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=89b5cc9ac157da82c1205c626fc3186d40668764a54690dc28c6bedd4d9e4085
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.547414 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh..test/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4aad81f77eb972cdbe7f496b10150d2179578837dd010191a604c041d7cd888b
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.547589 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh..test%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=15f38ac59f9c9a01945e6d958c7bbfad0fbd0e3db6076c8ba60e78ac5620155c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.567059 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: 2o08ptqr2oPWp/XS2h4zq7QfOJiD897m4eEjAy9QAkOcG9l8duLF1KJ1NabG8DkH7RWufm/tKC0g
X-Amz-Request-Id: AC84DFB711E1CC3E

-----------------------------------------------------
2022/07/29 01:45:51.567147 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: AC84DFB711E1CC3E, host id: 2o08ptqr2oPWp/XS2h4zq7QfOJiD897m4eEjAy9QAkOcG9l8duLF1KJ1NabG8DkH7RWufm/tKC0g
2022/07/29 01:45:51.567394 s3.DEBUG HEAD .wh..test = no such file or directory
2022/07/29 01:45:51.570143 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: LpszYukDKZuAxZau8iQEgt1UhGEzXgqNYFe9UPtgTL1lWQ6/AiDsCsenl3XnsWd4JY9yd+1Gla+N
X-Amz-Request-Id: 0993DF431033138D

-----------------------------------------------------
2022/07/29 01:45:51.570250 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: J85m1O6vd75p7eY2tMILZ/hBubNp+81yYa9FC2H3gpN7AXgJ11Jgn9V1AY9nkPH47Or+PqRWRXyH
X-Amz-Request-Id: B984F90CBD68AE86

-----------------------------------------------------
2022/07/29 01:45:51.570320 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.570259 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 0993DF431033138D, host id: LpszYukDKZuAxZau8iQEgt1UhGEzXgqNYFe9UPtgTL1lWQ6/AiDsCsenl3XnsWd4JY9yd+1Gla+N
2022/07/29 01:45:51.570700 s3.DEBUG HEAD .wh..test/ = no such file or directory
2022/07/29 01:45:51.570961 fuse.DEBUG <-- LookUpInode 1 .wh..test no such file or directory
2022/07/29 01:45:51.571338 fuse.DEBUG Op 0x000005ca        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.572568 fuse.DEBUG Op 0x000005cc        connection.go:408] <- LookUpInode (parent 1, name ".test")
2022/07/29 01:45:51.573397 fuse.DEBUG Ref 4 .test [10]
2022/07/29 01:45:51.573539 fuse.DEBUG <-- LookUpInode 1 .test <nil>
2022/07/29 01:45:51.574375 fuse.DEBUG Op 0x000005cc        connection.go:491] -> OK (inode 4)
2022/07/29 01:45:51.575301 fuse.DEBUG Op 0x000005ce        connection.go:408] <- OpenFile (inode 4)
2022/07/29 01:45:51.575356 fuse.DEBUG OpenFile 4 .test []
2022/07/29 01:45:51.576730 fuse.DEBUG Op 0x000005ce        connection.go:491] -> OK ()
2022/07/29 01:45:51.578060 fuse.DEBUG Op 0x000005d0        connection.go:408] <- GetXattr (inode 4, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.579404 fuse.DEBUG GetXattr 4 .test [trusted.overlay.origin]
2022/07/29 01:45:51.580093 fuse.DEBUG Op 0x000005d0        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.580900 fuse.DEBUG Op 0x000005d2        connection.go:408] <- GetXattr (inode 4, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.581608 fuse.DEBUG GetXattr 4 .test [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.581776 fuse.DEBUG Op 0x000005d2        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.582274 fuse.DEBUG Op 0x000005d4        connection.go:408] <- FlushFile (inode 4)
2022/07/29 01:45:51.584109 fuse.DEBUG FlushFile 4 .test []
2022/07/29 01:45:51.585351 fuse.DEBUG <-- FlushFile 4 .test [<nil> 106 4]
2022/07/29 01:45:51.587168 fuse.DEBUG Op 0x000005d4        connection.go:491] -> OK ()
2022/07/29 01:45:51.588513 fuse.DEBUG Op 0x000005d6        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.589474 fuse.DEBUG ReleaseFileHandle .test 106 4
2022/07/29 01:45:51.590030 fuse.DEBUG Op 0x000005d6        connection.go:491] -> OK ()
2022/07/29 01:45:51.808008 fuse.DEBUG Op 0x000005d8        connection.go:408] <- LookUpInode (parent 1, name ".wh.haha")
2022/07/29 01:45:51.808099 fuse.DEBUG Inode.LookUp 1  [.wh.haha]
2022/07/29 01:45:51.811189 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.haha HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a53161a789e2a10d02bbc54a813c9892151e04d4859d2d490b009c9f7b597515
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.812090 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.haha%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bc79b3d45270aca40e6f2c565aceff841e1400684839961c0993119aa2b43a8c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.814377 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.haha/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=5ffc2abc3d8ece44a4c18bb874f5efcfbfeed724b15ac6d75054d27421ea41c2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.830757 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Id-2: zTvkZYwE2Ei02Gya6/v7MBoIv+uYXcQXypKcecdDseH7r8Eo0FTm9lVPixb4yPBGCNHC8rGHQ6RH
X-Amz-Request-Id: 9466DDFEF4181B5D

-----------------------------------------------------
2022/07/29 01:45:51.831003 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 9466DDFEF4181B5D, host id: zTvkZYwE2Ei02Gya6/v7MBoIv+uYXcQXypKcecdDseH7r8Eo0FTm9lVPixb4yPBGCNHC8rGHQ6RH
2022/07/29 01:45:51.831079 s3.DEBUG HEAD .wh.haha = no such file or directory
2022/07/29 01:45:51.840213 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: /qnDWFrhfynWMRHqSRPQ6bxRNxZbnPZ8272lu46jlbZSSCdaSgw/6OYplW2tjoQOtSnyFjiJmtz6
X-Amz-Request-Id: FD4AB376B3AF1318

-----------------------------------------------------
2022/07/29 01:45:51.840297 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 5amz28o40UMvC7F1n21CZBNInhcrfQ6GBQ5v3iMAzgjroqOkw2ztSYb4mw/9eeZ605U8LkNu0wyS
X-Amz-Request-Id: B19CA80E804AF06B

-----------------------------------------------------
2022/07/29 01:45:51.840337 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.840363 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: B19CA80E804AF06B, host id: 5amz28o40UMvC7F1n21CZBNInhcrfQ6GBQ5v3iMAzgjroqOkw2ztSYb4mw/9eeZ605U8LkNu0wyS
2022/07/29 01:45:51.840399 s3.DEBUG HEAD .wh.haha/ = no such file or directory
2022/07/29 01:45:51.840523 fuse.DEBUG <-- LookUpInode 1 .wh.haha no such file or directory
2022/07/29 01:45:51.840739 fuse.DEBUG Op 0x000005d8        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.841878 fuse.DEBUG Op 0x000005da        connection.go:408] <- OpenFile (inode 5)
2022/07/29 01:45:51.842053 fuse.DEBUG OpenFile 5 haha []
2022/07/29 01:45:51.842712 fuse.DEBUG Op 0x000005da        connection.go:491] -> OK ()
2022/07/29 01:45:51.843597 fuse.DEBUG Op 0x000005dc        connection.go:408] <- GetXattr (inode 5, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.844040 fuse.DEBUG GetXattr 5 haha [trusted.overlay.origin]
2022/07/29 01:45:51.844340 fuse.DEBUG Op 0x000005dc        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.845838 fuse.DEBUG Op 0x000005de        connection.go:408] <- GetXattr (inode 5, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.846167 fuse.DEBUG GetXattr 5 haha [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.846494 fuse.DEBUG Op 0x000005de        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.848572 fuse.DEBUG Op 0x000005e0        connection.go:408] <- FlushFile (inode 5)
2022/07/29 01:45:51.852183 fuse.DEBUG FlushFile 5 haha []
2022/07/29 01:45:51.852389 fuse.DEBUG <-- FlushFile 5 haha [<nil> 107 5]
2022/07/29 01:45:51.853049 fuse.DEBUG Op 0x000005e0        connection.go:491] -> OK ()
2022/07/29 01:45:51.859218 fuse.DEBUG Op 0x000005e2        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.859769 fuse.DEBUG ReleaseFileHandle haha 107 5
2022/07/29 01:45:51.860643 fuse.DEBUG Op 0x000005e2        connection.go:491] -> OK ()
2022/07/29 01:45:51.920701 fuse.DEBUG Op 0x000005e4        connection.go:408] <- LookUpInode (parent 1, name ".wh.s3fs_exist.txt")
2022/07/29 01:45:51.921255 fuse.DEBUG Inode.LookUp 1  [.wh.s3fs_exist.txt]
2022/07/29 01:45:51.922328 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.s3fs_exist.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2f4c453b4de05443a21c9b31075f358205aa59e5c9c7c387466646e3a7a09db2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.923417 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.s3fs_exist.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58d11bec31bb6b766d8232728fa2f1faf58690c993af06fa18db0124f5d16e5d
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.923771 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.s3fs_exist.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3df4bf319abf6e55b325c29689362923bbb97207bceb270feac233460af3cdd4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z

-----------------------------------------------------
2022/07/29 01:45:51.946318 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: jeaAs7HMTXglfG9pjWkom1t468q7MENevxdoPJq33suur251hV/oqdvfhE6es4QLOazdlpGgGWzU
X-Amz-Request-Id: 9CCB1755A711AF40

-----------------------------------------------------
2022/07/29 01:45:51.946397 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 9UVjPe26Q+wISRFFEn6X7kJ709n4IOEBwBR6Q087I2irgUiDd40q290bTR3egkBUJCr3WWjnIPNH
X-Amz-Request-Id: 08749EF136495B19

-----------------------------------------------------
2022/07/29 01:45:51.946450 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 9CCB1755A711AF40, host id: jeaAs7HMTXglfG9pjWkom1t468q7MENevxdoPJq33suur251hV/oqdvfhE6es4QLOazdlpGgGWzU
2022/07/29 01:45:51.946466 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 08749EF136495B19, host id: 9UVjPe26Q+wISRFFEn6X7kJ709n4IOEBwBR6Q087I2irgUiDd40q290bTR3egkBUJCr3WWjnIPNH
2022/07/29 01:45:51.946529 s3.DEBUG HEAD .wh.s3fs_exist.txt/ = no such file or directory
2022/07/29 01:45:51.946632 s3.DEBUG HEAD .wh.s3fs_exist.txt = no such file or directory
2022/07/29 01:45:51.947630 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: ThWofjROPq6WoL8yFmEWKZ6Wcplf1XFJxfYCybRkQbV9W1a9VFDWuCH54WfmiT58B3XufOzSvUwQ
X-Amz-Request-Id: 5E26FDE30355C42C

-----------------------------------------------------
2022/07/29 01:45:51.947724 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.947864 fuse.DEBUG <-- LookUpInode 1 .wh.s3fs_exist.txt no such file or directory
2022/07/29 01:45:51.948078 fuse.DEBUG Op 0x000005e4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.949042 fuse.DEBUG Op 0x000005e6        connection.go:408] <- OpenFile (inode 6)
2022/07/29 01:45:51.949251 fuse.DEBUG OpenFile 6 s3fs_exist.txt []
2022/07/29 01:45:51.950152 fuse.DEBUG Op 0x000005e6        connection.go:491] -> OK ()
2022/07/29 01:45:51.950514 fuse.DEBUG Op 0x000005e8        connection.go:408] <- GetXattr (inode 6, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.951392 fuse.DEBUG GetXattr 6 s3fs_exist.txt [trusted.overlay.origin]
2022/07/29 01:45:51.951507 fuse.DEBUG Op 0x000005e8        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.953028 fuse.DEBUG Op 0x000005ea        connection.go:408] <- GetXattr (inode 6, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.953194 fuse.DEBUG GetXattr 6 s3fs_exist.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.954655 fuse.DEBUG Op 0x000005ea        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.955234 fuse.DEBUG Op 0x000005ec        connection.go:408] <- FlushFile (inode 6)
2022/07/29 01:45:51.956098 fuse.DEBUG FlushFile 6 s3fs_exist.txt []
2022/07/29 01:45:51.957088 fuse.DEBUG <-- FlushFile 6 s3fs_exist.txt [<nil> 108 6]
2022/07/29 01:45:51.957627 fuse.DEBUG Op 0x000005ec        connection.go:491] -> OK ()
2022/07/29 01:45:51.958085 fuse.DEBUG Op 0x000005ee        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.958241 fuse.DEBUG ReleaseFileHandle s3fs_exist.txt 108 6
2022/07/29 01:45:51.958309 fuse.DEBUG Op 0x000005ee        connection.go:491] -> OK ()
2022/07/29 01:45:52.033627 fuse.DEBUG Op 0x000005f0        connection.go:408] <- LookUpInode (parent 1, name ".wh.test.txt")
2022/07/29 01:45:52.034377 fuse.DEBUG Inode.LookUp 1  [.wh.test.txt]
2022/07/29 01:45:52.037651 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.test.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7bd9b9d9fff3d07e79178bbd8e20b11bba02f5dba7251921283ee0d6a936c51f
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.039428 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.test.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b093f5b1f460dfd6e842cef244207e39dd14be5b8486e12b7560793453a1d618
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.037775 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.test.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7d7b32cc46b5d4874722633b07ce9c8484ccab3b94a7f45487963e39cb68a137
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.068083 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: kUHYNm5p+BbZborrPncv1anlBLu3VuzvjoVib7ZrIBe/vFxTeC4cZLNdEIgTz7OxkHn6JD1EB25S
X-Amz-Request-Id: 8A50D08FB2138805

-----------------------------------------------------
2022/07/29 01:45:52.068361 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:52.068674 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: oT2/2rcUut7XLn7Arx2l7Y7sSpsn8XTajo96yBX2Qqx3AOSK5l1TlvyU2ErxCO2WC2Y5p8NdVnSV
X-Amz-Request-Id: 0B1D33662AC1C5DE

-----------------------------------------------------
2022/07/29 01:45:52.069099 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 0B1D33662AC1C5DE, host id: oT2/2rcUut7XLn7Arx2l7Y7sSpsn8XTajo96yBX2Qqx3AOSK5l1TlvyU2ErxCO2WC2Y5p8NdVnSV
2022/07/29 01:45:52.069327 s3.DEBUG HEAD .wh.test.txt = no such file or directory
2022/07/29 01:45:52.072286 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: e1SJ9AGKrKsy3JOLPMZdZMmW8zl7ZZgENnxtgPT41YZs3Xjo3S6moKOvE/Ck/iuR6vwgwLPa+lC5
X-Amz-Request-Id: 1F1BEAE88758367B

-----------------------------------------------------
2022/07/29 01:45:52.072411 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 1F1BEAE88758367B, host id: e1SJ9AGKrKsy3JOLPMZdZMmW8zl7ZZgENnxtgPT41YZs3Xjo3S6moKOvE/Ck/iuR6vwgwLPa+lC5
2022/07/29 01:45:52.072507 s3.DEBUG HEAD .wh.test.txt/ = no such file or directory
2022/07/29 01:45:52.072547 fuse.DEBUG <-- LookUpInode 1 .wh.test.txt no such file or directory
2022/07/29 01:45:52.073505 fuse.DEBUG Op 0x000005f0        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:52.074021 fuse.DEBUG Op 0x000005f2        connection.go:408] <- LookUpInode (parent 1, name "test.txt")
2022/07/29 01:45:52.074387 fuse.DEBUG Ref 7 test.txt [10]
2022/07/29 01:45:52.075800 fuse.DEBUG <-- LookUpInode 1 test.txt <nil>
2022/07/29 01:45:52.076003 fuse.DEBUG Op 0x000005f2        connection.go:491] -> OK (inode 7)
2022/07/29 01:45:52.076649 fuse.DEBUG Op 0x000005f4        connection.go:408] <- OpenFile (inode 7)
2022/07/29 01:45:52.077008 fuse.DEBUG OpenFile 7 test.txt []
2022/07/29 01:45:52.078126 fuse.DEBUG Op 0x000005f4        connection.go:491] -> OK ()
2022/07/29 01:45:52.079043 fuse.DEBUG Op 0x000005f6        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:52.079204 fuse.DEBUG GetXattr 7 test.txt [trusted.overlay.origin]
2022/07/29 01:45:52.079292 fuse.DEBUG Op 0x000005f6        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.079641 fuse.DEBUG Op 0x000005f8        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:52.080601 fuse.DEBUG GetXattr 7 test.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:52.080818 fuse.DEBUG Op 0x000005f8        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.081528 fuse.DEBUG Op 0x000005fa        connection.go:408] <- FlushFile (inode 7)
2022/07/29 01:45:52.082721 fuse.DEBUG FlushFile 7 test.txt []
2022/07/29 01:45:52.083034 fuse.DEBUG <-- FlushFile 7 test.txt [<nil> 109 7]
2022/07/29 01:45:52.084312 fuse.DEBUG Op 0x000005fa        connection.go:491] -> OK ()
2022/07/29 01:45:52.085691 fuse.DEBUG Op 0x000005fc        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:52.085901 fuse.DEBUG ReleaseFileHandle test.txt 109 7
2022/07/29 01:45:52.086050 fuse.DEBUG Op 0x000005fc        connection.go:491] -> OK ()
2022/07/29 01:45:52.148247 fuse.DEBUG Op 0x000005fe        connection.go:408] <- LookUpInode (parent 1, name ".wh.wasabi.txt")
2022/07/29 01:45:52.148605 fuse.DEBUG Inode.LookUp 1  [.wh.wasabi.txt]
2022/07/29 01:45:52.150579 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.wasabi.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=898dc42e06044a270000fc1ae8499bb0c12a9f42855e2cc7a7478d6e409a0703
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.150636 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.wasabi.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=90f2caa96ba12a73751e4ecd2bf22e02aaa450f2947b9a7f46196844a8f4598a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.150782 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.wasabi.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bce7abb2794d937b78750472b882d0eba2918043f61db5a731970c95d29d62de
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z

-----------------------------------------------------
2022/07/29 01:45:52.171246 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: o3jsQfpnL/8mRYbQ8S/6bSui8gHDEJxWpw5BxBlDAtx+UaYz3IegbOCGJdVBmYUNIfeYZ5Gd/jZy
X-Amz-Request-Id: B662050F4099F5BE

-----------------------------------------------------
2022/07/29 01:45:52.171312 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: B662050F4099F5BE, host id: o3jsQfpnL/8mRYbQ8S/6bSui8gHDEJxWpw5BxBlDAtx+UaYz3IegbOCGJdVBmYUNIfeYZ5Gd/jZy
2022/07/29 01:45:52.171409 s3.DEBUG HEAD .wh.wasabi.txt/ = no such file or directory
2022/07/29 01:45:52.175005 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: 7Ttw9LjYUGmXMm3Q2ZGowVE2qytY4eclaD6esqB+fzqMCuIEXwEoz/qWeWvR2kNrDShA4QWBt+r3
X-Amz-Request-Id: E4FD9FEB56CA7874

-----------------------------------------------------
2022/07/29 01:45:52.175562 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:52.211338 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: l/drxGJvFfCRhzdnj7Wly8ZyQs95kl6emwu3byBGcugOoxFMk/0hTatIMPc9BvC7z+Ym913sd8Tj
X-Amz-Request-Id: 710A9CCB1C33F10C

-----------------------------------------------------
2022/07/29 01:45:52.211474 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 710A9CCB1C33F10C, host id: l/drxGJvFfCRhzdnj7Wly8ZyQs95kl6emwu3byBGcugOoxFMk/0hTatIMPc9BvC7z+Ym913sd8Tj
2022/07/29 01:45:52.211527 s3.DEBUG HEAD .wh.wasabi.txt = no such file or directory
2022/07/29 01:45:52.211559 fuse.DEBUG <-- LookUpInode 1 .wh.wasabi.txt no such file or directory
2022/07/29 01:45:52.212998 fuse.DEBUG Op 0x000005fe        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:52.213220 fuse.DEBUG Op 0x00000600        connection.go:408] <- OpenFile (inode 8)
2022/07/29 01:45:52.213358 fuse.DEBUG OpenFile 8 wasabi.txt []
2022/07/29 01:45:52.214050 fuse.DEBUG Op 0x00000600        connection.go:491] -> OK ()
2022/07/29 01:45:52.214475 fuse.DEBUG Op 0x00000602        connection.go:408] <- GetXattr (inode 8, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:52.215072 fuse.DEBUG GetXattr 8 wasabi.txt [trusted.overlay.origin]
2022/07/29 01:45:52.215288 fuse.DEBUG Op 0x00000602        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.215855 fuse.DEBUG Op 0x00000604        connection.go:408] <- GetXattr (inode 8, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:52.216184 fuse.DEBUG GetXattr 8 wasabi.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:52.217218 fuse.DEBUG Op 0x00000604        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.218428 fuse.DEBUG Op 0x00000606        connection.go:408] <- FlushFile (inode 8)
2022/07/29 01:45:52.219100 fuse.DEBUG FlushFile 8 wasabi.txt []
2022/07/29 01:45:52.219242 fuse.DEBUG <-- FlushFile 8 wasabi.txt [<nil> 110 8]
2022/07/29 01:45:52.219425 fuse.DEBUG Op 0x00000606        connection.go:491] -> OK ()
2022/07/29 01:45:52.221339 fuse.DEBUG Op 0x00000608        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:52.221744 fuse.DEBUG ReleaseFileHandle wasabi.txt 110 8
2022/07/29 01:45:52.222356 fuse.DEBUG Op 0x00000608        connection.go:491] -> OK ()
2022/07/29 01:45:52.293515 fuse.DEBUG Op 0x0000060a        connection.go:408] <- ReadDir (inode 1)
2022/07/29 01:45:52.293733 fuse.DEBUG ReadDir 1  [7]
2022/07/29 01:45:52.294060 fuse.DEBUG Op 0x0000060a        connection.go:491] -> OK ()
2022/07/29 01:45:52.294882 fuse.DEBUG Op 0x0000060c        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:52.295223 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:52.295596 fuse.DEBUG Op 0x0000060c        connection.go:491] -> OK ()
2022/07/29 01:45:52.296401 fuse.DEBUG Op 0x0000060e        connection.go:408] <- GetXattr (inode 1, name "trusted.overlay.opaque", name trusted.overlay.opaque)
2022/07/29 01:45:52.297256 fuse.DEBUG GetXattr 1  [trusted.overlay.opaque]
2022/07/29 01:45:52.299099 fuse.DEBUG Op 0x0000060e        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.299594 fuse.DEBUG Op 0x00000610        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.299774 fuse.DEBUG Op 0x00000612        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:52.300041 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:52.300785 fuse.DEBUG Op 0x00000612        connection.go:491] -> OK ()
2022/07/29 01:45:52.301137 fuse.DEBUG Op 0x00000614        connection.go:408] <- GetXattr (inode 1, name "user.fuseoverlayfs.opaque", name user.fuseoverlayfs.opaque)
2022/07/29 01:45:52.301543 fuse.DEBUG GetXattr 1  [user.fuseoverlayfs.opaque]
2022/07/29 01:45:52.302702 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.302399 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303376 s3.DEBUG DEBUG: Build Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303411 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303499 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303550 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.305516 fuse.DEBUG Op 0x00000610        connection.go:491] -> OK ()
2022/07/29 01:45:52.306354 fuse.DEBUG Op 0x00000614        connection.go:493] -> Error: "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, HeadObjectInput.Key.\n"
2022/07/29 01:45:52.307036 fuse.ERROR *fuseops.GetXattrOp error: InvalidParameter: 1 validation error(s) found.
2022/07/29 01:45:52.308539 fuse.ERROR - minimum field size of 1, HeadObjectInput.Key.
2022/07/29 01:45:52.308241 fuse.DEBUG Op 0x00000616        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.308382 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.309270 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.309715 fuse.DEBUG Op 0x00000618        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.310812 fuse.DEBUG Op 0x0000061a        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.310812 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.311243 fuse.DEBUG Op 0x00000616        connection.go:491] -> OK ()
2022/07/29 01:45:52.311317 fuse.DEBUG Op 0x0000061a        connection.go:491] -> OK ()
2022/07/29 01:45:52.311676 fuse.DEBUG Op 0x00000618        connection.go:491] -> OK ()

Any ideas would be appreciated" Thank you

[giuseppe]

workdir and upperdir must be two different directories, but they must be on the same file system

[corbinjurgens]

Thank you! That is one thing I was unsure about. When you say same file system, do you mean it can be a entirely different goofys s3 mount location (as long as its still a fuse s3 mount), or should be the same disk (meaning i would need to make two folders on a disk, one for data and one for workdir)?

I had tried making workdir to something like workdir=/mnt/html/upload_wasabi/overlay but that doesn't work either.

Also, in my particular use case where these same disks are mounted on many machines, I would want to make sure workdir contents are shared correct?

[corbinjurgens]

I tried

sudo fuse-overlayfs -f -d  -o lowerdir=/mnt/html/upload_aws -o upperdir=/mnt/html/upload_wasabi/data -o workdir=/mnt/html/upload_wasabi/overlay /mnt/html/upload

uid=unchanged
uid=unchanged
upperdir=/mnt/html/upload_wasabi/data
workdir=/mnt/html/upload_wasabi/overlay
lowerdir=/mnt/html/upload_aws
mountpoint=/mnt/html/upload
plugins=<none>
fuse-overlayfs: cannot read upper dir: Input/output error

But still same error.

Full goofys log

2022/07/29 17:46:01.219660 fuse.DEBUG Op 0x000000d2        connection.go:408] <- GetInodeAttributes (inode 1)
2022/07/29 17:46:01.219953 fuse.DEBUG GetAttributes 1  []
2022/07/29 17:46:01.220819 fuse.DEBUG Op 0x000000d2        connection.go:491] -> OK ()
2022/07/29 17:46:01.222007 fuse.DEBUG Op 0x000000d4        connection.go:408] <- LookUpInode (parent 1, name "data")
2022/07/29 17:46:01.222556 fuse.DEBUG Ref 7 data [2]
2022/07/29 17:46:01.222773 fuse.DEBUG lookup expired 7 data []
2022/07/29 17:46:01.222806 fuse.DEBUG Inode.LookUp 1  [data]
2022/07/29 17:46:01.223650 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7be1a2a276cfcb383040d6c77546fffd62640ccba1be09bf9aeb1193d727a99d
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.223938 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ac560fdc31d6bcecb1dcf6cf3bb47315b2e0dabdcd74b29db0a64ff55d873939
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.225187 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7c6e50c17c0bb189b49d035e9457612ba0d5912a6efc26327c21ab7d9876c7fd
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.936117 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: mLm3MHEtwJqQxV+dNYRqtzi56H+VLZZ80B9e8oKMPFPrURRyPPRnod6wcYMT5YwPLNSpmiLunSm4
X-Amz-Request-Id: 5D87A5C31251D28D

-----------------------------------------------------
2022/07/29 17:46:01.936191 s3.DEBUG &{[] [data/: 0] 0xc0002c0388 true }
2022/07/29 17:46:01.936531 fuse.DEBUG invalidate cache because attributes changed 7 data [{4096 2022-07-29 08:43:31 +0000 UTC} {4096 0001-01-01 00:00:00 +0000 UTC}]
2022/07/29 17:46:01.936813 fuse.DEBUG <-- LookUpInode 1 data <nil>
2022/07/29 17:46:01.937402 fuse.DEBUG Op 0x000000d4        connection.go:491] -> OK (inode 7)
2022/07/29 17:46:01.942743 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Fri, 29 Jul 2022 08:46:02 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Fri, 29 Jul 2022 08:43:16 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: GIyLMoxKyrMp+o2OAOxOqpPiI37UYSN895bZXACFaGBcGWCYm+/3W7iYwb1XgpVK1jz+BaAzEt9I
X-Amz-Request-Id: 9E1E57B996BDD12C
Content-Length: 0

-----------------------------------------------------
2022/07/29 17:46:01.942873 s3.DEBUG data/: 0
2022/07/29 17:46:01.943797 fuse.DEBUG Op 0x000000d6        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.944254 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.947118 fuse.DEBUG Op 0x000000d6        connection.go:491] -> OK ()
2022/07/29 17:46:01.947741 fuse.DEBUG Op 0x000000d8        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.948159 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.948534 fuse.DEBUG Op 0x000000d8        connection.go:491] -> OK ()
2022/07/29 17:46:01.949749 fuse.DEBUG Op 0x000000da        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:01.950377 fuse.DEBUG GetXattr 7 data [trusted.overlay.origin]
2022/07/29 17:46:01.950780 fuse.DEBUG Op 0x000000da        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:01.951910 fuse.DEBUG Op 0x000000dc        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:01.952191 fuse.DEBUG GetXattr 7 data [user.fuseoverlayfs.origin]
2022/07/29 17:46:01.952416 fuse.DEBUG Op 0x000000dc        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:01.953248 fuse.DEBUG Op 0x000000de        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:01.953757 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:01.954239 fuse.DEBUG Op 0x000000de        connection.go:491] -> OK ()
2022/07/29 17:46:01.956795 fuse.DEBUG Op 0x000000e0        connection.go:408] <- LookUpInode (parent 7, name ".wh.")
2022/07/29 17:46:01.957716 fuse.DEBUG Inode.LookUp 7 data [.wh.]
2022/07/29 17:46:01.958970 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh. HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fa8423dc364c1142e3e367f701aa4678810233dada6d427bf5a719331c448417
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.959832 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh./ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d5c8c0ca2643f275feb27be461c9acdd0ae99f3059ca968ec33f9c2418352998
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.960422 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Id-2: BTqVNNgbeW1rIvoHZvN6ThWu0JdBXvKWOrIYRrSXGgp+tjNuhhE4B/xwgp+aNCOyrcYLx7/qQPOV
X-Amz-Request-Id: DBBF63656B04E180

-----------------------------------------------------
2022/07/29 17:46:01.960791 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: DBBF63656B04E180, host id: BTqVNNgbeW1rIvoHZvN6ThWu0JdBXvKWOrIYRrSXGgp+tjNuhhE4B/xwgp+aNCOyrcYLx7/qQPOV
2022/07/29 17:46:01.962197 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=90285c9b0b15cdf07382b72b3ff6ced703098e92f3762dadbf5974a8b7ad1db3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:01.983259 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: RzBuNOnrbWj0d90gcWm0zQuj5QvYdaw/OCfbxmNznNWoM2K7z/DxumFvP0J9cR1T+lngozbjWp/a
X-Amz-Request-Id: 9C2CF3986FB0E938

-----------------------------------------------------
2022/07/29 17:46:01.983326 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 9C2CF3986FB0E938, host id: RzBuNOnrbWj0d90gcWm0zQuj5QvYdaw/OCfbxmNznNWoM2K7z/DxumFvP0J9cR1T+lngozbjWp/a
2022/07/29 17:46:01.983855 s3.DEBUG HEAD data/.wh. = no such file or directory
2022/07/29 17:46:01.987219 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: I19h+kTzy3WbF/3ISma+/myGtmUnP9Oj0fYQWpxr9IMQ5EXsi0vIWFwBLjJtvxHlMNiWm0bZ+GW2
X-Amz-Request-Id: 79F6CFC5796AE57B

-----------------------------------------------------
2022/07/29 17:46:01.987281 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 79F6CFC5796AE57B, host id: I19h+kTzy3WbF/3ISma+/myGtmUnP9Oj0fYQWpxr9IMQ5EXsi0vIWFwBLjJtvxHlMNiWm0bZ+GW2
2022/07/29 17:46:01.987380 s3.DEBUG HEAD data/.wh./ = no such file or directory
2022/07/29 17:46:01.987861 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: MausWABbFee4cpL/xd9AQk+D2ifnehEj+MPNfdAmcZI3Oan8g1WUAY0dQWFQsOaJ/yKh1BUnF6BB
X-Amz-Request-Id: 3F282589FCBCAA17

-----------------------------------------------------
2022/07/29 17:46:01.988769 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:01.988927 fuse.DEBUG <-- LookUpInode 7 .wh. no such file or directory
2022/07/29 17:46:01.989321 fuse.DEBUG Op 0x000000e0        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:01.990229 fuse.DEBUG Op 0x000000e2        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.991152 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.991981 fuse.DEBUG Op 0x000000e2        connection.go:491] -> OK ()
2022/07/29 17:46:01.993630 fuse.DEBUG ReadDir 7 data [0]
2022/07/29 17:46:01.993354 fuse.DEBUG Op 0x000000e4        connection.go:408] <- ReadDir (inode 7)
2022/07/29 17:46:01.994741 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&prefix=data%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=951ec57ba0654355286846c1e5078022654d2729577742daafc6a26f8498feab
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z

-----------------------------------------------------
2022/07/29 17:46:02.017300 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: Qf7g/B/leLEP+FfTUeQJVDP19gOkriuUVxW9MzKK89qYSjvFBB7brmkgnIzyaGFKkC4bLVR3iJZ+
X-Amz-Request-Id: 64DC7AE6FA417D90

-----------------------------------------------------
2022/07/29 17:46:02.017482 s3.DEBUG &{[] [data/: 0 data/haha: 2 data/s3fs_exist.txt: 0 data/test.txt: 6 data/wasabi.txt: 11] <nil> false }
2022/07/29 17:46:02.017844 fuse.DEBUG <-- ReadDir 7 data [. 1]
2022/07/29 17:46:02.017915 fuse.DEBUG <-- ReadDir 7 data [.. 2]
2022/07/29 17:46:02.018034 fuse.DEBUG <-- ReadDir 7 data [haha 3]
2022/07/29 17:46:02.018095 fuse.DEBUG <-- ReadDir 7 data [s3fs_exist.txt 4]
2022/07/29 17:46:02.018401 fuse.DEBUG <-- ReadDir 7 data [test.txt 5]
2022/07/29 17:46:02.018779 fuse.DEBUG <-- ReadDir 7 data [wasabi.txt 6]
2022/07/29 17:46:02.019234 fuse.DEBUG Op 0x000000e4        connection.go:491] -> OK ()
2022/07/29 17:46:02.020003 fuse.DEBUG Op 0x000000e6        connection.go:408] <- LookUpInode (parent 7, name ".wh.haha")
2022/07/29 17:46:02.021293 fuse.DEBUG Inode.LookUp 7 data [.wh.haha]
2022/07/29 17:46:02.022389 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.haha%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=773b3485a88f4ac8ee209291a6c7b3d983b6196a761d4631bbffd77963179a51
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.022441 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.haha/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=63a302708d40c0cddf1b3501a2505f9e954f3f3adfe369b2f8100d97f096ccd8
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.022817 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.haha HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e1d15934d1df54a449231e9c2d61d8962758684f0e1b0e6c3bcf92fb611ad684
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.042206 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: nQQNcRq2HMR2yZ6MJbhRjJDxnynkSzMaDKgsGw5TzmcI2N3z+Tn4K6kblkeHJAy/e2YpmQRzRssU
X-Amz-Request-Id: 9EAC2EC7AAA296B1

-----------------------------------------------------
2022/07/29 17:46:02.042321 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.046026 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: Q+RsYdLHT/C7kPIR6DP6YZ6Qd+8Q6MrKLZOBM2BK6X+1QYMJ8VlVu8pP3cvXBW8oI3TC5KisagKI
X-Amz-Request-Id: E5F18A1C8B39E5E3

-----------------------------------------------------
2022/07/29 17:46:02.046118 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: E5F18A1C8B39E5E3, host id: Q+RsYdLHT/C7kPIR6DP6YZ6Qd+8Q6MrKLZOBM2BK6X+1QYMJ8VlVu8pP3cvXBW8oI3TC5KisagKI
2022/07/29 17:46:02.046296 s3.DEBUG HEAD data/.wh.haha/ = no such file or directory
2022/07/29 17:46:02.047439 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: QD5MtiUM5Jtm8uuwD1vM+HtJ/TnnEHHlIq8g5MLTj0c7RxpwWdgkNr8n5AgNbELisk6fGXbqmzYS
X-Amz-Request-Id: C939F87AE3440EB5

-----------------------------------------------------
2022/07/29 17:46:02.047741 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: C939F87AE3440EB5, host id: QD5MtiUM5Jtm8uuwD1vM+HtJ/TnnEHHlIq8g5MLTj0c7RxpwWdgkNr8n5AgNbELisk6fGXbqmzYS
2022/07/29 17:46:02.047809 s3.DEBUG HEAD data/.wh.haha = no such file or directory
2022/07/29 17:46:02.047892 fuse.DEBUG <-- LookUpInode 7 .wh.haha no such file or directory
2022/07/29 17:46:02.048262 fuse.DEBUG Op 0x000000e6        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.048921 fuse.DEBUG Op 0x000000e8        connection.go:408] <- LookUpInode (parent 7, name "haha")
2022/07/29 17:46:02.049199 fuse.DEBUG Ref 3 data/haha [1]
2022/07/29 17:46:02.050276 fuse.DEBUG <-- LookUpInode 7 haha <nil>
2022/07/29 17:46:02.050744 fuse.DEBUG Op 0x000000e8        connection.go:491] -> OK (inode 3)
2022/07/29 17:46:02.051780 fuse.DEBUG Op 0x000000ea        connection.go:408] <- OpenFile (inode 3)
2022/07/29 17:46:02.051931 fuse.DEBUG OpenFile 3 data/haha []
2022/07/29 17:46:02.052524 fuse.DEBUG Op 0x000000ea        connection.go:491] -> OK ()
2022/07/29 17:46:02.053639 fuse.DEBUG Op 0x000000ec        connection.go:408] <- GetXattr (inode 3, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.053812 fuse.DEBUG GetXattr 3 data/haha [trusted.overlay.origin]
2022/07/29 17:46:02.053969 fuse.DEBUG Op 0x000000ec        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.054509 fuse.DEBUG Op 0x000000ee        connection.go:408] <- GetXattr (inode 3, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.055040 fuse.DEBUG GetXattr 3 data/haha [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.055163 fuse.DEBUG Op 0x000000ee        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.056082 fuse.DEBUG FlushFile 3 data/haha []
2022/07/29 17:46:02.056162 fuse.DEBUG <-- FlushFile 3 data/haha [<nil> 18 3]
2022/07/29 17:46:02.056359 fuse.DEBUG Op 0x000000f0        connection.go:408] <- FlushFile (inode 3)
2022/07/29 17:46:02.056742 fuse.DEBUG Op 0x000000f0        connection.go:491] -> OK ()
2022/07/29 17:46:02.057059 fuse.DEBUG Op 0x000000f2        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.057202 fuse.DEBUG ReleaseFileHandle data/haha 18 3
2022/07/29 17:46:02.057421 fuse.DEBUG Op 0x000000f2        connection.go:491] -> OK ()
2022/07/29 17:46:02.228875 fuse.DEBUG Op 0x000000f4        connection.go:408] <- LookUpInode (parent 7, name ".wh.s3fs_exist.txt")
2022/07/29 17:46:02.229026 fuse.DEBUG Inode.LookUp 7 data [.wh.s3fs_exist.txt]
2022/07/29 17:46:02.230504 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.s3fs_exist.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ff16ef129599723d94550ae861f87f163ff8db6f1c722099db548989dc4db0f4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.230578 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.s3fs_exist.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=dd41551993fe9d417d4c383b46edd44f218867ba74a5f1d4dd2dad7b99edac75
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.232342 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.s3fs_exist.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=672a2076c585517a517568c7c872e3e84fcad57ef09e86cd85e287beebfcb48a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.250949 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: gosoqlk0W6lgRpgcBcP5KHxgMyBoMjuMkJEjZgq+Dc9zv3yT0/VTN2ohvsGNVCfH87aI1qMbbRDv
X-Amz-Request-Id: A7DA0CDF774196EF

-----------------------------------------------------
2022/07/29 17:46:02.251015 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: A7DA0CDF774196EF, host id: gosoqlk0W6lgRpgcBcP5KHxgMyBoMjuMkJEjZgq+Dc9zv3yT0/VTN2ohvsGNVCfH87aI1qMbbRDv
2022/07/29 17:46:02.251190 s3.DEBUG HEAD data/.wh.s3fs_exist.txt/ = no such file or directory
2022/07/29 17:46:02.254850 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: jbfBflgsD6TG6bR8yjrUtltHxvPxtnX+2PMNWqabEk9/kvyLyPoJexH/4J5hiys4RA1uO1IZQZWN
X-Amz-Request-Id: 5182B56A26578396

-----------------------------------------------------
2022/07/29 17:46:02.254987 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 5182B56A26578396, host id: jbfBflgsD6TG6bR8yjrUtltHxvPxtnX+2PMNWqabEk9/kvyLyPoJexH/4J5hiys4RA1uO1IZQZWN
2022/07/29 17:46:02.255015 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: hf6s+XjpisCKS99E3YePvxhFUqPTCsOSprBKl8Qop3hm595WHLuE+fB9etB58YLoBAGD3gvoAxNq
X-Amz-Request-Id: 725A324681BB353D

-----------------------------------------------------
2022/07/29 17:46:02.256131 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.255145 s3.DEBUG HEAD data/.wh.s3fs_exist.txt = no such file or directory
2022/07/29 17:46:02.257221 fuse.DEBUG <-- LookUpInode 7 .wh.s3fs_exist.txt no such file or directory
2022/07/29 17:46:02.258195 fuse.DEBUG Op 0x000000f4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.259275 fuse.DEBUG Op 0x000000f6        connection.go:408] <- LookUpInode (parent 7, name "s3fs_exist.txt")
2022/07/29 17:46:02.260405 fuse.DEBUG Ref 4 data/s3fs_exist.txt [1]
2022/07/29 17:46:02.261002 fuse.DEBUG <-- LookUpInode 7 s3fs_exist.txt <nil>
2022/07/29 17:46:02.262174 fuse.DEBUG Op 0x000000f6        connection.go:491] -> OK (inode 4)
2022/07/29 17:46:02.262933 fuse.DEBUG Op 0x000000f8        connection.go:408] <- OpenFile (inode 4)
2022/07/29 17:46:02.263181 fuse.DEBUG OpenFile 4 data/s3fs_exist.txt []
2022/07/29 17:46:02.263485 fuse.DEBUG Op 0x000000f8        connection.go:491] -> OK ()
2022/07/29 17:46:02.265339 fuse.DEBUG Op 0x000000fa        connection.go:408] <- GetXattr (inode 4, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.265800 fuse.DEBUG GetXattr 4 data/s3fs_exist.txt [trusted.overlay.origin]
2022/07/29 17:46:02.265932 fuse.DEBUG Op 0x000000fa        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.267369 fuse.DEBUG Op 0x000000fc        connection.go:408] <- GetXattr (inode 4, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.267413 fuse.DEBUG GetXattr 4 data/s3fs_exist.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.268331 fuse.DEBUG Op 0x000000fc        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.269584 fuse.DEBUG Op 0x000000fe        connection.go:408] <- FlushFile (inode 4)
2022/07/29 17:46:02.269999 fuse.DEBUG FlushFile 4 data/s3fs_exist.txt []
2022/07/29 17:46:02.270088 fuse.DEBUG <-- FlushFile 4 data/s3fs_exist.txt [<nil> 19 4]
2022/07/29 17:46:02.271082 fuse.DEBUG Op 0x000000fe        connection.go:491] -> OK ()
2022/07/29 17:46:02.271454 fuse.DEBUG Op 0x00000100        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.271791 fuse.DEBUG ReleaseFileHandle data/s3fs_exist.txt 19 4
2022/07/29 17:46:02.272102 fuse.DEBUG Op 0x00000100        connection.go:491] -> OK ()
2022/07/29 17:46:02.349135 fuse.DEBUG Op 0x00000102        connection.go:408] <- LookUpInode (parent 7, name ".wh.test.txt")
2022/07/29 17:46:02.349368 fuse.DEBUG Inode.LookUp 7 data [.wh.test.txt]
2022/07/29 17:46:02.351002 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.test.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0f57739327956f6b6bb6b70fd7d008a6ffd417d9f6e6465fc335e919bddaa562
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.351070 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.test.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a26b7d7dda17c5c2deffea4bd2c5dabf4458f6cb69339b0ed43be7395c00c927
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.350959 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.test.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4f56cc5b160ab936d3cea85bd1df2b575f9bb1104ef05dd0aadbae8fe988f7cc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.370923 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: aywVI6em2Phum6noQVVETvUCN62pHaE/6BqfgL0Ive+ENrLiYrB3SCwv0uFdNU5uITbIy0DXA76w
X-Amz-Request-Id: 31395DB1FA78EBCD

-----------------------------------------------------
2022/07/29 17:46:02.370983 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.439007 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: Mlqwmen02/EnzQYYfLe0nWIinY+VVZJlbZoSO1o+1eHd+swKDY9aqs59Im5Ex1ZQvXbeqP9t+NuM
X-Amz-Request-Id: 6E1E6A9F4236FA7D

-----------------------------------------------------
2022/07/29 17:46:02.439165 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 6E1E6A9F4236FA7D, host id: Mlqwmen02/EnzQYYfLe0nWIinY+VVZJlbZoSO1o+1eHd+swKDY9aqs59Im5Ex1ZQvXbeqP9t+NuM
2022/07/29 17:46:02.439439 s3.DEBUG HEAD data/.wh.test.txt = no such file or directory
2022/07/29 17:46:02.440161 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: mUbhsVvg5OjqexfMlj8LnwIblEGyI6GbCZ51I+ExljwGXCeejAs7pm6NIlMfIcojQAXUIJAzCJoT
X-Amz-Request-Id: 52A6BD86437DE9F8

-----------------------------------------------------
2022/07/29 17:46:02.440290 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 52A6BD86437DE9F8, host id: mUbhsVvg5OjqexfMlj8LnwIblEGyI6GbCZ51I+ExljwGXCeejAs7pm6NIlMfIcojQAXUIJAzCJoT
2022/07/29 17:46:02.440373 s3.DEBUG HEAD data/.wh.test.txt/ = no such file or directory
2022/07/29 17:46:02.440466 fuse.DEBUG <-- LookUpInode 7 .wh.test.txt no such file or directory
2022/07/29 17:46:02.440569 fuse.DEBUG Op 0x00000102        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.441240 fuse.DEBUG Op 0x00000104        connection.go:408] <- LookUpInode (parent 7, name "test.txt")
2022/07/29 17:46:02.441986 fuse.DEBUG Ref 5 data/test.txt [1]
2022/07/29 17:46:02.442142 fuse.DEBUG <-- LookUpInode 7 test.txt <nil>
2022/07/29 17:46:02.443075 fuse.DEBUG Op 0x00000104        connection.go:491] -> OK (inode 5)
2022/07/29 17:46:02.444337 fuse.DEBUG Op 0x00000106        connection.go:408] <- OpenFile (inode 5)
2022/07/29 17:46:02.444800 fuse.DEBUG OpenFile 5 data/test.txt []
2022/07/29 17:46:02.445494 fuse.DEBUG Op 0x00000106        connection.go:491] -> OK ()
2022/07/29 17:46:02.446433 fuse.DEBUG Op 0x00000108        connection.go:408] <- GetXattr (inode 5, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.446755 fuse.DEBUG GetXattr 5 data/test.txt [trusted.overlay.origin]
2022/07/29 17:46:02.447210 fuse.DEBUG Op 0x00000108        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.448272 fuse.DEBUG Op 0x0000010a        connection.go:408] <- GetXattr (inode 5, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.448409 fuse.DEBUG GetXattr 5 data/test.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.448566 fuse.DEBUG Op 0x0000010a        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.448800 fuse.DEBUG Op 0x0000010c        connection.go:408] <- FlushFile (inode 5)
2022/07/29 17:46:02.449234 fuse.DEBUG FlushFile 5 data/test.txt []
2022/07/29 17:46:02.449840 fuse.DEBUG <-- FlushFile 5 data/test.txt [<nil> 20 5]
2022/07/29 17:46:02.449932 fuse.DEBUG Op 0x0000010c        connection.go:491] -> OK ()
2022/07/29 17:46:02.451087 fuse.DEBUG Op 0x0000010e        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.451377 fuse.DEBUG ReleaseFileHandle data/test.txt 20 5
2022/07/29 17:46:02.451642 fuse.DEBUG Op 0x0000010e        connection.go:491] -> OK ()
2022/07/29 17:46:02.515944 fuse.DEBUG Op 0x00000110        connection.go:408] <- LookUpInode (parent 7, name ".wh.wasabi.txt")
2022/07/29 17:46:02.516110 fuse.DEBUG Inode.LookUp 7 data [.wh.wasabi.txt]
2022/07/29 17:46:02.519987 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.wasabi.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f9a388012941337807606fb0b449063f7a307549af1b3766021ffbcef3ca532e
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.524253 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.wasabi.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fb6d31d2d6df977ae603c498d85e9983b6857a2e1fab6788eeb0bc40bfec4d88
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.525530 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.wasabi.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b49edf03ad429107a2fa5ed18bb054edda4b3717f893c1e4eb52be6dc89f24cc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.540007 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: e5AQxSTMgBhP3dCdmXHOkjdIX0iaFjLv1hS1Ro4r8GZeZBxEjEISLIpWYLRFvWNovObbOkGKZyzh
X-Amz-Request-Id: A51E2DE582F141A8

-----------------------------------------------------
2022/07/29 17:46:02.541142 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: A51E2DE582F141A8, host id: e5AQxSTMgBhP3dCdmXHOkjdIX0iaFjLv1hS1Ro4r8GZeZBxEjEISLIpWYLRFvWNovObbOkGKZyzh
2022/07/29 17:46:02.541345 s3.DEBUG HEAD data/.wh.wasabi.txt = no such file or directory
2022/07/29 17:46:02.546012 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: sNE/VifE1Qra2p/sxOEuCYuWLJtq3EbftoaVcBeFn8pNJD25yvMdnwF8UwyQbmjBt1u1wJtKXIos
X-Amz-Request-Id: 3FCDB4C5F790C3C3

-----------------------------------------------------
2022/07/29 17:46:02.546339 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 3FCDB4C5F790C3C3, host id: sNE/VifE1Qra2p/sxOEuCYuWLJtq3EbftoaVcBeFn8pNJD25yvMdnwF8UwyQbmjBt1u1wJtKXIos
2022/07/29 17:46:02.548187 s3.DEBUG HEAD data/.wh.wasabi.txt/ = no such file or directory
2022/07/29 17:46:02.549309 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: cgW5kcWA5gxe1sdx2fCDjXkkaGfoGw2Bur8r/ddbKtV/7vKJ20IpuePN0NZy+6vcfCb3HaOTm0Pw
X-Amz-Request-Id: DEA8FAC2C8D6366C

-----------------------------------------------------
2022/07/29 17:46:02.549407 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.549508 fuse.DEBUG <-- LookUpInode 7 .wh.wasabi.txt no such file or directory
2022/07/29 17:46:02.549750 fuse.DEBUG Op 0x00000110        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.550265 fuse.DEBUG Op 0x00000112        connection.go:408] <- LookUpInode (parent 7, name "wasabi.txt")
2022/07/29 17:46:02.550768 fuse.DEBUG Ref 6 data/wasabi.txt [1]
2022/07/29 17:46:02.550937 fuse.DEBUG <-- LookUpInode 7 wasabi.txt <nil>
2022/07/29 17:46:02.551049 fuse.DEBUG Op 0x00000112        connection.go:491] -> OK (inode 6)
2022/07/29 17:46:02.552086 fuse.DEBUG Op 0x00000114        connection.go:408] <- OpenFile (inode 6)
2022/07/29 17:46:02.553206 fuse.DEBUG OpenFile 6 data/wasabi.txt []
2022/07/29 17:46:02.555026 fuse.DEBUG Op 0x00000114        connection.go:491] -> OK ()
2022/07/29 17:46:02.555732 fuse.DEBUG Op 0x00000116        connection.go:408] <- GetXattr (inode 6, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.556361 fuse.DEBUG GetXattr 6 data/wasabi.txt [trusted.overlay.origin]
2022/07/29 17:46:02.556484 fuse.DEBUG Op 0x00000116        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.556862 fuse.DEBUG Op 0x00000118        connection.go:408] <- GetXattr (inode 6, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.557220 fuse.DEBUG GetXattr 6 data/wasabi.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.558169 fuse.DEBUG Op 0x00000118        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.558869 fuse.DEBUG Op 0x0000011a        connection.go:408] <- FlushFile (inode 6)
2022/07/29 17:46:02.559324 fuse.DEBUG FlushFile 6 data/wasabi.txt []
2022/07/29 17:46:02.559366 fuse.DEBUG <-- FlushFile 6 data/wasabi.txt [<nil> 21 6]
2022/07/29 17:46:02.559450 fuse.DEBUG Op 0x0000011a        connection.go:491] -> OK ()
2022/07/29 17:46:02.560467 fuse.DEBUG Op 0x0000011c        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.561035 fuse.DEBUG ReleaseFileHandle data/wasabi.txt 21 6
2022/07/29 17:46:02.561221 fuse.DEBUG Op 0x0000011c        connection.go:491] -> OK ()
2022/07/29 17:46:02.629178 fuse.DEBUG Op 0x0000011e        connection.go:408] <- ReadDir (inode 7)
2022/07/29 17:46:02.629709 fuse.DEBUG ReadDir 7 data [6]
2022/07/29 17:46:02.629807 fuse.DEBUG Op 0x0000011e        connection.go:491] -> OK ()
2022/07/29 17:46:02.630161 fuse.DEBUG Op 0x00000120        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:02.630298 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:02.630757 fuse.DEBUG Op 0x00000120        connection.go:491] -> OK ()
2022/07/29 17:46:02.631249 fuse.DEBUG Op 0x00000122        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.opaque", name trusted.overlay.opaque)
2022/07/29 17:46:02.631501 fuse.DEBUG GetXattr 7 data [trusted.overlay.opaque]
2022/07/29 17:46:02.631787 fuse.DEBUG Op 0x00000122        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.632119 fuse.DEBUG Op 0x00000124        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.632733 fuse.DEBUG Op 0x00000126        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:02.633164 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.633303 fuse.DEBUG Op 0x00000124        connection.go:491] -> OK ()
2022/07/29 17:46:02.633766 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:02.634176 fuse.DEBUG Op 0x00000126        connection.go:491] -> OK ()
2022/07/29 17:46:02.634988 fuse.DEBUG Op 0x00000128        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.opaque", name user.fuseoverlayfs.opaque)
2022/07/29 17:46:02.635971 fuse.DEBUG GetXattr 7 data [user.fuseoverlayfs.opaque]
2022/07/29 17:46:02.636127 fuse.DEBUG Op 0x00000128        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.636486 fuse.DEBUG Op 0x0000012a        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.636778 fuse.DEBUG Op 0x0000012c        connection.go:408] <- LookUpInode (parent 7, name ".wh..wh..opq")
2022/07/29 17:46:02.636874 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.636947 fuse.DEBUG Op 0x0000012a        connection.go:491] -> OK ()
2022/07/29 17:46:02.637269 fuse.DEBUG Inode.LookUp 7 data [.wh..wh..opq]
2022/07/29 17:46:02.638420 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh..wh..opq/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7138f2ef2ccbbe01c06c591462da26400b3a1aca0ca267dfefd0cee8944fad64
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.638821 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh..wh..opq HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=97d0c8d3d6a7985a5f6d73c6477e5b369795de05e989ef6f676bb90fbcf563c2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.640115 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh..wh..opq%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=5f3ea37f58f8a12be21eeffdf6ec3f543e7c713fbd483b9ae34de8cac52f2e21
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z

-----------------------------------------------------
2022/07/29 17:46:02.658716 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Id-2: VC9ZVLtmySeIBRu/nvEjLqzTHKLFcZELMIgVmtKbOmHPPToCjnEvdZn9DU9BHy2Dkt6XjJ0h+H/8
X-Amz-Request-Id: 67896E55D02482A9

-----------------------------------------------------
2022/07/29 17:46:02.658831 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 67896E55D02482A9, host id: VC9ZVLtmySeIBRu/nvEjLqzTHKLFcZELMIgVmtKbOmHPPToCjnEvdZn9DU9BHy2Dkt6XjJ0h+H/8
2022/07/29 17:46:02.658922 s3.DEBUG HEAD data/.wh..wh..opq/ = no such file or directory
2022/07/29 17:46:02.659313 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: iHynM0NrNX3x9tl8Tkqs0sxhOAwBnzhNWvpQ4feb93Rsr7462X0UO+t3V5l5G4jAmAyHBp2lkxuV
X-Amz-Request-Id: 1FD2E5F062775ACD

-----------------------------------------------------
2022/07/29 17:46:02.659758 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
    status code: 404, request id: 1FD2E5F062775ACD, host id: iHynM0NrNX3x9tl8Tkqs0sxhOAwBnzhNWvpQ4feb93Rsr7462X0UO+t3V5l5G4jAmAyHBp2lkxuV
2022/07/29 17:46:02.659888 s3.DEBUG HEAD data/.wh..wh..opq = no such file or directory
2022/07/29 17:46:02.661941 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: H0TEv5ADuOuFLKKcAO23xbZ5a+T3odScGr7PbjaHrmNYiwSgoGDyceky9c27V2gOf1p9j9mYOjh+
X-Amz-Request-Id: A72AD4F1A6548A9E

-----------------------------------------------------
2022/07/29 17:46:02.662049 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.662101 fuse.DEBUG <-- LookUpInode 7 .wh..wh..opq no such file or directory
2022/07/29 17:46:02.662204 fuse.DEBUG Op 0x0000012c        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.664324 fuse.DEBUG Op 0x0000012e        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.665381 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.668281 fuse.DEBUG Op 0x0000012e        connection.go:491] -> OK ()
2022/07/29 17:46:08.052420 fuse.DEBUG Op 0x00000130        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:08.052761 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:08.053438 fuse.DEBUG Op 0x00000130        connection.go:491] -> OK ()

[giuseppe]

could you run fuse-overlayfs with the -d option?

EDIT: nevermind, sorry, I see you already did

[giuseppe]

does it work if you try as root with native overlay?

[corbinjurgens]

I tried native overlay but it said that this type of filesystem was not supported (in the case of goofys aws/wasabi s3) which is why I then moved onto migratefs.

So i tried something like

sudo mount -t overlay -o lowerdir=/home/ec2-user/overlay_test/lower,upperdir=/home/ec2-user/overlay_test/upper,workdir=/home/ec2-user/overlay_test/work none /home/ec2-user/overlay_test/overlay

I should mention I have been trying to use fuse-overlayfs inside a docker instance. (of amazonlinux2). I tested migratefs on a production server with same results but have not yet tried fuse-overlayfs. Could this be a cause?