goofys will stop when the aws s3 request response 4xx errors are continuous

[shimomuramnxlab]

hi kahing.

I mount S3 bucket as a file system using goofys in the AWS environment.

Goofys may stop if you move a large number of files to the mount point and refer recursively with the tree or ls - ltR command.

The output of the debug log is as follows.

Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0c        connection.go:395] <- OpenDir (inode 1)
Apr 13 14:43:13 ip-10-28-16-224 journal: Suppressed 129 messages from /user.slice/user-0.slice
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0c        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0d        connection.go:395] <- ReadDir (inode 1)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0d        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0e        connection.go:395] <- GetInodeAttributes (inode 1)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0e        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0f        connection.go:395] <- LookUpInode (parent 1, name "IphoneDefaultImage")
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef0f        connection.go:478] -> OK (inode 9859)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef10        connection.go:395] <- LookUpInode (parent 1, name "IphoneDefaultImage")
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef11        connection.go:395] <- ForgetInode (inode 9859)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef11        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef12        connection.go:395] <- ForgetInode (inode 7952)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef12        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef10        connection.go:478] -> OK (inode 9861)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef13        connection.go:395] <- ReadDir (inode 1)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef13        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef14        connection.go:395] <- ReleaseDirHandle
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef14        connection.go:478] -> OK ()
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef15        connection.go:395] <- GetInodeAttributes (inode 1)
Apr 13 14:43:13 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef15        connection.go:478] -> OK ()
Apr 13 14:43:28 ip-10-28-16-224 /usr/local/go/bin/goofys[13028]: fuse.DEBUG Op 0x0007ef16        connection.go:395] <- OpenDir (inode 9860)

When checking the request metrics of S3 at the time when goofys stopped in CloudWatch, 4xx error occurred in large quantities.

Can you help me in resolving this issue? Thanks.

[kahing]

could you enable --debug_s3 and see which 4xx error it's getting, and what were the requests that were causing the errors?

[shimomuramnxlab]

I tried to enable --debug_s3 option. But, I think only --debug_fuse option is enabled.

version.

[root@AP2 ~]# go version
go version go1.9.2 linux/amd64
[root@AP2 ~]# goofys -v
goofys version 0.19.0-use `make build' to fill version hash correctly

What I have set are as follows. /etc/fstab

/usr/local/go/bin/goofys#mobeix-images-4213-0238-5942 /opt/Process-jboss-eap-6.4/welcome-content/mobeiximages       fuse    _netdev,allow_other,--region=ap-northeast-1,--file-mode=0666,--debug_fuse,--debug_s3,--uid=1000,--gid=1000       0 0

I will attach a syslog, so check it.

goofys.log.gz

Thanks.

[kahing]

could you try running goofys in the foreground manually with --debug_s3?

[shimomuramnxlab]

I try running goofys in the foregground.

attach log file. so please check it. goofys_debug.log1.gz

Thanks.

[kahing]

the only 4xx error is 404 for dir blobs which is normal. What version of goofys is this?

[shimomuramnxlab]

I use goofys version 0.19.0.

[ec2-user@AP2 ~]$ goofys -v
goofys version 0.19.0-use `make build' to fill version hash correctly

[kahing]

the crash is a regression from the fix of #279, need to think about a clean way to fix this

[kahing]

if you use the tagged 0.19.0 instead of master, you should be able to avoid this problem

[shimomuramnxlab]

Thanks for replying.

I installed the v0.19.0 in the next procedure. 1) go get github.com/kahing/goofys 2) cd go / src / github.com / kahing / goofys 3) git checkout v0.19.0 4) go install

I think the problem seems to be solved. I will look at the situation for one or two weeks.