Fsck says uncleanly unmounted after umount used to unmount

[GoogleCodeExporter]

I'm using s3backer with an ext2 filesystem.
Recently, I unmounted the ext2 loop filesystem mount and then unmounted the 
s3backer filesystem. It unmounted immediately even though I believe that there 
were pending writes that had not yet been flushed to S3.
Then I remounted the s3backer filesystem and ran fsck on the "file" file in it. 
Fsck reported that the device was not cleanly unmounted and forced an fsck, 
thus I believe confirming that there were pending writes that had not yet been 
flushed.
Shouldn't umount of the FUSE filesystem block until all pending writes are 
flushed to disk? This is how local filesystems work, obviously.
If not, then what's the recommended way to ensure that data is not lost?
I'm using r437 on Linux (Fedora 14).
Thanks.

Original issue reported on code.google.com by jikam...@gmail.com on 15 Oct 2010 at 2:53

• Merged into: #40

[GoogleCodeExporter]

Yes, umount is supposed to block until all writes are flushed. In my experience 
it seems to properly do this (I used openSUSE but the behavior should be the 
same).

Are you sure s3backer wasn't killed somehow?

Can you create a simple script that reproduces the problem? If you do this, 
there is no need to include ext2fs in the test. Just mount an s3backer file, 
write some data into it via dd(1), and then unmount it.. the umount should 
block until s3backer finished writing all the blocks.

Original comment by archie.c...@gmail.com on 15 Oct 2010 at 3:55

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by archie.c...@gmail.com on 19 Oct 2010 at 4:51

• Changed state: Feedback

[GoogleCodeExporter]

I just unmounted my ReiserFS filesystem that's on top of s3backer and then 
unmounted s3backer. Both unmounted successfully and cat /proc/mounts doesn't 
show either of them. Nevertheless, s3backer is still running:

root       335  0.6  0.0 1080536 12004 ?       Ssl  07:51   3:30 s3backer 
jik2-backup-dev2 /mnt/s3backer-dev -o 
rw,noatime,accessFile=/etc/passwd-s3fs,compress=9,rrs,blockCacheFile=/var/cache/
s3backer-dev-cache,size=100G,blockSize=128k,blockCacheSize=250000,blockCacheThre
ads=5,dev,suid

And wireshark says it's still generating traffic.

This is sub-optimal.

Original comment by jikam...@gmail.com on 19 Oct 2010 at 9:02

[GoogleCodeExporter]

Hmmm, interesting. Sounds like FUSE is allowing the file system to unmount 
before fuse_op_destroy() returns, which would be weird/bad. Other possibility 
is an s3backer bug of course :-)

Please try that again with the attached patch. Shutting down s3backer should 
produce these messages in the logs:

  fuse_op_destroy(): shutting down s3backer...
  fuse_op_destroy(): shut down of s3backer complete

You should see BOTH messages before the umount command returns, and then 
s3backer should exit immediately thereafter.

Original comment by archie.c...@gmail.com on 19 Oct 2010 at 10:17

Attachments:

• s3bpatch.txt

[GoogleCodeExporter]

It's LOG_WARNING, not LOG_WARN.
I'll install your patch and let you know what happens if/when I reproduce the 
issue.

Original comment by jikam...@gmail.com on 20 Oct 2010 at 12:27

[GoogleCodeExporter]

Oct 19 22:39:44 jik2 s3backer: fuse_op_destroy(): shutting down s3backer...
Oct 19 22:39:49 jik2 q-backup: s3backer is still running?
Oct 19 22:42:44 jik2 s3backer: fuse_op_destroy(): shut down of s3backer complete

The second line above is logged by my nightly backup script after it (a) 
umounts the s3backer fuse filesystem, (b) sleeps 5 seconds, and (c) checks if 
s3backer is still running. So yes, it looks like FUSE is allowing the 
filesystem to unmount before fuse_op_destroy() returns. I think you might need 
to look into that. :-/

Original comment by jikam...@gmail.com on 20 Oct 2010 at 2:44

[GoogleCodeExporter]

Although I don't completely understand it, this thread sheds some light on 
things:

http://lists.freebsd.org/pipermail/freebsd-hackers/2007-December/022598.html

Apparently FUSE will not wait for DESTROY to finish in certain situations, 
i.e., depending on whether the user or root originally mounted the filesystem. 
They are trying to avoid a security problem where via FUSE a regular user could 
indefinitely block the system shutdown process.

For a test, try your test case again but do all mounting and unmounting as root 
and see if you get the same result... 

Original comment by archie.c...@gmail.com on 20 Oct 2010 at 1:30

[GoogleCodeExporter]

I'm already mounting as root.

Original comment by jikam...@gmail.com on 20 Oct 2010 at 2:31

[GoogleCodeExporter]

Linux also has the concept of "lazy umount" which could possibly cause this. 
Verify that your umount command is not being invoked with the `-l' flag which 
would be a lazy umount.

Otherwise I'm not sure what the problem is.

Original comment by archie.c...@gmail.com on 21 Oct 2010 at 7:57

[GoogleCodeExporter]

See also the --blockCacheMaxDirty flag, which can limit the effects of this 
problem.

But apply my patch in issue #25 first to fix a bug in this flag.

Original comment by archie.c...@gmail.com on 21 Oct 2010 at 7:59

[GoogleCodeExporter]

Not doing a lazy unmount.

Original comment by jikam...@gmail.com on 21 Oct 2010 at 8:08

[GoogleCodeExporter]

Even with blockCacheMaxDirty, if you want to protect the integrity of the S3 
data, you probably need to specify the "sync" mount option for the ext2 / 
reiserfs / whatever filesystem that's on top of the S3 file. That will prevent 
the kernel from doing lots of buffering of blocks in the S3 filesystem and 
essentially force end-to-end compliance with your blockCacheMaxDirty setting.

Original comment by jikam...@gmail.com on 22 Oct 2010 at 4:10

[GoogleCodeExporter]

Same as issue #40, merging this one into that one.

Original comment by archie.c...@gmail.com on 1 May 2013 at 3:02

• Changed state: Duplicate