fuse permanently caches incorrect information

[GoogleCodeExporter]

I've got a test that frequently winds up with an "empty" permanode coming back 
from a describe on a directory at just the right time such that it's treated as 
a directory and can never go away.

I've verified that camlistore sees the claims attached and believes it's a file 
(with content) and that fuse is making a request for describe that's returning 
this data *and* it's being applied and put into place.

The only thing I *don't* see happening is FUSE asking cammount for information 
about this file.  Waiting doesn't help.  All the internal caches clear and 
refresh happily, but no changes are reflected in the filesystem state.

Restarting cammount instantly causes my file to become a file again.

Original issue reported on code.google.com by dsalli...@gmail.com on 27 Dec 2013 at 9:32

[GoogleCodeExporter]

This seems to mostly be a problem when a directory secretly becomes a file 
behind the scenes.  This happens when we request a permanode description before 
it has a chance to roll up its claims.  It's a race condition, but a 
particularly nasty one because the APIs we're using seem to have no way to 
force FUSE to forget what we told it* so that it can move on.

Events that fuse believes *should* change the item would update its internal 
cache.  e.g. if I raced in an empty permanode "x" such that we reported it as a 
directory before any claims came in, it would stay a directory indefinitely, 
but if I rename it to y, it would lookup y and happily see it as a file.

The best, and hopefully correct way to handle this situation I found for this 
is to explicitly set the camliType to "directory" and only return a child as a 
directory if we can verify either it's explicitly considered a directory by the 
above or implicitly by having camliPath: attributes within it.

My understanding is that there's a lower-level FUSE API which pushes all of 
these concerns to the application itself.  This may be the right thing to do as 
we can more accurately reflect what's happening in camlistored, but I'd guess 
this is a fairly large undertaking.

* I invalidated just about everything in fs (aggressively dropped all child 
caches, randomized inodes, etc...) and watched it refetch stuff, but fuse 
itself never did a Lookup on the item in question.

Original comment by dsalli...@gmail.com on 27 Dec 2013 at 10:52

[GoogleCodeExporter]

The change below at the very least prevents this "forever a directory" issue 
from occurring, though I don't think it's a complete solution to this class of 
problem.

https://camlistore-review.googlesource.com/1720

Original comment by dsalli...@gmail.com on 28 Dec 2013 at 12:40

[GoogleCodeExporter]

Something like https://camlistore-review.googlesource.com/1720 seems fine, but 
I'd still like to understand this more.

a) when the indexer + blob server are in the same server, they should be 
configured such that HTTP uploads of claim blobs don't return with a 200 OK 
until the blob has been synchronously indexed. so if you wait for the upload 
before doing a describe, you shouldn't see a race.  but it's possible another 
thread/process did.

b) I thought all the FUSE operations had, in the binary protocol to the kernel, 
a validity/expiration time.

Original comment by bradfitz on 28 Dec 2013 at 6:54

[GoogleCodeExporter]

I would also like to understand this more. 

The most fascinating part of this is that the test executes flawlessly if I run 
it with cwd in the test directory in camlistore.  It's quite likely a lot of 
this is OS X specific buggery.

Here's an example of the extraction of the mutation test running inside vs. 
outside of the fuse directory (inside == cwd is test dir and path base is "." 
vs. being in ~ and having path base inside my camlistore root)

https://gist.github.com/dustin/8154060

Original comment by dsalli...@gmail.com on 28 Dec 2013 at 8:56

[GoogleCodeExporter]

I've not been able to reproduce this since we stopped considering a permanode 
with no claims a directory.

Original comment by dsalli...@gmail.com on 7 Jan 2014 at 11:32

• Changed state: Fixed

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/293

Original comment by bradfitz on 14 Dec 2014 at 11:36

• Added labels: IssueMoved, Restrict-AddIssueComment-Commit