search

mar-file-system / marfs

MarFS provides a scalable near-POSIX file system by using one or more POSIX file systems as a scalable metadata component and one or more data stores (object, file, etc) as a scalable data component.
Other
97 stars 27 forks source link

should we associate files with fds, and add locking? #89

Open jti-lanl opened 8 years ago

jti-lanl commented 8 years ago

This comes up in 2 places.

  1. NFS server appeared to be sending open / fsync / close to the fuse mount, while it (NFS) also had another thread or threads with an open file handle to that same file. It seemed to me that NFS was thinking we would be able to associate the pathname used in the open/fsync/close with the other fd we had open to the same path.
  2. If a user does two writes to the same file through fuse, even if they are done serially, it's possible that they will conflict. In the example below, the release from the first open may not have finished updating xattrs, or closing the object, when the second one tries its open, or tries its ftruncate to move to the trash. Maybe there's a fuse option short of making all writes synchronous, to fix this.

( echo test > /marfs/jti/myfile; echo test > /marfs/jti/myfile)

A quick test suggests that adding a 'sync' makes it work, but I'm not sure whether this is absolutely reliable:

( echo test > /marfs/jti/myfile;sync; echo test > /marfs/jti/myfile)

The heavy-duty fix for both (1) and (2) would be to have all opens do something like the following:

If the filename is not in the table, we'd install it there, associated with the new fd (while the lock was held). If it is in the table, we'd get that fd and then have nightmares about all the ways things can go wrong when one thread is trying to fsync an fd that another thread is using, etc. But this is presumably the kind of thing we'd have to do to handle the NFS behavior described above in (1), assuming I've understood it correctly. Regarding (2), perhaps we can reasonably require users to just never do this. But if they are allowed to do this, then we could detect it in the second open() and wait for the first thread to finish. But that would involve more locking.

So far, we've managed to avoid adding any locking to fuse proper. (There is locking in per-thread interactions with their own object-streams, but that's all.) Also, just because we can make this association doesn't mean we should. It costs us some ugliness, and maybe risks some hard-to-diagnose performance problems, and for what?

garygrider commented 8 years ago

Files are associated with fds if Marfs is written to spec and thus most normal locking functions exist

When operating on a file, the mdfile should be open by the operating process even though you don't read or write into the mdfile This gets us proper behavior for things like unlink

As for locking within a byte range, not sure that matters, if the user chooses to shoot his foot he can do that with posix too, with posix he gets ordering but that's not much help given false sharing

There are some areas where atomic ops would be nice, mostly in the area of batch ops changing metadata that are perhaps worth a chat to minimize the vulnerability

Again, need more info on the concern

From: Brett Kettering notifications@github.com Sent: Wednesday, April 13, 2016 7:22:15 AM To: mar-file-system/marfs Cc: Grider, Gary Alan Subject: Re: [mar-file-system/marfs] should we associate files with fds, and add locking? (#89)

Assigned #89https://github.com/mar-file-system/marfs/issues/89 to @garygriderhttps://github.com/garygrider.

You are receiving this because you were mentioned. Reply to this email directly or view it on GitHubhttps://github.com/mar-file-system/marfs/issues/89#event-625757289