Windows mount support

[djdv]

Tracking/discussion issue for the topic of implementing ipfs mount on Windows. This is a broad topic, all comments and criticisms are welcome here as long as they help drive us towards a solution. Nothing is set in stone yet and we'd like to implement this correctly.

Currently we have no first party support for this:

Error: Mount isn't compatible with Windows yet

For third party, I'm aware of these projects: https://github.com/alexpmorris/dipfs https://github.com/richardschneider/net-ipfs-mount both utilize the IPFS API and Dokany(a Windows FUSE alternative).

dipfs does not appear to be maintained. In my experience it will succeed in mounting IPFS as a drive letter, and allows you to traverse IPFS paths via CLI, however it hangs in explorer (likely from it trying to access metadata), or when trying to read data through any means.

net-ipfs-mount is currently being maintained. This is our best contender, everything appears to work as intended unless you have a non-tiny pinset. When trying to traverse IPFS paths with a large enough pinset, passing the list of pins from the API to net-ipfs-mount can take long enough for Windows to deem /ipfs inaccessible. For local testing you can run ipfs pins ls and see how long it takes to return.

---

I would like to start an initiative, aimed at getting Windows mounting on par with the other platforms. That is to say, at the very least, exposing read only access to /ipfs and /ipns constructs. If possible, it would be nice to extend the feature set to expose a writable MFS root as well, similar to this https://github.com/tableflip/ipfs-fuse

Most likely, this will mean implementing first party support for mount in go-ipfs, utilizing core APIs where possible.

Mention of https://github.com/billziss-gh/winfsp has come up as an alternative to Dokany. It's likely that winfsp's native API will be our target, however this is not locked down yet. If you have opinions on Windows filesystem APIs, positive or negative, please post them here.

cc: @mgoelzer, @alanshaw, @dryajov, @mrlindblom, @Kubuxu, @whyrusleeping

Edit: forgot to cc: @alexpmorris, @richardschneider

[xelra]

My vote is for WinFsp.

It has an active developer and in my opinion is the best FUSE alternative for Windows. I'm using it daily in conjunction with SSHFS.

[billziss-gh]

Adding myself here so that I can receive notifications and provide help.

[billziss-gh]

I add here some comments I made over private email regarding the task of creating a Go shim for WinFsp.

Some initial thoughts on the kind of tasks you will have to handle for a Go shim. The WinFsp Tutorial may also be useful to you as it lists the major tasks needed for a successful file system using the native API.

• Find and load the WinFsp DLL.
  • This is normally done using the inline C function FspLoad, which looks into the registry to find where the WinFsp DLL is installed and loads it. This will have to be rewritten in Go.
• WinFsp includes API’s for creating Windows services, etc. You do not need to use those if you do not plan to run IPFS as a Windows service, but if you do they are named FspService*.
• You create a file system by using FspFileSystemCreate and you mount it using FspFileSystemSetMountPoint. You can mount on a drive or directory, but mounting on a directory requires a case-insensitive file system under Windows.
• The file system starts handling operations when you call FspFileSystemStartDispatcher. This will start a number of threads and will call your file system operations on those threads. The default number of threads is the same as the number of processors (with a minimum of 2).
• Most of the file system operations should be relatively easy to understand, especially if you have experience with FUSE. Of notable difference is how files are deleted on Windows and the security model (which uses security descriptors and not POSIX permissions).

I am happy to elaborate on any of this.

[djdv]

Much appreciated @billziss-gh. Very glad to have your support with this, and a clear outline of what's necessary. 👍

WinFsp includes API’s for creating Windows services

I have considered packaging go-ipfs in an MSI that optionally installs a service for the ipfs daemon. There seems to be some support for this in golang already but it's nice to know about interactions with services and WinFSP. We may be able to take advantage of it.

[billziss-gh]

NP. Feel free to ping anytime and I will continue monitoring this issue.

WinFsp includes API’s for creating Windows services

I have considered packaging go-ipfs in an MSI that optionally installs a service for the ipfs daemon.

While I do not understand the internals of IPFS, in general file systems benefit from being run as Services under Windows. I outline some of the reasons here:

https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture

[djdv]

No going back now 👀

[BillDStrong]

https://github.com/billziss-gh/cgofuse is a go library for fuse maintained by the winfsp maintainer. It might be best in the long run to use it to create one ipfs-mount that works on all platforms. winfsp has a fuse compatability mode, when used with a cygwin dll. This would also allow decoupling somewhat the mount functionality.

[djdv]

I should mention I'll be dumping my work here from time to time https://github.com/djdv/go-ipfs/tree/feat/win-mount It will probably be ugly for a while. I'll divvy up the source files and refactor after things have at least a basic implementation. That is to say, create the binds, and rework them to be more go-like where/when possible.

@BillDStrong I agree that it would be better to have a unified mount implementation, however, I don't want to introduce a C compiler as a build dependency if it can be avoided. We can write an independent Windows variant now, wait for the *nix version to be refactored, and maybe derive an interface from commonality between them if it's desirable. We'll have to see how redundant it would be and if there'd be any benefit in making porters implement another abstraction layer or just be independent.

For the moment, I'm dumping this in /mount/winfsp, we may move /fuse into there and have a structure like this:

mount/
    fuse/
    interface/
    winfsp/

or

mount/
    fuse/
    winfsp/
    mount_interface_and_friends.go

We'll have to see in time.

[BillDStrong]

I looks like cgofuse has a cross compilation option based on xgo. It looks to include the necessary windows winfsp libraries setup. This would still add a dependency on the c compiler, but xgo's ease of use should be enough to offset that downside. It is contained in a docker container, so compilation shouldn't require to much setup over the current workflow.

This is just my opinion, though.

Prerequisites: docker, xgo Build:

$ docker pull billziss/xgo-cgofuse
$ go get -u github.com/karalabe/xgo
$ cd YOUR-PROJECT-THAT-USES-CGOFUSE
$ xgo --image=billziss/xgo-cgofuse \
    --targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 .

[Kubuxu]

@BillDStrong thanks for showing me xgo. Few blockers: 1. teaching https://github.com/ipfs/distributions how to use xgo 2. making sure that Jenkins can handle windows cgo on Windows.

[djdv]

Excluding the build dependencies, I'm concerned with some of the API differences http://www.secfs.net/winfsp/develop/native-api-vs-fuse/

I'd rather not target FUSE on Windows if it means making compromises. Since we already have FUSE for the other platforms, it feels like we're not restricted to it (on Windows) now. It may still be best to have separate, but native, implementations. Giving us as much compatibility/(flexibility later) with the host as we can get.

Thoughts on this? @BillDStrong @Kubuxu @billziss-gh

[billziss-gh]

TLDR

Cgofuse gets you nice cross-platform compatibility (on the 3 OS'es) but at the cost of having some major external dependencies (native toolchain or docker+xgo). These may or may not be appropriate for a large project with an established community and expectations.

In detail

I can see both sides of the argument.

I originally created cgofuse to support rclone. As I was new with Golang at the time, I followed @ ncw 's pointers and used cgo to create a single layer that would run on the 3 OS'es.

I find that cgofuse has been a success as it allows the creation of a file system that runs on the 3 OS'es with relative ease. For example, I use cgofuse in my fledgling project objfs (currently pre-alpha) and I do not have to think very much about cross-platform issues. This benefit outweighs for me any potential negatives.

Now for those negatives:

• Since cgofuse is a cgo project, the native toolchain is required for every user of cgofuse. @ ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

• Xgo works, but is not the same as the familiar go build. So I find that in practice I do not use it as much as I thought I would. (I also tend to have the right compilers at the ready on all my machines/VM's, so go build is easier for me.) And of course xgo requires Docker and the xgo-cgofuse docker image.

• Cgofuse currently works on "the 3 OS'es" (Windows, macOS and Linux). I am hoping to eventually extend to at least FreeBSD, but this does limit the target platforms.

Additional considerations for go-ipfs

• An additional negative for go-ipfs is that it appears to have a large community, which might balk at the idea of introducing a native toolchain requirement in order to support "Windoze" :-) But I am new to IPFS, so I cannot speak on that.

Native API vs FUSE.

Given that IPFS is likely closer to the POSIX file system model rather than the Windows file system model, the differences between the two API's might not mean much for IPFS. For example, if IPFS has no need for and does not use "security descriptors" (Windows ACL's), the ability to use the native GetSecurity and SetSecurity might not be very important.

[djdv]

@billziss-gh Thanks for sharing the experience! It's very helpful to hear.

ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

I'm curious to hear more about this, since that seems to be the current task at hand for me. Wrapping the DLL and making Go and C play nice with each other at runtime, and trying to make a sane api around it that allows us to implement an FSP filesystem in Go.

I'm wondering if something couldn't be done to extend cgo-fuse to add dynamic-link support via build tags. However this still restricts us to FUSE.

---

On the topic of FUSE vs Native. While IPFS itself has POSIX constructs, there's nothing preventing us from reading and storing metadata like ADS, ACL, etc. alongside normal data. The practical utility of this is unknown though. It would be interesting to see a single file hash that contains file data as well as multiple sets of platform specific metadata. At the same time though, non-portable platform metadata should probably be avoided altogether by now. Hard to say if we should strive to support or ignore these things when dealing with a bridge between multiple systems.

I'm being extra cautious to harp on these details now, only to make sure we don't lock ourselves into something that will cause problems later. It seems like FUSE at least provides the necessities, but I want to make sure this is talked about first.

While not critical at the moment, I should also bring up performance as something to consider as well here. Will the lack of async i/o bite us?

---

On the topic of more build dependencies. I'd rather decrease the requirements to build on Windows rather than increase them, but if we decide cgo-fuse is the way to go, I won't be the one to object. Not to mention the possibility of a dynamic cgo-fuse varient.

[billziss-gh]

ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

I'm curious to hear more about this...

My recollection is that we were trying to create what golang calls "binary only packages", which were implemented in Go 1.7 (I think). While this experiment was successful the native toolchain was still required to build the final executable of a project that uses cgofuse (perhaps only the native linker was required).

Ping @ncw who may remember more on this. Nick, the question is if you recall what the issues with using cgofuse as a binary only package were.

I'm wondering if something couldn't be done to extend cgo-fuse to add dynamic-link support via build tags. However this still restricts us to FUSE.

I have considered before the possibility of modifying cgofuse to use the syscall.LoadLibrary approach on Windows to hook into the WinFsp-FUSE interface and avoid using the C compiler on that platform. I did not follow through because the Go libraries lack good dlopen support for non-Windows platforms, which made the use of cgo mandatory.

It may still be worthwhile to do this on the Windows platform only.

Another approach (and perhaps the one you allude to @djdv) is to build cgofuse as a shared library or a plugin. Unfortunately buildmode=shared and buildmode=plugin are still not supported on Windows (I think).

While IPFS itself has POSIX constructs, there's nothing preventing us from reading and storing metadata like ADS, ACL, etc. alongside normal data.

This is actually related to something I deeply care about: how to streamline the differences between the 2 major approaches to file systems (POSIX and Windows). ACL's are a big part of that.

For example, I would love to be able to set ACL's on Windows and have them easily translate into ACL's on my Mac, especially because the ACL systems are conceptually compatible. The biggest hurdle I usually face is the need for a user name/id mapping service, but it sounds like IPFS may already have that problem solved.

While not critical at the moment, I should also bring up performance as something to consider as well here. Will the lack of async i/o bite us?

In general I have not found the lack of async I/O in the FUSE interface to be a performance problem.

One complaint from some users with heavy-weight scenarios, is that their file system may sometimes stall while doing too many blocking operations, because they run out of threads. The remedy is usually to increase the number of threads; some even go as far as to implement asynchronous I/O. Only a small number of commercial users with highly parallel file systems have had the need to implement asynchronous I/O.

Increasing the number of OS threads is not very costly in WinFsp (in terms of context-switching), because it uses IOCP scheduling under the hood. More details in this document.

[billziss-gh]

In the last 2 days I spent considerable time refactoring cgofuse with the goal of eliminating the need for cgo on Windows. My intent was to allow building with either CGO_ENABLED=1 to build the (default) "cgo" version of cgofuse, or CGO_ENABLED=0 to build a "nocgo" version of cgofuse.

My thinking was that with the help of syscall.DLL, syscall.Proc and syscall.NewCallbackCDecl, I should be able to replace/rewrite all Windows related C code in cgofuse. I was indeed successful in this endeavor (see the windll cgofuse branch), but unfortunately this story does not have a happy ending.

A FUSE file system needs to be able to receive file system requests (fuse_operations). In the "nocgo" version of cgofuse I setup fuse_operations using syscall.NewCallbackCDecl. These operations will be invoked in the context of threads that were NOT created by the Go runtime. For example, the very first operation init is called in a special thread that WinFsp-FUSE creates for initialization purposes. Other file system operations will be invoked on threads created by the WinFsp-FUSE dispatcher.

This all works fine in the "cgo" version. However the initial init call hangs in the "nocgo" version. It turns out that the Go runtime has a hard bug: golang/go#6751. According to that bug report callbacks created with syscall.NewCallback will hang the runtime if they are invoked in a non-Go thread. This bug spells doom for the "nocgo" version of cgofuse.

Even worse it spells doom for any effort that wishes to eliminate the need for cgo (regardless of whether cgofuse is used), because syscall.NewCallbackCDecl is required to receive file system operations when cgo is not used.

(There is also the alternative of rewriting the WinFsp DLL in Go, so that it interfaces directly with the WinFsp FSD, but this is a huge undertaking and I would not recommend it.)

[djdv]

In the last 2 days I spent considerable time refactoring cgofuse with the goal of eliminating the need for cgo on Windows. I was indeed successful in this endeavor

This is great to hear! As always, the continued effort is appreciated.

...

It's unfortunate that we're blocked on Go itself, but that seems to be a trend for our Windows tasks recently. Since we can't go ahead with either proposed solution, I suppose the only option is to try and resolve the issue upstream. However, I haven't gauged the complexity of it yet. Dipping into the Go runtime and how it interacts with the C runtime and threads between them, seems like it could be fun(a hassle).

Based on our needs and what was said, it seems like cgofuse (sans cgo) would be a good solution after the fix is implemented. This will halt efforts on the winfsp native bindings unless we encounter problems with the fuse bindings.

[ncw]

@billziss-gh wrote:

My recollection is that we were trying to create what golang calls "binary only packages", which were implemented in Go 1.7 (I think). While this experiment was successful the native toolchain was still required to build the final executable of a project that uses cgofuse (perhaps only the native linker was required).

As far as I can remember we were trying to make binary blob you could just link using the go linker so with the C parts of it already linked in. I can't remember why it didn't work though and I can't find any notes I made either :-(

This all works fine in the "cgo" version. However the initial init call hangs in the "nocgo" version. It turns out that the Go runtime has a hard bug: golang/go#6751.

:-( I've found the go team very good to work with so if you had the time to track the go bug down you would have a receptive audience!

[djdv]

suppose the only option is to try and resolve the issue upstream.

Correction, it seems like we have 2 upstream options. There is handling the thread problem, but it may also make sense to handle the plugin problem mentioned earlier (https://github.com/golang/go/issues/19282).

If I understand correctly, I believe this would sidestep the issue. cgofuse would still use cgo, but we would at least be able to link with it, without adding any build dependencies. We could just check for the existence of the plugin and either link with it or not.

I'd still prefer to take advantage of the cgo-less version of cgofuse if we can though.

[billziss-gh]

@ncw wrote:

As far as I can remember we were trying to make binary blob you could just link using the go linker so with the C parts of it already linked in. I can't remember why it didn't work though and I can't find any notes I made either :-(

My understanding (based on assumptions rather than facts) is that when you do not use cgo, go uses its own linker and avoids using the native linker. But when you do use cgo, the native linker must be used in order to correctly create the executable; the reason being that the standard C library has to be linked in which normally the go linker avoids to do.

The fact that go uses its own linker is probably why we do not have dlopen and why go makes its own system calls instead of going through the standard library (even on platforms like Darwin, where the system call API is not supported by Apple.)

Some relevant golang issue links are here:

• golang / go # 17490
• golang / go # 18296

(I have not linked directly to them to avoid adding extraneous references.)

Additional interesting links:

• https://blog.hashbangbash.com/2014/04/linking-golang-statically/
• https://groups.google.com/d/msg/golang-dev/0i_zJRJsmcY/trpufE_kBAAJ

[billziss-gh]

An update on this.

There is a PR (golang/go#25575) that addresses the thread hanging issue with syscall.NewCallback (golang/go#6751). It is currently under review by the Go team and should be hopefully accepted.

Over the weekend I was able to further develop the "nocgo" version of cgofuse, using a locally patched version of go. I was able to bring up the "nocgo" version of the 64-bit memfs sample and it worked without problems. There is additional work to be done for the 32-bit version and some more rigorous testing to be performed. But I am now fairly confident that this approach will work.

(The reason for the 32-bit vs 64-bit difference is that syscall.Syscall* and syscall.NewCallbackCDecl only allow/expect uintptr arguments, which are 4 bytes on 32-bit and 8 bytes on 64-bit. I have some int64 arguments that I need to pass, so on 32-bits I have to split them into 2 uintptr's to make the stack look right...)

[whyrusleeping]

@billziss-gh thanks for the update! and for pushing this so hard :) Its exciting to think that we might actually get decent windows filesystem support (something I had previously thought would never happen).

[billziss-gh]

@whyrusleeping thanks :)

BTW, I have finished updating cgofuse so that it now has cgo and !cgo variants. The updated cgofuse passes all tests on macOS, Linux and Windows. On Windows both cgo and !cgo variants are tested.

So we are now waiting on the Go team and hopefully an approval of golang/go#25575.

[whyrusleeping]

@billziss-gh any thought towards getting the Lock functionality working? That's my biggest complaint about our current fuse lib, it doesnt support file locking. We can also take this to a different issue if that makes sense

[billziss-gh]

The problem with user-mode locking is that it is not currently supported on OSXFUSE or WinFsp. So you have 2 out of the "3 OS'es" without locking support by default. This is why I have chosen not to implement it in cgofuse.

(To clarify file locking does work at the kernel level, it is just that it is not exposed to user mode. In practice this means that file locking will work for processes on the same machine, but it cannot be made to work for processes on different machines.)

Consider adding your voice at billziss-gh/winfsp#116. I actually have an unpublished implementation of user-mode locking support for WinFsp, but have not incorporated it into its public repo because of the problems documented in that issue.

[billziss-gh]

The PR golang/go#25575 has been merged into go as commit golang/go@bb0fae603bd19e096e38c3321d95bf114f40dcff.

[djdv]

Status update [5965a258f66b036bb819aa59a2d260452a2e7dd7]

https://www.youtube.com/watch?v=9cf2wKA3WMw

[billziss-gh]

Fantastic work! :tada: :tada: :tada:

[djdv]

Status update [4f078cc83a90bbee292736f80b31c3c5dc431a0d] Since last time, I've equipped the filesystem index, to account for arbitrary record modifications. If the state of a record has changed (either locally because of a fuse operation, or globally because of an IPFS operation), we need to be able to propagate that change, so that everyone who is currently accessing a record (via a handle), doesn't receive stale data. While avoiding the hard requirement that we perform an external request each operation.

The foundation in place should allow us to coordinate operations and data between systems and handles, in a way that lets us reduce our outbound requests, while still maintaining a reasonable level of global consistency.

I've posted an example here: https://www.youtube.com/watch?v=8JtzsjY0hNA which demonstrates a file being opened locally, changed globally, and then accessed again locally, with the operation reflecting the changed state.

In any case, records are split between their local (path) and global (cid) reference, so for N local-handles, there should always be 1 global reference, and thus only 1 lookup if required. To put this another way, you can imagine each of these paths resolving to the same CID. /ipfs/Qm..., /ipns/keyname, /mfs/string-name -> /ipld/cid Regardless of who requests the operation, the state will be refreshed, once, and be made valid again for everyone currently accessing that global reference.

Currently we're not aware of much in the global context, so metrics around it, are arbitrary. For now, IPNS and MFS records remain valid for some amount of realtime before they are rendered invalid. So subsequent operations on the same record may or may not incur a global lookup. Anything before the threshold returns instantly, but has the potential to be stale.

It is possible for us to keep records alive until they are explicitly invalidated, getting the best of both worlds, however, this would require some form of synchronizing with the node. A separate issue is somewhat related to this: https://github.com/ipfs/go-ipfs/issues/5221#issuecomment-417489617

For the local state, I'm intending to signal on operations such as write and delete, so that we can update/invalidate records, exclusively and immediately.

[djdv]

Brief status update, I implemented basic write support for MFS. No commit yet as I have some issues to work out with it, but a lot of programs seem happy operating in that root. https://www.youtube.com/watch?v=PytWEw9aFjk

[billziss-gh]

@djdv this is progressing very nicely. Good work!

[billziss-gh]

BTW, if you need any help/pointers testing the beast rigorously using file system test suites ping me.

A good starting point are the test suites that cgofuse uses:

• Windows: winfsp-tests, fsx
• macOS/Linux: fstest, fsx

Fsx is invaluable at getting I/O (read/write) correct. Winfsp-tests and fstest test for correct semantics (e.g. file permissions, access control, etc.) so they may or may not work well for IPFS.

[djdv]

@billziss-gh Thanks 😊

I almost have a basic implementation for each operation on each root-type, so I've been thinking about ways of checking their correctness recently. I'll for sure look into these.

so they may or may not work well for IPFS

I'm considering how best to handle systems like that. I would like to implement all operations where possible so that we can play nicely with things that expect them.

My current approach for filling in fields that are absent on IPFS objects themselves, is to maintain a non-persistent store in memory, with defaults if there's no secondary record. For example, we may default to 0775 permissions, but users should be able to call Chmod and Chown to change them, and they should persist at least during runtime. These would have to be stored separate from the record itself, since we don't expect our primary metadata records to stay active past requests/handle-sessions.

Same for times and Utimens. Currently I have ctime and atime set to Now() at cache/object initialization, but programs like touch update the atime as expected.

While it does't make so much sense to IPFS, I'd like the bridge to still be there for the OS.

This seems more useful to people who intend to fork, than it is useful in its own right, if people want persistent systems around these, they should be able to swap out only how they're populated, and the rest of the file system should already expect and account for them.

The goal I have in mind is to have (at least) the local/MFS root be as spec-correct as possible, so that programs can live and run there without issues. This seems doable, but I'll have to see what complications arise in testing ;^)

[djdv]

Hello all, sorry for the silence. I meant to post a working status update earlier, but unexpectedly got kidnapped for a bit. 👀

I've updated the branch with the latest WIP https://github.com/ipfs/go-ipfs/commit/ec5978c3c0413ed40ef0c4fecf6cb6f358bdae74

This is generally just more refactoring, to improve stability and attempt to reduce the complexity. Recently, I've experimented with the CID caching system mentioned above and it seems to have made perceptible improvements in operation. And generally tried to reduce the complexity of the index lookup. There is still more to be done here but in performance and stability, this seems better than the previous commit. I'm encountering an issue at the moment around deleting directories from MFS. Rarely, on slow (network attached) media, a directories may only be partially deleted (without reporting an error).

The current goals are to continue to be more POSIX compliant where possible, rework how non-ipfs metadata (permissions, timestamps, etc.) is stored, and properly handle the current CLI problems (things like ipfs daemon --mount don't work, no way to gracefully un-mount right now either)

In addition, Windows users have been requesting builds from me, so I've started hosting them here: https://ipfs.io/ipns/12D3KooWNDi7V2xYU6Xtq6Ef913EfwcUvSGgVPP4RtQwML6TD45u (currently: https://ipfs.io/ipfs/QmV7UcN88czNxYKkGFHoDAjr6YXadcH2b9njnHGXJexGZL) Availability depends on my node being up, so if it's down, try again later ;^)

[xelra]

@djdv Mapping POSIX to Windows permissions is an impossible task to do. Windows users usually expect their files to be fully available to the user account they're using and that's pretty much it. Translating any Windows-side permissions changes or complex permission schemes that someone would use in a Windows server environment isn't something that can be done without pushing assumptions onto users that almost certainly won't fit people's use cases.

Something that can be done is having Windows-specific mount options that determine what POSIX permissions (umask) a newly created file should have. Other than that I'd just keep permissions of existing files the way they are. Of course this also calls for commands for POSIX filesystem manipulation that will enable Windows users to change permissions of files outside of the context of their mount and operating system.

[Stebalien]

That's probably not going to be an issue in IPFS because IPFS doesn't support complex permissions (yet, at least).

[djdv]

While not fully thought out, the thing that I've been considering, is storing metadata client side. The data could be stored in some datastore, in a format that we define.

If we have the opportunity to store arbitrary data, we can just add parsers as necessary for various fields/formats, that can store and manipulate this data into a common set. Ignoring and/or translating it on platforms that don't understand it.

For example, permissions would be inferred at runtime, if nothing is stored; defaults for the environment would be provided. However, upon specific calls, changes would be stored locally, so effects will persist on subsequent runs.

For instance, storing a file's ctime when mfsMknod is called, or permission bits for Chmod. If the system calls are being made, it's reasonable to believe the data may be important to the caller, so we should likely store it and make the effects persist through multiple instances.

@warpfork may also have input on this topic. ^

If we have a way of dealing with existing permissions, we have a foundation to add support for other platforms, or metadata of our own, through the OS interface, where it is applicable.

Opinions on any of this are welcomed.

[xelra]

My personal use-case goes specifically in the direction of private networks and ipfs-cluster. Replacing DFS in a workplace environment. That means multiple clients are going to be hooked into the same data and there needs to be a way to make permissions work across all of them. I also think that's the usecase of most corporate setups where ipfs has been eyed as a promising replacement for much more complicated setups with AndrewFS or GlusterFS or Hadoop WANdisco.

This stuff somehow needs to be woven into the filesystem. I don't think local solutions are going to work. Or maybe a "local solution" that can be run as a layer on top of ipfs. A second daemon that just handles metadata across linked clients. That's how other DFS solutions have tackled this problem also. This brings consistency issues though. Because the system needs to make sure that you don't have metadata of data that isn't yet available to you. After file creation or after updating files and metadata.

In XtreemFS you have a separate server just for metadata and it's similar with other DFS.

The big selling point of ipfs as a DFS is its simplicity. So tackling "the metadata server" in a decentralized ipfs-way also needs to be straightforward or the whole point is moot.

EDIT: Shamelessly stolen from the XtreemFS project, just for illustration: Here ipfs would be the lower arc.

EDIT2: So the idea is that when you want to open a file, you ask the node that has the file if he also has metadata for it. And then you fetch the metadata before or along with the file.

EDIT3: The question is whether this is possible with a decentralized system. There's always the possibility that you will get outdated data. At least the system must make sure that you get the outdated data along with the proper outdated metadata. Then it's simply a propagation issue.

[Stebalien]

While not fully thought out, the thing that I've been considering, is storing metadata client side. The data could be stored in some datastore, in a format that we define.

That's probably going to confuse users more than it'll help:

1. Things will appear to work but they won't actually be replicated.
2. Files will appear to have some restricted permissions but they won't. That's a pretty large security concern.

We'll probably want some kind of ipfs add --archive flag that copies everything it can into IPFS (storing it in the file's metadata). This will most likely be a UnixFSv2 thing. However, this archived metadata can't actually affect the operation of IPFS and probably shouldn't be exposed it over FUSE. Basically, this is an orthogonal issue.

How does windows handle removable drives? How do the EXT2 etc. drivers work?

That means multiple clients are going to be hooked into the same data and there needs to be a way to make permissions work across all of them.

So, we do need to make this work eventually but this falls under the heading of "private content". Unfortunately, the solution you're proposing won't work anywhere but in centrally controlled private networks (where users don't have administrative access to their own machines).

A more general, decentralized solution will require a decentralized notion of "identity". For convenience, we'd probably (eventually) provide a way to integrate these decentralized identities into directory services (e.g., Window's Active Directory). However, this is all a ways off.

[xelra]

Archiving metadata directly within ipfs would probably work for everything but ownership.

I think that it would make sense to go this route if kept strictly opt-in. Even exposing and manipulating the metadata through the FUSE mount should be fine.

Through ipfs --archive one would get permissions, attr and xattr. The caveat would be that ownership doesn't make sense. But that's not an issue, because ownership is handled through mounting via FUSE.

The bigger questions are:

• How does the ipns node update?
• Who can write or publish metadata?
• What about file locks?
• What about multiple users editing and creating conflicts?
• What about trusted vs non-trusted environment?
• How can this archive and metadata opt-in system work for global ipfs and how and why would it make sense?

[Stebalien]

The caveat would be that ownership doesn't make sense. But that's not an issue, because ownership is handled through mounting via FUSE.

Ah, sorry, I may have misunderstood the issue. I was thinking permissions as in ACLs as in ownership.

Non-ACL permissions and extended attributes should probably be preserved as much as possible, by default. That's something we probably can do although translating between unix/windows will be tricky. At the end of the day, we'd probably expose them as much as possible if the current platform supports them.

As you say, the hard part really is ownership.

[djdv]

🎅 A buggy but quick holiday update! We're getting there! https://www.youtube.com/watch?v=Xhjg63DUwPc

I'm reworking how handles are managed and implementing more specific "File" and "Directory" types on top of the generic "Record"s . Storing data where appropriate instead of generating it dynamically upon each request. A good example of this is storing the actual IO Golang interfaces between operations, instead of reconstructing them each request. This cuts out a lot of back and forth between mount and the API node. At the cost of bringing some more (sometimes redundant) responsibility into the local scope.

Operations are much much faster, but I'm encountering some critical problems at this time. Binaries won't execute, and something may be wrong with the cache library I'm using, or my path parser pipeline (or both).

I expect to find the source of the problems as the refactoring continues. But no source or binaries for now as things are broken and changing.

[billziss-gh]

@djdv a couple of considerations for running executables on Windows (I watched your video):

• Ensure that the exec bits are properly set in Getattr.
• Windows runs executables by memory mapping them. This should not however make any difference in a FUSE file system.

[djdv]

Binaries updated: https://ipfs.io/ipns/12D3KooWNDi7V2xYU6Xtq6Ef913EfwcUvSGgVPP4RtQwML6TD45u (currently: https://ipfs.io/ipfs/QmWeKT2hBz2Pz62WKyVS82L2ekFGBxGqQtUWcZZZBL8YvM)

@xelra, @Stebalien Just as an aside, I've tried to keep a hard separation of "network" stats and "local" stats. FS nodes try to store a single copy of their network stats, and everything else is computed dynamically on request. Such as the permission bits, access times, etc. https://github.com/djdv/go-ipfs/commit/9ceec2019fc6df4639aa6a9b50c1f3bd29bf1564#diff-0f550ead73a2dd919b337c43ab97eaabR121 The hope was that these should be easily pluggable. In theory we could pull the bits from anywhere at any point. i.e Getting a paths owner from a datastore, some metadata-node, or even from a dag. For now it's more or less just debug values to satisfy the interface. (Since there's no such data to actually retrieve currently. IPFS has no concept of an owner at this point)

The XtreemFS style was what I had in mind, but I was thinking of developers who would fork this and plug those values in as needed. If we want to do anything specific in the mainline branch, we should be able to.

---

Thanks @billziss-gh It turns out I implemented Read() incorrectly. I was treating the offset value as relative instead of absolute. Now read calls with offset values should succeed again. I pushed the changes to the usual branch https://github.com/ipfs/go-ipfs/compare/master...djdv:feat/win-mount

Programs are still having some issues that I need to look into more. (vidoc) My hunch is that something is wrong with Write() or Open(). For Open() I neglected to respect this

return...representing the lowest numbered unused file descriptor

during the refactor. For the moment, I'm just using the address of a handle struct as a handle int which isn't ordered at all.

Could also be seek and/or sync issues with Write() too.

There is a permission oddity that I've ignored up until now as well. The mount is being called with these args

fsh.Mount(mountPoint, []string{"-o uid=-1,gid=-1,pid=-1,fstypename=IPFS"})

However this sets the owner to what looks like a literal -1. On Windows the owner appears as (S-1-0-65534). But does add my account to the security entries with special permissions. 0775 seems to map to this which hasn't given me much issue. But does seem incorrect.

When the options are omitted

fsh.Mount(mountPoint, []string{"-o fstypename=IPFS"})

Everything is the same, except my account is missing from the security entries.

For what it's worht DIR /Q is saying the owner is ...

For now I'll be checking traces to try and find the problem area.

[billziss-gh]

For Open() I neglected to respect this

return...representing the lowest numbered unused file descriptor

during the refactor. For the moment, I'm just using the address of a handle struct as a handle int which isn't ordered at all.

I do not believe this matters much for FUSE in general. It certainly does not matter at all for WinFsp (i.e. there is no order required for file descriptors).

The mount is being called with these args

fsh.Mount(mountPoint, []string{"-o uid=-1,gid=-1,pid=-1,fstypename=IPFS"})

However this sets the owner to what looks like a literal -1.

The pid=-1 part is not needed (at least for WinFsp). The uid=-1,gid=-1 parts are correct and should correctly set the owner/group for all files to be the current user and group.

What do you get if you run the command: fsptool-x64 id? The fsptool-x64.exe program can be found in the \Program Files (x86)\WinFsp\bin directory.

[billziss-gh]

I wrote:

The pid=-1 part is not needed (at least for WinFsp). The uid=-1,gid=-1 parts are correct and should correctly set the owner/group for all files to be the current user and group.

Also this assumes that you run the file system from a command prompt under the current user account and not using the Launcher or an Administrator elevated command prompt. Is that the case?

[djdv]

@billziss-gh

I do not believe this matters much for FUSE in general.

Good to know. The old implementation worked that way by coincidence, but this new version doesn't. I've been reading POSIX documents lately and erring on the side of POSIX when unsure about FUSE requirements. But if we don't need to be as strict, even better. :^)

I'll have to trace the Write() calls some more.

What do you get if you run the command: fsptool-x64 id?

For reference I'm using the latest WinFSP beta (2018.2 B4) but I also had this issue on the latest stable (2018.1) and older versions as well.

User=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (uid=197609) Owner=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (uid=197609) Group=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (gid=197609)

---

run the file system from a command prompt under the current user account Is that the case?

Typically, yes. Perhaps the IPFS daemon is at fault here, although it should just be a background process of the same user/SID.

For what it's worth, I'm seeing IRP_MJ_QUERY_SECURITY calls getting a return of STATUS_BUFFER_OVERFLOW for the binary I've been testing. I'm assuming that it sees the high UID/GID in the IRP_MJ_QUERY_INFORMATION and decides to trigger cleanup, but I'm not actually sure what is responsible for this. Edit: Actually this looks like it's just done reading the file into memory so it decides to close. But it does still provide an overflow.

2866    16:41:27.521        jelly.exe   IRP_MJ_CREATE   00000884    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  FILE_OPEN CreOpts: 00000060 Access: 00120089 Share:  00000001 Attrib: 0 Result: FILE_OPENED
2867    16:41:27.522        jelly.exe   IRP_MJ_READ 00060900    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  Offset: 00000000-00000000 ToRead: 40 Read: 40 
2868    16:41:27.522        jelly.exe   IRP_MJ_READ 00060900    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  Offset: 00000000-00000118 ToRead: F8 Read: F8 
2869    16:41:27.523        jelly.exe   FASTIO_QUERY_STANDARD_INFO      FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  FAILURE 
2870    16:41:27.523        jelly.exe   IRP_MJ_QUERY_INFORMATION    00060870    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  FileStandardInformation AllocationSize: 00000000-00288E00 EndOfFile: 00000000-00288E00 NumberOfLinks: 1 DeletePending: FALSE
2871    16:41:27.523        jelly.exe   IRP_MJ_CLEANUP  00000404    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  
2872    16:41:27.523        jelly.exe   IRP_MJ_CLOSE    00000404    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  
2873    16:41:27.523        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_BUFFER_OVERFLOW  
2874    16:41:27.524        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  
2875    16:41:27.524        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_BUFFER_OVERFLOW  
2876    16:41:27.524        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  
2877    16:41:27.524        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_BUFFER_OVERFLOW  
2878    16:41:27.524        jelly.exe   IRP_MJ_QUERY_SECURITY   00000000    FFFFC686333E4910    \Device\Volume{c763a4fb-0956-11e9-877b-1831bf516fbe}\files\jelly1000\jelly.exe  STATUS_SUCCESS  

[billziss-gh]

For reference I'm using the latest WinFSP beta (2018.2 B4) but I also had this issue on the latest stable (2018.1) and older versions as well.

User=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (uid=197609) Owner=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (uid=197609) Group=S-1-5-21-1034095386-1200325712-4187133966-1001(DESKTOP-S8T7UIT\Dominic Della Valle) (gid=197609)

Interesting! For some reason your user and group id are the same, which is not usually the case. This is related to billziss-gh/winfsp#204.

If your UID and GID are the same you can have permission problems, because WinFsp chooses the most restrictive permissions:

https://github.com/billziss-gh/winfsp/blob/v1.4B3/src/dll/posix.c#L471-L478

I may have to address this somehow in v1.next, but in the meantime you may be able to launch the file system with -o uid=-1,gid=11, which will set the group as "Authenticated Users".

For what it's worth, I'm seeing IRP_MJ_QUERY_SECURITY calls getting a return of STATUS_BUFFER_OVERFLOW for the binary I've been testing.

The STATUS_BUFFER_OVERFLOW is fairly benign in this case, because the IRP_MJ_QUERY_SECURITY is retried afterwards and results in STATUS_SUCCESS. STATUS_BUFFER_OVERFLOW simply says to retry with a larger buffer because the security descriptor cannot fit in the provided buffer.

[djdv]

Less formal than usual update:

https://youtu.be/FB9hzDY0njA

I tried investigating what is causing the hang in the test binary, but failed to find anything obviously wrong. Diverting from this for a minute, I added back in support for ipfs daemon --mount, and got things running on non-Windows platforms.

This exposed a strange bug where duplicate entries are being displayed sometimes. Which leads me to believe there is an issue around either the initialisation, or listing of directories. I'm glad to see a visible problem instead of silent failure, now I have something to target that isn't just verbose trace logs.

@billziss-gh Ahh, interesting. Thanks for the information.

but in the meantime you may be able to launch the file system with -o uid=-1,gid=11, which will set the group as "Authenticated Users".

This seems to have taken effect for the group, but the owner is still incorrect.

[djdv]

Minor update, I was not handling Readdir()'s offset parameter properly, This was the cause of duplicate entries on other platforms. 77bce1e84446eb8b4a657596ac47416f5f2f116d

Still investigating other issues.

[magik6k]

Got this panic when not really doing anything:

panic: sync: negative WaitGroup counter

goroutine 149078 [running]:
sync.(*WaitGroup).Add(0xc0003765c0, 0xffffffffffffffff)
    /usr/lib/go/src/sync/waitgroup.go:74 +0x137
sync.(*WaitGroup).Done(0xc0003765c0)
    /usr/lib/go/src/sync/waitgroup.go:99 +0x34
github.com/ipfs/go-ipfs/core/commands/mount.(*FUSEIPFS).dbgBackgroundRoutine.func2(0xc0003765c0, 0xc00084f100, 0xc0003765b8)
    /home/magik6k/.opt/go/src/github.com/ipfs/go-ipfs/core/commands/mount/event.go:54 +0x24c
created by github.com/ipfs/go-ipfs/core/commands/mount.(*FUSEIPFS).dbgBackgroundRoutine
    /home/magik6k/.opt/go/src/github.com/ipfs/go-ipfs/core/commands/mount/event.go:42 +0xeb