mount: support IPFS' MFS

[djdv]

We have an old implementation of this here that was written before Go's fs.FS interface. This needs to be updated for UFS 1.5 and since we're no longer directly within go-ipfs, we'll need to update it to use the client APIs.

Last time I checked, the API does not allow overwriting the root node like we could do before. I think we'll also have to do synchronization manually (instead of being able to pass in a callback that's triggered on modifications).

With these limitations, I'm thinking about just creating a mount subdirectory within the MFS root, and treating that as our mutable root instead of the actual root (since we can't tell the IPFS node to replace it as-is). Maybe allowing this to be changed with parameters so that users can mount arbitrary directories within the MFS root.

[djdv]

Preliminary support for this was added in caa06f58819863270f4994701a81eabce2680090 of a development branch (j/ipfs-fs-extensions). These will be broken out into a PR sometime later.

In that branch, our mounting utility will look in an IPFS node's Files API root, for a directory named /mount (creating it on the node if it doesn't exist), and mount it on the host.

• Right now, only read methods are implemented and wired up, but they seem to work.
• File creation and write methods ported over from the old prototypes are in that branch, but need to be reviewed, and wired up to the host API bridges.
• The biggest problem we face, is that the IPFS node does not have any kind of notification system.
  • This is something we'll need to coordinate with upstream about. Or otherwise have some out-of-band hacks to make it work.
  • We basically get a snapshot of the Files API directory at the time of mounting it. If something changes on the remote, we won't be made aware of that, and thus, our mountpoint won't reflect that. Likewise, this means when write methods are implemented, we'll just clobber any external changes, with our own modifications.
  • The easiest thing to do in the meantime is to just make users aware of this and tell them to not make calls to the node's Files API while its mounted. go-ipfs itself does this for IPNS, locking the whole API while its own mount implementation is in use. Neither should be necessary with a proper resource locking and/or notification API on the node's side. The prototype code I had for writable IPNS mounts did this internally. When users would publish from the API, the mount logic would be made aware of it and could update its references as a result. This could be done via some pubsub thing for external clients. We'll have to investigate and flesh this out at some point.
• There should also be a CLI flag to specify which Files API directory to use as the mountpoint's root. Mounting multiple directories should be fine, but I'm wary of synchronization problems, especially with nested directories. Like mounting /a and /a/b at the same time. This seems likely to deadlock somewhere in the MFS API, and/or desync.
• The node's actual Files API root could be mounted as read-only, but the Files API does not currently expose any way to update/replace the root CID, so we can't mount it with write access. (We can, and making writes would sync blocks back to the node's DAG service, but the root CID of the Files API wouldn't reflect those changes.)
  • This might be something worth asking upstream about. Being able to swap CIDs (atomically) would be nice even for subdirectories of the root.
  • I haven't tested it yet but I'm worried about the write performance when using the Files API. The prototype had direct access to the IPFS node's root CID and just updated it directly. Since we're now a remote client of the node, we're bound by its public API. So the current strategy we'll use for syncing the root directory is a typical (but not ideal) shuffling of references. (basically cp $new-root-cid /stage; mv /current-root /temp; mv /stage /current-root; rm /temp.) These should be quick reference updates, but there could also just be an API endpoint that allows us to replace-root $newCID; where the IPFS node atomically handles this internally and safely.