Virtual filesystem "mount" support from Lua?

[MCJack123]

I've worked on many projects that interact with the filesystem. While developing one of them (an FTP client), I wanted to be able to mount a remote filesystem as a folder underneath the root, that would dynamically update the contents of the folder with the responses from the server. However, this proved difficult to implement as I would need to set up an overridden fs API that would redirect any file accesses in the specified folder to the program, and have anything else call back to the original fs. This process is a) time consuming, b) dangerous if it goes wrong (could cause errors if not implemented correctly), c) would break things when a new function is added (such as fs.attributes), and d) would mostly be incompatible with any other program that shares the same goal.

I propose a new way to be able to mount virtual filesystems from Lua. A function called fs.mount could take both a path to mount to, and a table with the callbacks to implementations, and would create a mount that all fs functions can access as if it was any other mount, like /rom or /disk. This mount would use the functions in the table to implement things such as fs.list() or fs.open(), but only for the folder specified. Another function called fs.unmount would take a path and removes a mount previously added with fs.mount. There could potentially also be a fs.getMounts function, but this may not be necessary.

There are a few ways this could be implemented.

• The first and easiest way would be to use a setup like term redirects, adding an fs.lua file to rom/apis, which redefines the base fs set and inserts mount checks for each function. The table passed in would have the same contents as fs.

  
  local nativeFS = fs
  local mounts = { "mymount" = { --[[ ... ]] } }
  local function getMount(path)
  path = fs.combine(path)
  for k,v in pairs(mounts) do
      if path:find(path, mounts, 1, true) <= 2 then return v, fs.combine(path:sub(#k + 1)) end
  end
  return nativeFS, path
  end

function mount(path, tab) mounts[fs.combine(path)] = tab end

function list(path) local m, p = getMount(path) return m.list(p) end -- ...

* Another method could be to use the pre-existing `IMount` functionality on the Java side. This would save the hassle of having to redefine `fs` from inside Lua, and would reuse the existing mount code that's already been written. However, an extra Lua -> Java mount class would have to be written. This would help reduce the complexity of code for Lua implementors.
```java
public class LuaMount implements IWritableMount {
    private Map<String, LuaFunction> m_table;
    public LuaMount(Map<String, LuaFunction> tab) {
        m_table = tab;
    }
    public void list(@Nonnull String path, @Nonnull List<String> contents) throws IOException {
        LuaFunction fn = m_table.get("list");
        LuaValue tab = fn.call(state /* get this from somewhere! */, LuaString.valueOf(path));
        if (!tab.isTable()) throw IOException("bad return from 'list' (expected table)");
        LuaValue v = ((LuaTable)tab).rawget(1);
        for (int i = 1; !v.isNil(); v = ((LuaTable)tab).rawget(++i) {
            if (!v.isString()) throw IOException("bad value in table returned from 'list' (expected table)");
            contents.add(v.toString());
        }
    }
    // ...
}
// in FSAPI.lua
@LuaFunction
public final void mount( String path, Map<String, LuaFunction> tab ) {
    fileSystem.mountWritable("vfs", path, new LuaMount(tab));
}

Of course, the code samples above are very rudimentary and would need to be fleshed out much more, but that's the gist of how it would work.

This is a bit of a feeler post to see what people's opinions are about this. Adding native mounting support would allow programs to set up their own virtual file mounts without having to override the filesystem manually or conflicting with other setups. This would open the door to many programs such as temporary folders that delete on reboot, virtual hard disks, and remote folder sharing, among other things. However, it would add a bit of extra complexity to the fs API, and would break pre-existing mounting APIs that aren't ready (:P).

Thoughts?

[Lupus590]

Would be cool to be able to interact with the disk mounts that get automatically created when a disk is placed in a neighbouring drive.

[spr2-dev]

While we're at it, why not extend that feature to attached disk drives ? Not only it would be more consistent with the other half of the feature, but it could be a great opportunity to redo the whole system to mount disk drives dynamically in code. Being able to put custom mounting paths would be great ! No more fiddling with peripheral.wrap to get our disk drives to work.

The only issue I would see is for music discs and anything else than floppies being inserted into the drive.

[SquidDev]

Not really having any opinions here _{(future squid here - I lied)}, just giving some technical commentary

However, an extra Lua -> Java mount class would have to be written.

We currently don't allow calling Lua functions from Java. While this is definitely possible, I think it's a route we want to avoid going down.

Whenever you need to call a Lua program, you need to handle the case that it yields. If #535 ever happens (and the current API is very much designed to facilitate that), you also need to handle the fact that it yields, and then your computer is shutdown (and restored) This means you need to write the Java in such as way that it can be (un)serialized. And so on.

Much safer just to write it in Lua :p.

The first and easiest way would be to use a setup like term redirects, adding an fs.lua file to rom/apis, which redefines the base fs set and inserts mount checks for each function.

Agreed that this is the easier route. One thing to watch out for though is that Java needs to know about these mounts (as disk drives need to know whether to mount to disk, disk2, etc...).

Obviously we could expose some internal method which relays what Lua-based mounts are currently reserved, though it's a less than elegant solution.

[Lupus590]

One thing to watch out for though is that Java needs to know about these mounts (as disk drives need to know whether to mount to disk, disk2, etc...).

Obviously we could expose some internal method which relays what Lua-based mounts are currently reserved, though it's a less than elegant solution.

Could we have somthing like the rednet coroutine and have the disk mounts be managed on the lua side? Disk drive peripherals might need to change maybe so that they expose their own fs API, which means that things might behave kinda like how Windows does in having seperate drives.

[SquidDev]

Could we have somthing like the rednet coroutine and have the disk mounts be managed on the lua side?

Yes, though that'd require a breaking change in the CC API, so we'd need to sync this change with a major Minecraft version.

[Ocawesome101]

we'd need to sync this change with a major Minecraft version

Perhaps with 1.18 in December?

When you say a breaking change, do you mean to the Java API, or the Lua API?

[SquidDev]

I was planning to do this (or at least the preliminary work) for 1.19.3, as there's a whole load of other Java API changes in that version (see #1214). However, I've really struggled to find an API design which I'm happy with.

To start from the beginning, the "core" API mentioned in Jack's original comment is definitely the most intuitive and obvious one:

• fs.mount(path: string, mount: table): Add a mount
• fs.unmount(path: string): Remove a mount.

The tricky thing here is how to handle peripheral-provided mounts (i.e. disk drives). Currently when mounting a drive, we loop through disk, disk2, disk3, etc... to find a free location and use that. However, if the mounts are all handled in the Lua code, this is obviously much harder to get right!

The CC way to handle this would be to queue an event when a disk drive creates a mount, so something which looks like "mount", path:"disk", mount_object:table. CraftOS then receives this event, finds a free path (disk, disk1, ...) and mounts it (with some provision for user code overriding this behaviour).

However, disk drives need to know where they're mounted (for disk.getMountPath). This means we need a way to signal back to Java where the drive is mounted. This probably means a mount_object.setMountedAt(path: string) method (and then on the Java side, IComputerAccess.mount returns a handle which can read this mount path).

This is definitely a viable API, I'm just not very happy with it. It feels very complex, in a way which doesn't feel like idiomatic ComputerCraft. I don't know, do people have ideas?

[MCJack123]

I was thinking that instead of passing the mount in a new event, we could piggyback off of the existing disk and disk_eject events, and then use a drive.getFilesystem() method to get the filesystem table. We'd probably have to use that function to set the mount path though. (If it was only necessary to get the disk API working, then a simple shim could have been useful, but there's more than that that needs the path. There's also the possibility of a shim in peripheral.call, but that's more complex.)

Also, as for the ROM, it would be neat if this functioned in the same way - perhaps through a os.getROMFilesystem() function or something like that. Then the BIOS would start without the ROM mounted on the Java side, handling mounting the ROM itself. I know you've been trying to move fs stuff to an API file, but to load that, the ROM would already need to be loaded, which would require fs to be set up with mounts... however, we could swap around the load method to load all APIs directly from the ROM mount, and then proceed to mount the ROM before starting the shell. (Or we could just load fs first, then mount the ROM, then load all the other APIs.)

Mod mounts would be a bit tougher to move to Lua, but for peripherals we could use the same getFilesystem() method with a mount_object type. Normal mounts are probably not going to be added after boot (I hope), so some sort of function to expose those to Lua could be good... maybe put them in a os.getFilesystems(): {path: mount} sort of function, which would replace os.getROMFilesystem()? Ideally the user would never have to even see that these functions exist, so it wouldn't really be important to describe how it works far in detail - the mod simply calls IComputerAccess.mount, and that gets exposed on startup.

If we can move all of the mount stuff to the Lua side, then that could mean that we can remove the Java mount logic entirely, which could help clean some stuff up.

[SquidDev]

I think my worry with piggy backing on top of the disk event is that it then means CraftOS needs to be specifically aware of disk drive peripherals. If another mod wants to add an alternative storage medium, they wouldn't have a way to automatically mount their data.

We'd probably have to use that function to set the mount path though.

Yeah, this is really the bit I find ugliest in my proposed API. It's something I want to avoid, though it may be that this complexity is inevitable.

os.getROMFilesystem(), os.getFilesystems(): {path: mount}, etc...

This'd definitely be the plan. Though (as I think you're saying), these can just be an implementation detail and the bios can remove them before any user code actually runs.

[Ocawesome101]

Having CraftOS deal with mount paths entirely seems like the cleanest option here - though it might mean sacrificing some small bit of Lua API compatibility.

Just call it CC 2, and replace CraftOS with Recrafted! /s

In all seriousness it's quite impressive to me that old CC programs still work as well as they do. This seems like a small enough trade-off for a reasonable boost in flexibility?