Add support for loadable filesystems

[dirkwhoffmann]

This is a follow up to #651.

[dirkwhoffmann]

@mras0: I have a quick question about function handle_fs_initseg(). I understand that this function creates a linked list with hunk related data, but I haven't understood the details yet. My question is about the incoming pointer ptr_hold_. Does handle_fs_initseg() get a point to a native AmigaOS structure like handle_fs_resource() which gets a pointer to a struct FileSysResource? If yes, which structure is it?

[mras0]

struct fsinitseg {
    uint32_t hunk[/*max_hunks*/3];
    uint32_t num;
};

num is the filesystem being initialized (i.e. serves the same purpose as fsinfo_num in processInfoReq) hunk[0..fsinfo_numHunks] contains pointers to the memory allocated by the expansion ROM. EDIT: It's a structure specific to the expansion rom not a native structure (like the one used in processInfoReq)

For each file system that's to be initialized the process is:

• A FS info request is issued ($fee1) with fsinfo_num=which fs (index)
• Host fills in dostype, version, number of hunks and the memory flags (and size) for up to 3 hunks (looks like you're already doing this)
• The expansion rom allocates memory for the hunks (note: no error checking is done)
• A FS init seg request is issued ($fee2) with the above structure
• Host uses the allocated memory to do the work of "LoadSeg()"

Brief walkthrough of handle_fs_initseg:

• 1738: Extract fs number (i.e. which fs driver is being loaded)
• 1747-1751: Extract pointers to the allocated memory (and check for allocation failure)
• 1752-1765: Loop to handle the "LoadSeg()" part -> Builds seglist (i.e. afterwards you should be able to use OSDebugger:read(u32,os::SegList&) to get the expected result back).
• 1753: Looks like a bug on my part, should be ((fs.hunks[i].flags & 0x3FFFFFFF) << 2) + 8
• 1754: Link segments together (remember they are BPTRs)
• 1756-1757: Copy hunk data
• 1759-1764: Apply relocations (IIRC the FFS driver only has a single one - for the RomTag structure at the start)
• 1768: A BPTR to the segment list is saved for later use (when a partition that uses it is initialized - line 1407 in the same file) Hope that makes sense

EDIT2: Argh, ithould be +8 not -8 for the size. Doesn't matter too much, as the segments won't be deallocated anyway, but if you're trying to sanity check the result as I proposed you will notice.

[dirkwhoffmann]

@mras0: Again, thanks a lot. Great explanation!

I think I got it working. There is a new debug option NO_LOADABLE_FS to switch loadable fs support off and on. Having the option set to 0, test file pfs3test.hdf does not boot with Kick 1.3 (as expected):

With loadable file system support enabled, it boots just fine 😎:

There is more work to do on my side, though. E.g., I don't filter out unneeded file systems yet (like your code does). Right now, all drivers are loaded, no matter if they are needed or not.

BTW, is there an easy way in AmigaOS to see which device driver and version is used for a certain device?

[mras0]

Great!

There is more work to do on my side, though. E.g., I don't filter out unneeded file systems yet (like your code does). Right now, all drivers are loaded, no matter if they are needed or not.

Probably not a huge issue in practice, but shouldn't be too hard to implement. Note that I think my code is probably wrong if there a multiple versions embedded and they're in ascending version order. I also think that any handler for 'DOS\x' dostypes (for x>0, i.e. various FFS variants) can handler all other types, but I don't rely on that. Again probably isn't anything that'll matter in practice, it'll just result in some wasted memory.

In pseudo code it should probably be something like:

• Read all drivers
• Remove if the dos type is not used
• For each dos type, find highest version number and remove all others (maybe treat all 'DOS\x' as being the same)
• If HdController::processResource is called (will only happen if filesytem.resource is present and there are filesystems to install), remove any older handlers (in practice only relevant for 'DOS\x'). The expansion rom makes sure to check the number of filesystems to install again after doing (processResource [0xfee0]).

BTW, is there an easy way in AmigaOS to see which device driver and version is used for a certain device?

Not that I'm aware of, but I can whip together a small program to check. Stay tuned.

Now a question from me: I updated vAmigaSDL to use latest dev branch, and my previous method of attaching HDFs no longer works as expected: amiga_.configure(OPT_HDCCONNECT, hd, 1); // hd = 0..3 amiga.hd[hd]->init(HDFFile(argv[i]));

it seems to always attach the default (empty, 10MB ofs image) as DH0 (or something, haven't looked too closely yet). What's the way to do it now?

(there are also two warnings when compiling under MSVC in debug mode, you may want a github action for that as well, but no big deal)

[dirkwhoffmann]

What's the way to do it now?

Strange. The old way is still the way to go. amiga_.configure(OPT_HDC_CONNECT, hd, 1) calls HardDrive::connect() which looks like this:

HardDrive::connect()
{
    auto path = writeThroughPath();

    if (!path.empty()) {

        try {

            debug(WT_DEBUG, "Reading disk from %s...\n", path.c_str());
            auto hdf = HDFFile(path);
            init(hdf);

            debug(WT_DEBUG, "Trying to enable write-through mode...\n");
            enableWriteThrough();

            debug(WT_DEBUG, "Success\n");

        } catch (VAError &e) {

            warn("Error: %s\n", e.what());
        }
    }

    // Attach a small default disk
    if (!hasDisk()) {

        debug(WT_DEBUG, "Creating default disk...\n");
        init(MB(10));
        format(FS_OFS, defaultName());
    }    
} 

The write-through stuff is new, but it shouldn't be the issue in this case.

Is it possible that HardDrive::init(const HDFFile &hdf) throws an exception for the HDF? It would explain why the default disk remains attached.

If you send me the HDF (via mail), I can double check if it works here.

[mras0]

Ah, it's because I'm testing with KS1.2 (my old nemesis). Not using any HDFs you don't already have, but KS1.2 + one (OFS) HDF should show the issue for you as well.

processInit is called twice in this case (sorry, I didn't mention that), and the numPartitions increment should only be done the first time.

Basically for KS1.2 the filesytem drivers can't be initialized the same way as for KS1.3+, so I do another run of all partitions to properly initialize them (I only really need the partition name, e.g. "DH0", but use the init function again to get it).

[mras0]

is there an easy way in AmigaOS to see which device driver and version is used for a certain device?

Turns out there's not an easy way to determine it (all that is known is which seglist or handler is used), but I've thrown together a small program that uses some heuristics to determine which driver is in use: hdtest.zip (also shows which are available in filesystem.resource to help show if too many are loaded unnecessarily).

BTW I think I've written earlier that the PFS3 disk doesn't work with KS1.2, it does now for me, but I've also made some changes (in my own repo) to the expansion ROM (mostly to support shared folders - works, but is still experimental). It's a really niche case, so probably not worth updating just for that, but just FYI.

[dirkwhoffmann]

it does now for me, but I've also made some changes (in my own repo) to the expansion ROM

I've updated the driver in vAmiga. Now, it boots here, too.

hdtest.exe is also working 😎:

[mras0]

Things also seem to work fine on my end in vAmigaSDL after your KS1.2 fix.

I'm experimenting a bit with exotic configurations, but there really aren't that many embedable filesystems out there. SmartFileSystem requires 020 (and KS2+) so it's not an issue for either of us at the moment (lucky, because it has 10 hunks). Fat95 says it's compatible with KS1.3, but as embedded FS it doesn't seem to work in KS1.3 and isn't really workable (I see the same thing in WinUAE, so I don't think it's really meant for that kind of thing).

Here's the HDF, just 4 empty partitions with a crazy setup: many.zip with a number of FS drivers embedded (Note again: guru in KS1.3 also happens in WinUAE). Seems to be handled as expect by very latest vAmiga dev branch at least.

Overall seems like you've bagged this one (until user complaints start at least :)). 👍

Final note: The updated expansion rom you've included contains code I use for the experimental shared folder support. It shouldn't affect anything you're doing (and expansion rom part is probably complete), but note that it's critical that HdController::spypeek16 returns 0 for EXPROM_SIZE + 6. You're already doing that, but it might be a good idea to add that as special case.

[dirkwhoffmann]

v2.0 is online.