Booting from FAT12 disk images

[ghaerr]

I'm not sure where to comment on MSDOS FAT12 issues, so I opened this.

@tkchia: Here's a link to the source code for the MSDOS 3.3 boot sector (http://read.pudn.com/downloads3/sourcecode/5346/SOURCE/SRC/BOOT/MSBOOT.ASM__.htm). Following is an extracted section.

I'm suggesting we use almost this entire block, from $START up to but not including CURHD. We can use the OEM label portion for our boot code, instead of jumping immediately to START, as this field isn't used in filesystems. However, our max heads and max sector count should be put in the HDLIM and SECLIM fields. Doing this will make our MSDOS boot compatible with the mtools utilities, which are well tested and used in my last PR for building correct MSDOS images.

The correct values for the other BPB fields are parameters in the package management PR, which are passed command line arguments for mformat. I will submit a PR for these values once this MINIX boot is updated.

I can start on modifying mfs to create a new, or update an existing, boot block with the ELKS head and sector limits once there is agreement on this.

Public $START $START: JMP START ;---------------------------------------------------------- ; ; THE FOLLOWING DATA CONFIGURES THE BOOT PROGRAM ; FOR ANY TYPE OF DRIVE OR HARDFILE ; DB "MSDOS" ; This section can contain ELKS boot code since DB "3.3" ; we only have 6 bytes free. ByteSec DW cbSec ; SIZE OF A PHYSICAL SECTOR set to 512 DB 8 ; SECTORS PER ALLOCATION UNIT set to 1 cSecRes DW 1 ; NUMBER OF RESERVED SECTORS cFat DB 2 ; NUMBER OF FATS DirNum DW 512 ; NUMBER OF DIREC ENTRIES I think this will be 244, will check DW 417305-1 ; NUMBER OF SECTORS - NUMBER OF HIDDEN SECTORS MEDIA DB 0F8H ; MEDIA BYTE cSecFat DW 8 ; NUMBER OF FAT SECTORS change to 9 SECLIM DW 17 ; SECTORS PER TRACK ELKS max_sect here HDLIM DW 4 ; NUMBER OF SURFACES ELKS max_heads here cSecHid DW 1 ; NUMBER OF HIDDEN SECTORS dw 0 ; high order word of Hiden Sectors dd 0 ; 32 bit version of NUMBER OF SECTORS ; (when 16 bit version is zero)

db  6 dup(?)        ; reserved for later expansion *** volume serial number and some other stuff is written here by the mtools mformat command, so we should keep this for ELKS.

*** ELKS doesn't need fields after this

CURHD DB ? ; Unitialized

; this is an image of the disk parameter table. Zero entries are copied ; from the rom table at boot. ; SEC9 DB 0 ; DISK_SPECIFY_1 DB 0 ; DISK_SPECIFY_2 DB 0 ; DISK_MOTOR_WAIT DB 0 ; DISK_SECTOR_SIZ DB 12h ; DISK_EOT DB 0 ; DISK_RW_GAP DB 0 ; DISK_DTL DB 0 ; DISK_FORMT_GAP DB 0 ; DISK_FILL DB 1 ; DISK_HEAD_STTL DB 0 ; DISK_MOTOR_STRT

[ghaerr]

@tkchia: I was reading over the MSDOS 3.3 boot source and a hunch you made earlier is shown to be correct: at the end of the listing, it shows that the physical drive (PHYDRV) needs to be reserved (0) in order for MSDOS FORMAT and SYS commands to work. So you might want to reserve that as well at 0x1FD. That will ensure full compatibility should someone run one of those DOS commands on our ELKS boot disk lol!

That means you're down to only 5 bytes left in the boot block!! Keep up the good work :)

[ghaerr]

@mfld-fr: I'm not sure who is going to write the MSDOS boot payload loader (I can if needed). Are you thinking that we just write /linux to a:\linux and then the second stage loader will read that image into memory at the same locations as MINIX and proceed?

Where will we be reading the second boot sector from on MSDOS disks? We could set the # reserved sectors to 2 on the mformat, which should allow it to be the second sector on disk... that would be tremendously simpler than having to find it elsewhere, but I haven't tested that yet. We could set the parameter "-R" to be 2, and see whether DOS will read the disk. (See elks/image/Make.packages for full explanation of the mformat parameters for testing).

If so, we we will need to use a few more values from the BPB, which can be defaulted and ignored for MINIX and created and/or updated by mformat for MSDOS. The calculation should be:

1. Determine root directory start location by using by adding the # reserved sectors and the fat table size: ROOTDIR_SECTOR = cSecRes + (cSecFat * cFat).

2. Search the root directory for "linux " for up to DIRNUM entries (the corrected value should be 224 = 14 * 16 entries rather than 244 as stated previously) to find the starting cluster #.

3. Read the file by following the FAT chain as per FAT12 rules.

For simplicity, the current MSDOS image creation system in the package management PR creates FAT12 only, using a format specification with minimal sizes, since our maximum ELKS distribution only contains 224 files at this point anyways. Thus we should always be able to get away with always using FAT12 for disk images and remain fully compatible with DOS, and simplify our kernel filesystem support initially.

Finally, the mfs program is now being used to write all MINIX boot sectors. Depending on where the second sector is, it can be quickly modified to update MINIX boot images with the head and sector max data, as well as write and update our two MSDOS boot blocks with the same information.

[ghaerr]

@mfld-fr and @tkchia:

Reading the MSDOS 3.3 source showed another trick the Microsofties use to put their loader in a single 512 byte boot block: the loader assumes that IO.SYS and MSDOS.SYS are the first two entries in the root directory, rather than having code to scan the entire directory.

We could do this too, simplifying our boot loader, by assuring that \linux is the first directory entry. This can easily be done by copying a zero-length file named \linux right after the disk format, whose directory entry will be reused when copying /linux later. I tested zero-length copies when building the MSDOS PM (package manager) copying \var\run\utmp and it works fine.

[ghaerr]

As referenced in PR #297, it is now clear exactly how MSDOS boot sectors will be initially copied, then using mformat to update the BPB and create the FAT12 filesystem. No other tool changes are required, and that PR should work with our updated DOS boot sector, with the possible exception of requiring two reserved boot sectors rather than one.

For MINIX bootable images, the sequence remains as it is now, which is that the filesystem image will be created first by mfs, then the two boot blocks will be added by "mfs boot ...". A new option will be added to mfs that will optionally be used by the package manager to update the BPB max heads and sectors in the first sector, when it is desired to create multiple disk images without a .config file recompilation.

[tkchia]

Hello @ghaerr,

We could set the # reserved sectors to 2 on the mformat, which should allow it to be the second sector on disk... that would be tremendously simpler than having to find it elsewhere, but I haven't tested that yet. We could set the parameter "-R" to be 2, and see whether DOS will read the disk.

Well, I just tried it a moment ago with MS-DOS 6 (under QEMU), and unfortunately this trick does not work. I can create such a disk image with mkdosfs -R 2, copy a file into it with mcopy, and list the copied files with mdir. But MS-DOS 6 does not see the copied files --- and what is more:

A:\>chkdsk b: /f
Volume Serial Number is 299E-497F
Probable non-DOS disk
Continue (Y/N)?

[tkchia]

Hello @ghaerr,

One thing I am trying out is to add an ELKS-bootloader-specific parameter block, which will be separate from the BPB but (hopefully) still easily locatable.

This parameter block can then hold e.g. the location (e.g. logical sector number) of the stage 2, OS-specific bootloader. This will point to the second sector for Minix and other filesystems, but will most probably point elsewhere for FAT12/16 filesystems.

I mentioned (https://github.com/elks-org/elks/pull/293#issuecomment-581499326) stashing the parameter block information into the 8-byte "OEM name" field. More specifically, I am currently trying out a structure like this:

     <----- OEM name ----->
     _______________________________________________
    |          | loc.| len.|  bytes   | secs. per |
BPB | "eLkS\a" | (*) | (*) | per sec. |  cluster  | ...
    |__________|__|__|__|__|__________|___________|_
               __/      |
              |         v
              v <-------------------------------------->
              ________________
Extra parms. |  2nd stage   |
             | payload loc. | ... (for future expansion)
             |______________|_

This should allow the OS-independent bootloader code to work without any knowledge of the specific filesystem. However, the burden then shifts to the filesystem machinery, which will have to deal with properly reserving space for the OS-specific bootloader.

Thank you!

[mfld-fr]

I am afraid we have to let even the OEM name field untouched and set by the formatter, from what I read on Wikipedia. As I wrote earlier (https://github.com/elks-org/elks/pull/293#issuecomment-581459133), ELKS parameters should be entirely out of the BPB space for the safest design and maximum FAT compatibility.

[ghaerr]

Well, I just tried it a moment ago with MS-DOS 6 (under QEMU), and unfortunately this trick does not work. I can create such a disk image with mkdosfs -R 2, copy a file into it with mcopy, and list the copied files with mdir. But MS-DOS 6 does not see the copied files

@tkchia: Thanks for testing that, amazing how MSFT produces all sorts of documentation on their filesystem and come to find it doesn't work! I want to double-check with the another formatting tool mformat, to make sure that it isn't just a mkdosfs bug. (Which package is mkdosfs from?)

I have another idea for where the second sector could be easily placed - we could use the same trick that MSDOS uses now with IO.SYS required to be in the first directory entry. That is, it would be a regular (hidden) 512 byte file but accessed by reading the first FAT entry. As you mention in your earlier comment, the tool to find that location would then write that sector location in the ELKS parameter block. But if we're going to do that, why not just prepend the second boot sector to \linux and have the second boot sector and ELKS all in one file?

I will get deeper into these issues in the response to your other comment above.

[ghaerr]

I am afraid we have to let even the OEM name field untouched and set by the formatter, from what I read on Wikipedia. As I wrote earlier (#293 (comment)), ELKS parameters should be entirely out of the BPB space for the safest design and maximum FAT compatibility.

Hello @tkchia and @mfld-fr,

It seems to me that the idea of trying to design a boot sector 1 that is identical for all operating systems (and test cases, like the cool probe-test) might be heading towards where it doesn't make much sense, given the complexity required around it. Consider the following, in no particular priority order:

o Exactly why is it that we want to have an identical initial boot sector?

o For MINIX and probe-testing relating to ELKS, it makes sense. Adding "boot applets" for ELKS and/or MINIX debugging, system/hardware information and other stuff is made far easier.

o Add in MSDOS, and we need to reserve at least offsets 11-61, now wiki says OEM field so 4-61. But are we going to support FAT32 volume booting ever? Then we need to reserve 4-90. Why should we reserve the BPB in the MINIX boot sector, it will 1) never be stamped or looked at by MSDOS systems, formatting tools or utilities, 2) the MSDOS FAT12/16 filesystem apparently doesn't work when built with reserve sectors, so we have to put the block in the filesystem, which requires all the boot-stamping tools to now understand and read each filesystem type to calculate the starting 2nd block number (lots of extra work and code duplication for not really any gain, especially when MSDOS boot doesn't actually need a second boot sector).

o We're talking about adding the MSDOS BPB into our ELKS MINIX boot sector, but then not using it (@tkchia 's and @mfld-fr suggestion above is with having a separate ELKS parameter block). This means that we would have to use our own special boot-stamping tools to set the ELKS BPB in the MSDOS and MINIX boot sector, but not use the MSDOS BPB head/sector values just stamped by the MSDOS format routine, and we ignore or have them conflict? For the MSDOS boot, why not just use a different booter, it would include and read the MSDOS BPB stamped by well-tested MSDOS tools, use the BPB to locate the first file by locating the first directory entry and associated FAT block (IO.SYS in MSDOS, linux for ELKS), and read in the entire OS image. The MSDOS 3.3 boot accomplishes all this in 512 bytes, why should we want to have an ELKS boot sector with two BPBs, possibly conflict, which just reads a single second boot sector off the MSDOS filesystem, only to have additional code then finally locate \linux? Lots of work, where's the benefit?

o Didn't @mfld-fr just spend a lot of effort rewriting the boot code so that it didn't require a multi-stage boot, that now all works in two sectors that are always placed at the beginning of the disk image (the current minix.bin)? Why take that design and add an MSDOS BPB into it, just for the sake of having an identical boot block, which won't even use the BPB data in its own code? Seems to me that we should keep our own boot system for ELKS MINIX and design a new one for MSDOS (especially if we ever want to support FAT32 boot). Reserving 57 bytes of a 512 byte sector that could be used for better ELKS MINIX boot-applets is a waste of space, and we introduce additional problems trying to figure out where to put the second sector only in MSDOS, only because of our requirement of making the boot sector identical for everything.

o Separate boot modules are all compiled in elks/elkscmd/bootblocks, the benefit of the current design is great for new payload "applets" that don't need to concern themselves with startup. Designing a new MSDOS-specific single-block boot sector that just loads \linux and uses its own BPB is far easier, and we don't have to get other tools (like mfs) updated with having to know about each separate filesystem some payload might be used for.

My take at this point is: No real gain and lots of tools code duplication and work for trying to make MSDOS and MINIX sector 1 boot code identical. Leave the current well-designed two sector MINIX boot mechanism unchanged (as far as BPB and MSDOS etc). Continue to build on it for ELKS/MINIX specific enhancements (like probing or other useful ELKS debugging tools that could be run at boot time), and start from scratch with a new design for an MSDOS boot block. That boot would 1) fit entirely within one 512 byte sector, 2) use an MSDOS-compatible BPB area for head and sector count, 3) use the BPB information to locate the first directory entry, just as MSDOS 3.3 boot does, and 4) load \linux to memory directly from that information. This can be made easy with no tool mods and no filesystem information for stamping boot BPBs because our current tools can ensure that \linux is the first directory entry, and also that the OS image file is contiguous sectors, meaning that the FAT chain does NOT have to be followed (just use the initial cluster/sector number from the first directory entry, just like MSDOS does for IO.SYS/MSDOS.SYS loading).

Sorry for the long comment, and thank you!

[ghaerr]

One thing I am trying out is to add an ELKS-bootloader-specific parameter block, which will be separate from the BPB but (hopefully) still easily locatable.

Hello @tkchia,

I have two concerns, 1) having a separate ELKS BPB with head/sector information that would required to be duplicated in the MSDOS BPB leaves open the problem of conflicting information - that is, if the head or sector information happens to be different in the BPBs, then all heck breaks loose at some unknown point after successful boot; 2) Since only our tools would be writing an ELKS BPB, there's no need for a searchable string, remove it and just use a fixed offset location for the ELKS BPB.

This parameter block can then hold e.g. the location (e.g. logical sector number) of the stage 2, OS-specific bootloader. This will point to the second sector for Minix and other filesystems, but will most probably point elsewhere for FAT12/16 filesystems.

As you probably read in my last comments, I've come to the conclusion that our attempts to make a single identical boot block is over-designing, without much benefit. In this case, the MINIX boot never needs to know where the second boot sector is located, its always sector #2. And as you mentioned, having to calculate the location of a second sector, only required for MSDOS boot, introduces the fairly heavy expense of having to integrate FAT filesystem knowledge/code into the boot-stamping tool, when its not even needed (see my earlier comments). In addition, it opens up the problem of if a user rewrites the OS file \linux on an MSDOS floppy, he MUST also use the ELKS-only boot-stamper after writing it or the system won't boot.

     <----- OEM name ----->
     _______________________________________________
    |          | loc.| len.|  bytes   | secs. per |
BPB | "eLkS\a" | (*) | (*) | per sec. |  cluster  | ...
    |__________|__|__|__|__|__________|___________|_

Thanks for the descriptive ascii drawing, pretty cool!

A quick comment on the length field: that information is only useful if the file is known contiguous. If not, then there would be no need for a length field since the loader would have to have full knowledge of the filesystem and would then read each sector from FAT until EOF. On MSDOS systems, locating the directory entry, required anyways, gives the same information about the file, without having to be duplicated in the boot block.

This should allow the OS-independent bootloader code to work without any knowledge of the specific filesystem. However, the burden then shifts to the filesystem machinery, which will have to deal with properly reserving space for the OS-specific bootloader.

Yes, the tools then would have all to be updated with both MINIX and MSDOS (FAT12/16/32) filesystem knowledge/code just to fill out these few fields. Lots of work, and not really needed since the MINIX boot currently works without it, and MSDOS can boot using a far simpler mechansim.

Thank you!

[mfld-fr]

Just 3 things to (re)state :

1. FAT32 is out of scope (no sense for the order of magnitude of the ELKS filesystem size),
2. Not a problem to move the computation of the payload (first, length) out of the boot sector,
3. ELKS is not MSDOS / FreeDOS. Just has to support read / write / boot from an already existing FAT.

[mfld-fr]

So, to be more clear on the expected design, and what would be my criteria to accept any more PR on this topic :

1. Generic boot sector for all supported filesystems : no filesystem knowledge in that sector, just target knowledge (including disk geometry), payload knowledge, and reserved space for filesystem specific data. No ELKS data sourced from that filesystem specific data (so from the reserved space). Reserved space can only be used as a scratch area after the boot sector is loaded in memory.

2. FAT or MINIX filesystem is worked at build time with standard tools (so without any knowledge of ELKS). No change to them that is not related to the filesystem itself. ELKS specific tools to install into an existing filesystem, including patching the existing boot sector (so preserving the existing filesystem data), and inserting the payload into some reserved sectors if possible, or as a file.

3. Payload is either a secondary boot loader with the knowledge of the filesystem, or directly the 'linux' image. The current MINIX boot block is not a reference but an intermediate version that has many drawbacks and is to reworked.

4. No more boot sector in the 'linux' image, just the 'setup' and the 'system' programs, with 'setup' only preparing the CPU & memory context for 'system' (all the 'setup' HW detection and initialization moved to 'system', no more INITSEG, no more splitting of the 'linux' image).

Edit : surely more work, but a better design to host the planned next features (like the 'initial root').

[tkchia]

Hello @ghaerr,

My take at this point is: No real gain and lots of tools code duplication and work for trying to make MSDOS and MINIX sector 1 boot code identical. Leave the current well-designed two sector MINIX boot mechanism unchanged (as far as BPB and MSDOS etc). Continue to build on it for ELKS/MINIX specific enhancements (like probing or other useful ELKS debugging tools that could be run at boot time), and start from scratch with a new design for an MSDOS boot block.

Me, I think I can abide by this general approach --- though it seems to me that @mfld-fr is not very keen on it. @mfld-fr: I am assuming you have your reasons for this, but perhaps you can elaborate a bit more?

For the time being, I will see how I can improve the boot code, in ways that are more straightforward (and less controversial).

Thank you!

[ghaerr]

ELKS specific tools to install into an existing filesystem, including patching the existing boot sector (so preserving the existing filesystem data)

We need a known (preferably just fixed) location in the current ELKS MINIX boot sector as to where the head_max and sector_max geometry values will live, so that "mfs boot -B," can write those values after the build. We can worry about the ELKS MSDOS boot sector later when that all gets decided.

The current MINIX boot sector puts the sect_max and head_max values almost at the end of the sector, but not at a known location. My suggestion would be to .org them two words before 0x1FE, as well a moving track_max before both of them, since it is not used and may eventually go away.

After @tkchia or @mfld-fr submits a PR for the bootblock change, I will submit a PR adjusting mfs boot generator to use the chosen values, which will then complete the changes for the ELKS MINIX test image generation to work for the 360k, 720k and 1440k images.

[ghaerr]

I plan to leave ELKS MSDOS boot issues aside for now, and instead dive into the ELKS MSDOS filesystem implementation, from which I understand from issues #123 and #129, @mfld-fr's comments that it was rushed into service, and possibly other issues that the MSDOS filesystem barely works and is quite buggy.

I have reviewed all the MSDOS filesystem source code under elks/fs/msdos and find that it is quite hard to read, uses non-essential #ifdefs (FAT32 only support for instance), has lots of code constructs generating extra code segment size (constant checking of FAT12/16/32 superblock type when a local variable would do), and of course it doesn't hardly work (apparently fails after a single 'rm'). This code definitely needs some TLC.

In addition, inspection shows that a usable and small implementation of an in-memory /dev device list, which would support our goal of booting from FAT12 images is included, but not currently compiled in nor tested.

Does anyone mind if I take this over, clean it up, and make it work? I propose doing it with a number of smaller PRs, for ease of code review, inspection, and testing.

[mfld-fr]

@tkchia: Yep !

Suppose you replace the PC architecture by an ARM SoC, and keep the same filesystem (on a SD card)... you would rewrite the boot sector in regard to that new SoC only, not the filesystem. Other tools to work with that filesystem would be the same. No many boot sectors to maintain, just one per architecture. Code maintainability and portability is one of my first concern as a maintainer.

About the current MINIX boot block : it is not a good design yet, I now realize it was partially better before I reworked it. The boot block should not have any knowledge of the actual OS. It should only search for a 'boot' file, load it and run it. So that it works for any OS installed in the filesystem. Today we have the 'run_prog' function that is ELKS specific... in a generic boot sector 🤢 !

The current filesystem tools already have all the intelligence to extract easily the first sector and the number of sectors of any contiguous file. Not a big deal to pass that data just after to a stamping tool that would be a companion of the generic boot sectors, and the same for all the architectures.

[mfld-fr]

@ghaerr : Please go on, it is better to improve a draft than starting from a blank page. Thank you for your energy !

[tkchia]

Hello @mfld-fr,

Suppose you replace the PC architecture by an ARM SoC, and keep the same filesystem (on a SD card)... you would rewrite the boot sector in regard to that new SoC only, not the filesystem. Other tools to work with that filesystem would be the same.

Thank you. For goals that far off though, I am more inclined towards a "we will cross the bridge when we get to it" approach.

In particular, my impression is that ARM computers do not have the practice of reading a 512-byte bootloader from the beginning of a filesystem partition --- plus, ARM computers do not even have a semi-standardized BIOS interface like IBM PC compatibles do. I suspect that if we try to engineer something portable without informed knowledge of platforms' actual workings, our "portability" efforts will likely fail.

Hello @ghaerr,

Does anyone mind if I take this over, clean it up, and make it work?

I think I am cool with that.

Thank you!

[mfld-fr]

@tkchia : not so far... my second SBC where I would like to run ELKS is a Cubieboard v1, with an Allwinner A10. It contains a small BROM that tries to load the first sector from the SD card.

[tkchia]

Hello @mfld-fr,

my second SBC where I would like to run ELKS is a Cubieboard v1, with an Allwinner A10. It contains a small BROM that tries to load the first sector from the SD card.

Hmm, I hope to learn more about this Cubieboard thing. I understand that Debian on a Cubieboard uses the u-boot bootloader.

Hello @ghaerr,

I want to double-check with the another formatting tool mformat, to make sure that it isn't just a mkdosfs bug. (Which package is mkdosfs from?)

It is from the dosfstools package, at least on my Ubuntu box.

Thank you!

[mfld-fr]

@tkchia : http://sunxi.org/Cubieboard

Yeah, UBoot & Linux... and ELKS one day 😆 ?

[tkchia]

Hello @mfld-fr,

Yeah, UBoot & Linux... and ELKS one day ?

☺️ For FAT12 boot code, I think it will be a tad more feasible to reuse some of the stage 1 bootloader code, at the source level (while keeping it within the first 512-byte sector).

For example, we will most probably want to reuse disk_read, as well as the code for modifying the drive parameter table at 0:0x78, in any FAT12 bootloader.

I believe the syslinux project uses a combination of assembler macros and linker magic, to make such code reuse possible.

Thank you!

[mfld-fr]

As you want for the stage 2 boot loader option...

Without a stage 2 boot loader and a direct 'linux' load, one has to stamp the boot sector each time the 'linux' file is modified in FAT (and to keep it contiguous), but things are easy to implement, and boot is faster.

With that loader, a bit slower boot, but the ability to move the linux file everywhere on the FAT and have it fragmented, at the cost of a bit more complex implementation and installation.

I am considering the same design for MINIX to improve the current design of the 'boot block', and put back the generic 'MINIX boot loader' that existed before my rework (= the 'boot' file). That way we could move out the ELKS relocation & splitting from the 'boot sector', to make room for more detailed error reporting.

[ghaerr]

Without a stage 2 boot loader and a direct 'linux' load, one has to stamp the boot sector each time the 'linux' file is modified in FAT (and to keep it contiguous), but things are easy to implement, and boot is faster.

Booting faster is a very good thing on old hardware :)

Before reversing direction and writing a 2-stage FAT boot loader (possibly for good reasons @mfld-fr details below), I think it would be worth the effort to try to get FAT boot working with just one sector. This simplifies the otherwise need to place the second stage somewhere on the FAT filesystem (since reserved blocks apparently don't work with real MSDOS), which would very likely also still require stamping since the second stage file would need to be in the first root directory sector, which can't be guaranteed otherwise.

With [the 2-stage] loader, a bit slower boot, but the ability to move the linux file everywhere on the FAT and have it fragmented, at the cost of a bit more complex implementation and installation.

If our FAT boot can in fact be put in one sector (likely so since MSDOS does it), then using the logic that the 'linux' file can be fragmented so we need a 2-stage loader isn't that important since the linux file will virtually never be written outside of the ELKS image building process.

I am considering the same design for MINIX to improve the current design of the 'boot block', and put back the generic 'MINIX boot loader' that existed before my rework (= the 'boot' file). That way we could move out the ELKS relocation & splitting from the 'boot sector', to make room for more detailed error reporting.

These are good reasons that allow more space for better reporting and room for expansion. Given that we're still heavily changing the basic boot mechanism and floppy drive recognition process in the kernel, IMO moving both MINIX and FAT boots (back) to 2-stage loaders might be best left until the 0.3.2 release, rather than now.

[tkchia]

Hello @mfld-fr, hello @ghaerr,

I have made a first attempt at coding an ELKS bootloader for FAT12/16. It works, but I had to cut several corners to get it all to fit into 512 bytes:

• I only managed to fit in a valid FAT BPB up to offset 0x1f.
• /linux must be the first entry in the root directory, and it must be in contiguous sectors.
• I #ifdef'd out the check for the "ELKS" signature in the /linux file.

(By the way, what exactly is a 1.68 MiB floppy disk? Apparently mkdosfs has no idea about such a thing, and thinks it is some sort of hard disk partition type.)

Thank you!

[tkchia]

Hello @mfld-fr, hello @ghaerr,

Here is a binary of the bootloader for (1.44 MiB) FAT: fat.bin.zip. There are only about 3 bytes of space left in the boot sector.

Thank you!

[ghaerr]

I have made a first attempt at coding an ELKS bootloader for FAT12/16. It works, but I had to cut several corners to get it all to fit into 512 bytes:

Wow, This is great news, great job!!

• I only managed to fit in a valid FAT BPB up to offset 0x1f.

The mformat tool I'm using to build FAT floppies always writes between offsets 11 and 61 of the FAT BPB. In particular in your case now, it will write over the volume label and ASCII volume type, even though these items aren't used by any OS to determine drive type or the volume label. (BTW, the spec states that when the volume label (a directory entry with the ATTR_VOLUME attribute set) changes, the filesystem driver must also write the FAT BPB volume field. The ELKS FAT filesystem driver doesn't read or write volume labels in either locations at this point).

So, unless we figure out how to get more space or use a couple tricks, we can't use the mformat (or likely any other) standard FAT formatting tool, which would be unfortunate. Despite Wiki apparently saying that the OEM field from 0-10 may get written, one idea is to use that space, which would gain 8 bytes, by putting a procedure or possibly data segment stuff there. Unfortunately, some the first byte (jump instruction) at location 0 is sometimes used as a signature byte, from what I've read, so only 3-10 is available, which gets 8 more bytes.

Another possibility, which would work, would be to first create the FAT filesystem with mformat, as is done now. Then, use setboot to read the disk image from the mformat image to get BPB parameters (with the sector/head max override also specified with -B) and your boot sector, combine the "limited FAT BPB (offsets 11-35)", then write the boot sector. This method would have the following limitations and requirements:

• FAT BPB must include dword total sector count at offsets 32-35 (this is a must for FAT32)
• ELKS boot disk would fail if user inserted it into MSDOS system and changed volume label
• MSDOS might report strange-looking volume label (unlikely, as I think most later versions always use directory entry with ATTR_VOL as volume label
• There would be no ELKS BPB, so kernel should be improved to recognize and read FAT BPB values instead , to get the sect_max and head_max values (see below)

• /linux must be the first entry in the root directory, and it must be in contiguous sectors.

No problem there, already ensured.

• I #ifdef'd out the check for the "ELKS" signature in the /linux file.

Keep the check in for MINIX BPB, which will fail, and then add FAT BPB boot block checking, which would succeed for you to get the sector and head values. Some ideas for FAT BPB signature:

• "boot signature" field at offset 38
• 2-4 bytes of ASCII filesystem type at offsets 54-55 (would be 'FA' or 'FAT1')
• perhaps jump instruction at byte 0 in lieu of one of the above

This would require very tricky .org/.byte areas in the boot sector at offset 38 and 54, until it was figured out how to save more space.

[ghaerr]

Here is a binary of the bootloader for (1.44 MiB) FAT: fat.bin.zip. There are only about 3 bytes of space left in the boot sector.

@tkchia: have you tested this boot on the package manager fd1440-fat.bin by rewriting its first sector ("setboot fd1440-fat.bin fat.bin")

I have just reviewed your work on the MSDOS-compatible FAT BPB. Very nicely done. I can see that you are very tight on space, and it may not be easily feasible to use the bs_oem_name field as previously suggested. Another thought is to reduce the size of msg_boot and msg_reboot for now.

[ghaerr]

Hello @tkchia:

In looking at your boot code and MSDOS 3.3 boot.asm a bit further, we may be able to get this to work without the complications mentioned in my first response. The MSDOS 3.3 boot sector does not use the volume label, ASCII filesystem type, nor extra boot signature. Thus, their own 3.3 boot sector would also fail should someone update the volume label on it with a MSDOS 6.0 system. Since nobody will likely be doing that on ELKS (or MSDOS 3.3) boot disks anyways, the problem of volume ID change is one to quit worrying about.

Instead, your fantastic first pass can be made to work with the following small changes:

• add one more .long after bpb_hidd_sec for total sector count, set to 0 which works for FAT12 and FAT16
• Use the OEMNAME field for kernel FAT BPB signature checking 'MSWI' for our boot disks. The OEM field could be changed to 'ELKSFAT1' to guarantee our kernel doesn't pick up bad info and just use 'EL' from this similarly to your 'eL' on ELKS BPB boot images.
• Shorten msg_boot 'Boot sector' by 1-4 bytes to get required space

That should do it, then we can use a modified setboot to sew the build together!

[ghaerr]

@tkchia and @mfld-fr:

The idea of having the kernel read the boot sector and attempt confirmation of either an ELKS MINIX boot disk, or an ELKS FAT boot disk by signature verification, would go a long way for efficient recognition of disk geometry for disks created for use with the ELKS system. ELKS-bootable FAT disks would be guaranteed to work well using this design. I support adding ELKS FAT BPB signature recognition for sectors/heads information being added to the kernel since it's likely to see a lot more FAT use in this next release.

However, for in-the-wild FAT disks, signatures will be missing, so we would be back to probing or the INT13 method, which design is still under discussion. So everyone knows, the current FAT filesystem will mount an image (without kernel signature verification) if it meets the following criteria:

• Non-zero FAT table count
• Non-zero cluster size
• number of directory entries evenly divisible by 16 (=entries/sector)

The FAT filesystem could also possibly mount an invalid disk, although unlikely given the number of dir entries check.

[tkchia]

Hello @ghaerr,

Instead, your fantastic first pass can be made to work with the following small changes:

Thanks for your suggestions! In addition, I managed to squeeze out several more bytes of space, by removing the run_prog() routine from the FAT code. (This should be OK, as it seems run_prog() is slated to be replaced anyway.)

Basically run_prog() assumes that /linux is loaded as a single binary blob at LOADSEG:0, splits it into its "setup" component (which does some initialization) and its actual kernel component, copies the setup code to ELKS_INITSEG:0 and the kernel code to ELKS_SYSSEG:0, then runs the setup code.

What my FAT code now does is to directly read sectors to ELKS_INITSEG:0 and ELKS_SYSSEG:0. This turned out to help me save quite a fair bit of space, :-) though perhaps still not enough to fit in the whole "extended BPB" structure (up to offset 61).

@tkchia: have you tested this boot on the package manager fd1440-fat.bin by rewriting its first sector ("setboot fd1440-fat.bin fat.bin")

I will do this soon. Thank you!

[ghaerr]

Great job! Its amazing how you're able to continue squeezing more space out of the sector! :)

This turned out to help me save quite a fair bit of space, :-) though perhaps still not enough to fit in the whole "extended BPB" structure (up to offset 61).

At this point, I'm not sure we will need an extended FAT BPB. Tests can be made to determine what this disk looks like on various MSDOS versions and then we'll know for sure.

We'll see what @mfld-fr thinks of it, separately from the kernel BPB signature verification discussion.

With these mods, it looks like we're ready to go with a prelim bootable ELKS FAT disk from a .config file, which will allow us to address the other remaining problems, such as 8.3 filename limitations in the root filesystem. We need some time to check how well ELKS runs on a working FAT filesystem.

After the PR is accepted, I can think about the setboot changes to allow for multi-disk image preparation.

@tkchia: have you tested this boot on the package manager fd1440-fat.bin by rewriting its first sector ("setboot fd1440-fat.bin fat.bin")

I will do this soon. Thank you!

I'm very interested in what you find. I'm preparing the PR for the FAT filesystem bug fixes shortly.

[mfld-fr]

How to load a contiguous boot or linux file from FAT without any FAT specialized code in the boot sector ?

1- At build time, get some data from the FAT image with the mtools:

mfld@home ~/advantech/elks-master/image $ minfo -i fd360-fat.bin
device information:
===================
filename="fd360-fat.bin"
sectors per track: 9
heads: 2
cylinders: 40

mformat command line: mformat -t 40 -i fd360-fat.bin  -h 2 -s 9 ::

bootsector information
======================
banner:""
sector size: 512 bytes
cluster size: 1 sectors
reserved (boot) sectors: 1
fats: 2
max available root directory slots: 224
small size: 720 sectors
media descriptor byte: 0xf0
sectors per fat: 9
sectors per track: 9
heads: 2
hidden sectors: 0
big size: 0 sectors
physical drive id: 0x0
reserved=0x0
dos4=0x29
serial number: 00000000
disk label="NO NAME    "
disk type="FAT12   "
mfld@home ~/advantech/elks-master/image $ mshowfat -i fd360-fat.bin ::linux
::/linux <550-649>

2- Then compute the sectors of the linux file:

first = boot sectors + numbers of fats * size of fat + root dir entries * 32 / 512 + 550 - 2
first = 1 + 2 * 9 + 224 * 32 / 512 + 550 - 2 = 581
count = 649 - 550 = 99

3- Patch the generic boot sector to set (first, count) of the payload.

You are (almost) done ! Almost because still this uggly splitting of linux before jumping, but planned to be removed.

[ghaerr]

1- At build time, get some data from the FAT image with the mtools

Shouldn't the boot code just use the CONFIG_ head, track and sector information for building, since that's known and already set? In addition, the FAT images aren't built until after the boot sector is assembled.

mformat command line: mformat -t 40 -i fd360-fat.bin -h 2 -s 9 ::

The mformat command line can be updated to set the head and sector information separately from the fat.bin build, especially since it's only run in the package manager, correct? Track information isn't kept in the FAT BPB so that would default to the CONFIG_ tracks setting used above.

mfld@home ~/advantech/elks-master/image $ mshowfat -i fd360-fat.bin ::linux ::/linux <550-649>

2- Then compute the sectors of the `linux` file:

first = boot sectors + numbers of fats size of fat + root dir entries 32 / 512 + 550 - 2 first = 1 + 2 9 + 224 32 / 512 + 550 - 2 = 581 count = 649 - 550 = 99

This calculation could be made at package manager image build time and then passed to setboot for patching (see below).

3- Patch the generic boot sector to set (first, count) of the payload.

setboot will also be required to copy only the FAT BPB portions of the formatted disk to the final image while also the other portions of fat.bin. Going this route allows us to use standard MSDOS/FAT tools to create the BPB, instead of adding that knowledge into our own tools, with the exception of the sector/head values in the FAT BPB, and the first/count values in the generic boot block. Correct?

[mfld-fr]

Almost correct:

• In generic boot sector code, disk geometry is neither extracted from the FAT image, nor from the FAT BPB, but coming from configuration, or later stamping, through the ELKS boot parameters. Only payload location is extracted at the time of the FAT bootable image build.

• No change to the standard mtools, to avoid maintaining our own version.

• setboot has nothing to do with the actual filesystem, except to keep the reserved space 'as is' when installing the generic boot sector (in order to keep the FAT BPB set by mformat in a previous step).

[mfld-fr]

Here is my proposal for the 'generic boot sector' : 2d133c907c15df74752a6f5615eb46621968b71a

[ghaerr]

@mfld-fr:

I think I understand more precisely now where you're going in the business of defining the various parts of the boot process and image making:

• ALL ELKS boot code will use the generic payload template as a base, whether it be for MINIX, FAT, PROBE or other uses.

• The generic template is not filesystem or payload specific.

• The generic template always includes an ELKS parameter block (EPB), which will include sector, head and track information required by the kernel later for efficient disk functioning without probing, and payload start and size information for getting the payload copied into RAM.

• The ELKS PB includes a signature for kernel identification of valid EPB data without regards to the boot filesystem.

• The ELKS generic template will include a reserved area coinciding with the MSDOS 3.3 compatible FAT BPB reserved requirements.

• The ELKS boot sector will be compiled/assembled according to the specs in the .config build configuration.

• No filesystem manipulation tools will be modified to know about EPB contents.

• The package manager will use standard filesystem manipulation tools to create each filesystem, copy the 'linux' kernel onto it, copy the package-selected applications onto it, then use setboot to copy the appropriate boot block onto it with EPB modifications.

• The setboot tool will not know about any particular filesystem, but will know about the EPB and set the passed EPB parameters required for the disk image.

• setboot will skip the FAT BPB area when copying the ELKS boot sector to the image.

Given the above, if setboot is also passed the 'linux' file copied previously as a parameter, then it can directly know the payload size without developers having to compute it based on filesystem. In addition, if the 'linux' file is copied first after filesystem creation, it's start location can be known from the parameters given for creation, which also removes the need to write complex scripts to somehow extract its location.

[ghaerr]

Here is my proposal for the 'generic boot sector' : 2d133c9

Given how extremely tight the boot sector could be for FAT, perhaps the generic boot sector should use bytes, not words, for the payload start and length (in sectors). Given that the payloads can easily be placed at the beginning of the filesystem, and will certainly be less than 128k in length, bytes would work and we save 2 bytes. These could always be increased should we really need them.

[ghaerr]

@mfld-fr and others: There are a couple other issues that need to be solved for successfully usage of ELKS on a (booted) FAT filesystem:

• At present, only 8.3 filenames supported. 4 filenames need changing. In particular, opening /etc/rc.d/rc.sysinit will fail, so no scripts can run at boot, and no /etc/motd. Also, /etc/resolve.conf won't open, so TCP will be affected. The only two other bad filenames are /root/.profile, and /var/www/index.html.

The FAT filesystem does support long filenames, but it seems to be only working for reading directories, not opening files. You will notice this when running ls on /etc/resolve.conf after boot. In any case, I don't want to get into upgrading the FAT filesystem for this before 0.3.1. I don't like the option of rebuilding the applications with changed strings for a FAT boot build, which leaves us with likely having to change the names in the applications that use them for MINIX and FAT (and make it easy to change if 14+ character filenames are later supported in FAT).

What do you suggest we do about the filenames?

• Small issue: /bin/ash and /bin/sh are copes, not symlinks. We should do away with this ash problem and rename or modify ash so that it installs only as /bin/sh and thinks its /bin/sh, and get rid of /bin/ash. This will greatly help space on smaller filesystems. We have the same issue with /bin/ftpget and /bin/urlget. Do we still have good reason to symlink them on our boot disk (which requires a copy on FAT and wastes space?) Lets delete one of them, the package manager doesn't support filename selection based on operating system yet, and its not worth the change for two files.

Thoughts on /bin/ash and /bin/urlget?

• FAT /dev directory. Have you looked at it? Does it contain the devices we most likely want? There are very much fewer entries than on our MINIX boot disk. EDIT: I don't think the required entries are included for TCP operation.

With these decisions, and @tkchia's FAT boot, ELKS is imminently usable on FAT. We can work on setboot and the package manager after getting a default 1440k ELKS FAT image booting and operating properly.

[tkchia]

Hello @mfld-fr,

If a grand unified bootloader --- even for FAT12/16 filesystems --- is what you prefer, I will see how I can modify my FAT boot code to fit into the two-stage scheme.

I guess a quick and easy thing for me to do would be to just separate out my boot_sect_fat.h code into a separate stage 2 bootloader, and stuff this stage 2 into a separate file, perhaps called /boot, at build time. /boot can then arrange to load /linux. I will see if I can get this implemented.

In the long term we can try to make /boot properly traverse the root directory and file allocation table.

(And yes, the whole division between the "setup" and "kernel" portions of /linux is kind of ugly.)

Given that the payloads can easily be placed at the beginning of the filesystem, and will certainly be less than 128k in length, bytes would work and we save 2 bytes. These could always be increased should we really need them.

Agreed.

Thank you!

[ghaerr]

If a grand unified bootloader --- even for FAT12/16 filesystems --- is what you prefer, I will see how I can modify my FAT boot code to fit into the two-stage scheme.

Wait, why?

I thought @mfld-fr was talking about keeping the source code to the generic boot sector from a common base structure, which is a good idea, but that doesn't mean having it do nothing else other than loading a payload, which loads another payload, does it?? That seems to be a big waste of what can otherwise be done in 512 bytes.

We currently have shown that we have the ability to boot from MINIX and FAT using a one-stage loading scheme (one sector for FAT, and two for MINIX), both of which load 'linux'. Why change that design now, on the eve of release 0.3.1, when it introduces needless complexity, just for being coded as a "unified bootoader"? If I understand correctly, this doesn't make any sense to me.

Why not use the current single stage boot mechanism we have working now for MINIX and FAT which require 'linux' or other payload to be in the first directory entry and contiguous, and get 0.3.1 out, THEN add multi-stage boot loaders for both and write the filesystem-dependent other portions that have the (weak) benefit of allowing the payload to be anywhere and non-contiguous?

In particular, introducing a two-stage loader for FAT just adds another file, whose code will duplicate what we already have in the boot sector, unless we add the somewhat-not-needed feature of allowing the payload to be anywhere.

There are a number of other things that need to get done to get 0.3.1 out, if we're trying to get out in the near future.

I guess a quick and easy thing for me to do would be to just separate out my boot_sect_fat.h code into a separate stage 2 bootloader, and stuff this stage 2 into a separate file, perhaps called /boot, at build time. /boot can then arrange to load /linux. I will see if I can get this implemented.

That shouldn't be hard, since you're already doing it in one sector. Unless you're going to write all the FAT logic for loading /linux from anywhere, in which case it will be a lot of work.

In the long term we can try to make /boot properly traverse the root directory and file allocation table.

That is the time when we should be thinking of moving your working single sector boot into a complex /boot, IMO.

[tkchia]

Hello @ghaerr,

OK, yes, I am also starting to be a bit confused. I think @mfld-fr is thinking of using the mechanism for loading the OS-specific payload to also load the /linux file itself.

Except --- from what I see --- this will not quite work yet, because the /linux file needs to be split into its "setup" and "kernel" components after loading (while the payload does not).

Almost because still this uggly splitting of linux before jumping, but planned to be removed.

(Edited to add: also, the payload and the kernel will likely need to be loaded at different addresses anyway. Unfortunately, code reuse has its limits...)

Maybe it is best to just try changing one thing at a time, to reduce confusion.

I think I will submit my working 1-sector FAT bootloader as a pull request first, and see what happens.

Thank you!

[mfld-fr]

Okay, let us proceed step by step, because there is still a lot of confusion after several posts, and we have to balance between short-term needs (= booting from FAT) and versatile design.

Just merged the FAT boot sector from @tkchia as a first step toward the 'unified boot loader'. I like that wording 👍 !

[xrayer]

Oh I see I'm late... but why did you used source of msdos 3.3 bootsector instead of freedos native bootsector? I think it have all the necessary:

• DOS compatible BPB
• relocate bootsector code itself on higher address to allow load bigger payload (>64kB)
• it doesn't have restriction that kernel file must be in 1st dir entry and probably can be fragmented too. It may happen when you use defrag and copy/move/delete files on FAT it may happen the kernel file entry move somewhere else from 1st entry. Except unless you will set SHR attributes (system-hidder-readonly) then disk tools wil keep the file and it's entry untouched.

I was trying to accomodate FD bootsector to load setup part and kernel part to it's memory locations and jump to setup begin but it failed for some reason and I hadn't time to debug why. OK I can see there was a progress since that so new version 0.3.1 should boot from FAT, I'll try...

[ghaerr]

Here's a screenshot of ELKS booting from a 1440k FAT floppy, then auto-mounting a 360k FAT floppy on the second drive /dev/fd1 :)

[mfld-fr]

why did you used source of msdos 3.3 bootsector instead of freedos native bootsector?

We actually started with our own boot sector and move toward a second stage boot loader, to have a generic boot sector only dependent on the platform (= BIOS), and a boot loader only dependent on the filesystems, for easier maintenance and portability.

must be in 1st dir entry and probably can be fragmented too. It may happen when you use defrag and copy/move/delete files on FAT it may happen the kernel file entry move somewhere else from 1st entry. Except unless you will set SHR attributes (system-hidden-readonly) then disk tools will keep the file and it's entry untouched.

Thanks for the hint ! Captured in #327.

[ghaerr]

Except unless you will set SHR attributes (system-hidden-readonly) then disk tools will keep the file and it's entry untouched.

Thanks for the hint ! Captured in #327.

I had already noted @xrayer's comment about adding SHR attributes to protect our contiguous /linux file on FAT images, and proceeded to add those attributes using the mattrib tool after /linux creation. That work is now included in 'Make.package', but is commented out because mattrib is buggy and doesn't work! I have yet to get it to change any file attribute on any file.

[tkchia]

Hello @ghaerr,

Strange... your copy of mtools seems to be having a lot of issues! I am not experiencing any issues on my end, though:

$ mattrib -i fd1440-fat.bin +h +r +s ::linux
$ mdir -i fd1440-fat.bin
 Volume in drive : has no label
 Volume Serial Number is 0000-0000
Directory for ::/

dev          <DIR>     2020-02-13  23:03 
bin          <DIR>     2020-02-13  23:03 
etc          <DIR>     2020-02-13  23:03 
home         <DIR>     2020-02-13  23:03 
lib          <DIR>     2020-02-13  23:03 
        5 files                   0 bytes
                            506 880 bytes free

$ mattrib
Mtools version 4.0.18, dated January 9th, 2013
Usage: mattrib [-p] [-a|+a] [-h|+h] [-r|+r] [-s|+s] msdosfile [msdosfiles...]

Hopefully the mtools you are using is not too dated or otherwise broken. Thank you!

[ghaerr]

$ mattrib -i fd1440-fat.bin +h +r +s ::linux

@tkchia, thank you for your quick testing!

I tried this just again, and it is now working. I am guessing that, in my previous testing, that mattrib failed due to our FAT BPB being incompatible after stamping with the previous boot block. it works now (on my branch with the enhanced FAT BPB), proving that we are in fact standard FAT tool-compatible.

I could uncomment the line in Make.packages and push that up to the PR now being reviewed, which would protect our /linux boot file against MSDOS, although the file will then not be visible. Perhaps it's best left commented out for the time being, so that users can see the linux image in the case of bugs.

[xrayer]

Adding SHR attributes to file is better than nothing but it would be preffered if bootsector code can find the kernel on any dir entry (if there would be enough space for such code). Most of newer DOSes can do it (MSDOS 7.x, DRDOS, FreeDOS...). But it's not some high-priority issue so for now use SHR that should prevent most file utils to manipulate such file.