search

ghaerr / elks

Embeddable Linux Kernel Subset - Linux for 8086
Other
960 stars 103 forks source link

FAT flash disk not detected #320

Closed mfld-fr closed 4 years ago

mfld-fr commented 4 years ago

(Moved from #306)

@georgp24 wrote:

I wrote fd1440.bin to a real floppy and booted my PC from an USB floppy disk drive with that. I can log in and use ELKS. However, both hard disks installed are NTFS so I cannot mount a FAT disk. I put in a FAT flash disk but that was not detected.

image1

image2

tkchia commented 4 years ago

Hello @georgp24,

Am I right that the FAT flash disk that is acting up is in drive B: (i.e. /dev/fd1, 03c0)? And that there is no problem with reading it under, say, MS-DOS or FreeDOS?

Thank you!

[edited to reply to the correct user]

mfld-fr commented 4 years ago

@tkchia : the problem was reported by @georgp24, not me ?

tkchia commented 4 years ago

@mfld-fr: oops, mea culpa.

georgp24 commented 4 years ago

I have tested it with DOS 7.0 now (Win98). The flash disk is not detected. Also tested a different one. The BIOS lists the flash disk in the boot menu as a hard disk, however.

georgp24 commented 4 years ago

I tried it on an old PC. DOS does detect the flash disk as B:, however, ELKS does not mount it.

tkchia commented 4 years ago

Hello @georgp24,

I wrote a small MS-DOS program (source; binary) to try to probe the capabilities of floppy and hard drives at the BIOS level, and also read from the drives.

May I know what happens when you run the program under DOS on your (newer) PC? And, what happens on the old PC?

Thank you!

Mellvik commented 4 years ago

@tkchia,

Thanks for the test-program - I just ran it to confirm what has been pointed out separately, the BIOS floppy parameter probe does indeed report the drive's physical capacity, not that of the medium.

It also confirmed the floppy probing problem reported in #123, that COMPAQ BIOS reports a second drive as present even if it isn't, and reports CHS-values of 0/0/0.

Can I ask what your cross-devel environ is for generating MSDOS executables?

—Mellvik

georgp24 commented 4 years ago

I would suggest to write the test program with DOS calls for printing to the screen. This way you can redirect the output to a file.

Here is a shot on my old PC without the flash disk inserted: oldpc-noflash

Now here is the same PC with flash disk inserted. Apparently it shows up as device 0x01 oldpc-withflash

Here is a screenshot on my new PC without flash disk inserted: newpc-noflash

Here is a screenshot of the PC with flash disk inserted: newpc-withflash

I used two different flash disks.

georgp24 commented 4 years ago

@tkchia A question on a different topic : if I set ELKS into unreal mode, could I address memory above 1 MB with GCC-IA16? E.g. with embedded ASM commands?

mfld-fr commented 4 years ago

@georgp24 : Please avoid to mix the topics and keep this thread focused on your problem. The mailing list is better suited to discuss transversal topics. Thanks !

Edit : Oup's, confused @georgp24 and @ghaerr, sorry 😕 !

tkchia commented 4 years ago

Hello @georgp24,

Regarding your question on GCC-IA16, to avoid cluttering up this thread, I think I would move it to its own thread (https://github.com/tkchia/gcc-ia16/issues/49). Thank you!

tkchia commented 4 years ago

Hello @georgp24,

I would suggest to write the test program with DOS calls for printing to the screen. This way you can redirect the output to a file.

Oops, my apologies. The program was writing the output to the standard error file (which is harder to redirect) rather than standard output. I will get around to updating it.

Also, thanks for the screenshots! It looks to me like there is some work to be done on ELKS, to properly support reading from your flash disk --- more specifically:

Thank you!

tkchia commented 4 years ago

Hello @Mellvik,

I am using GCC-IA16, together with my own libi86 library. Thank you!

georgp24 commented 4 years ago

@mfld-fr I am a bit confused now. One could discuss "ELKS into unreal mode" as a milestone which makes sense but I think one should make a new milestone then and not include the problems with FAT flash disks then. Also one could discuss improving the detection of flash disks as a milestone or in this thread. @tkchia opened a thread in gcc-ia16 for unreal mode adressing. I have no problem with that but the reason I asked was using it with ELKS so one could discuss this in an ELKS milestone thread. I admit that I was guilty but how is the best way to proceed now?

mfld-fr commented 4 years ago

Again, to discuss topics that are at the "white paper" stage, the mailing list with the threading tree is the best tool to handle multi-part conversations. Issues with linear threading are more suited for defect & change tracking.

tkchia commented 4 years ago

Hello @georgp24, I think unreal mode support is more toolchain-specific (i.e. GCC-specific) issue, and much less an OS-specific issue, which is why I thought to move the discussion there. I think any OS that can run 16-bit real mode code on a 386+ machine can easily be made to enter unreal mode. :-) Thank you!

tkchia commented 4 years ago

Hello @georgp24, hello @Mellvik,

I have included an updated version of my MS-DOS drive probing program under my libi86 project (https://gitlab.com/tkchia/libi86). I hope to release the binary soon.

Hello all,

For the second case (flash disk is drive 0x82), what will be a good way to accommodate a "third hard disk" as an ELKS device file? I think ELKS might be running a bit short of device minor numbers for this --- the <major, minor> pairs <3, 0>, <3, 0x40>, <3, 0x80>, and <3, 0xc0> have been taken up.

Thank you!

mfld-fr commented 4 years ago

Yeah, minor number should be reworked as BIOS number for raw disks. For partitions, it should be splitted in two nibbles, the most significant for the disk number, the least for the partition number.

ghaerr commented 4 years ago

Yeah, minor number should be reworked as BIOS number for raw disks. For partitions, it should be splitted in two nibbles, the most significant for the disk number, the least for the partition number.

That's a good idea, all we need are names for each of these now as well.

As a reminder, for any commits changing device files names or numbers, please also test on FAT and include changes in the FAT /dev elks/fs/msdos/inode.c and build image/Make.devices.

Mellvik commented 4 years ago

I may be missing something here, but let me contribute my point anyway: This is the traditional unix (and linux) way (I have noe idea what it looks like inside ELKS right now):

Would make sense to keep it that way.

--Mellvik

  1. feb. 2020 kl. 18:12 skrev Gregory Haerr notifications@github.com:  Yeah, minor number should be reworked as BIOS number for raw disks. For partitions, it should be splitted in two nibbles, the most significant for the disk number, the least for the partition number.

That's a good idea, all we need are names for each of these now as well.

As a reminder, for any commits changing device files names or numbers, please also test on FAT and include changes in the FAT /dev elks/fs/msdos/inode.c and build image/Make.devices.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ghaerr commented 4 years ago

@Mellvik: Yes. I think what @mfld-fr is suggesting is that:

mfld-fr commented 4 years ago

@Mellvik : don't agree. Major number is the key to the driver, so the device type / class. Then the driver uses the minor as a key to the device. It means that we should have one major for the raw HD, and another major for the partitionned HD. Reference : 'Linux Device Drivers', third edition, page 44.

ssieb commented 4 years ago

What do you consider to be the raw device? On linux, it's the 0th minor number:

# ls -l /dev/sd[ab]*
brw-rw----. 1 root disk 8,  0 Jan 31 00:57 /dev/sda
brw-rw----. 1 root disk 8,  1 Jan 31 00:57 /dev/sda1
brw-rw----. 1 root disk 8,  2 Jan 31 00:57 /dev/sda2
brw-rw----. 1 root disk 8,  3 Jan 31 00:57 /dev/sda3
brw-rw----. 1 root disk 8,  4 Jan 31 00:57 /dev/sda4
brw-rw----. 1 root disk 8, 16 Feb 15 15:51 /dev/sdb
brw-rw----. 1 root disk 8, 17 Feb 15 15:51 /dev/sdb1

I don't know what you're referencing on that page, but that's also documenting a very old kernel version.

ghaerr commented 4 years ago

@ssieb: Thank you.

So Linux's current scheme allows one to look at a disk via partition number

        minor = (disk_number << 4) | (partition_number + 1)

or as a raw device with all partitions

        minor = (disk_number << 4)

I think @mfld-fr was also talking about the ability to specify a disk using the BIOS rather than HD/floppy driver (using a different major number), where minor = BIOS AH disk value for accessing devices like @georgp24's BIOS flash drive 0x82. [EDIT: No, my misunderstanding, this is not needed; just changing ELKS to shift disk_number by 4 instead of 6 as it is currently solves this problem.]

Mellvik commented 4 years ago

@mfld-fr: [on major/minor] Yes, you're right, serious memory fault on my part. I stand corrected. :-)

mfld-fr commented 4 years ago

@Mellvik : No problem, was just to avoid any confusion :wink:

ghaerr commented 4 years ago

Hello @georgp24 and @tkchia,

Has this issue been resolved with the HD changes that were made in the last couple weeks?

Or does ELKS need some re-ordering of block devices to fix this problem? There are so many topics discussed in this issue that I don't quite understand what was originally needed.

tkchia commented 4 years ago

Hello @ghaerr,

ELKS still needs to support a third hard disk /dev/bdc (this also requires some changes in setup.S).

ELKS also needs to support "floppy disks" with weird geometries that are not handled by the trial-and-error probing procedure.

As far as I know these are yet to be implemented.

Thank you!

georgp24 commented 4 years ago

In my case it could skip the disks with an unrecognized file system and start with /dev/bda at the first disk it finds with a valid file system.

ghaerr commented 4 years ago

Hello @tkchia,

ELKS still needs to support a third hard disk /dev/bdc

Would you support the idea of changing the minor device numbers in the BIOS disk driver (doshd.c) per the following:

This should keep changes in the rest of the system to a minimum while allowing for more BIOS devices.

How would, in @georgp24's case, BIOS device 0x82 be mapped to /dev/bdc? I don't quite understand where/when the BIOS -> HD mapping takes place.

(this also requires some changes in setup.S)

This is for /dev/fd1 being at a different minor for setting ROOT_DEV, right?

tkchia commented 4 years ago

Hello @ghaerr,

Would you support the idea of changing the minor device numbers in the BIOS disk driver (doshd.c) per the following:

  • Use 16 rather than 64 as the spacing between HD disks (and FD disks)
  • Keep first HD (/dev/bda) at 0x00 and first FD (/dev/fd0) at 0x80

Yes. :-)

I don't quite understand where/when the BIOS -> HD mapping takes place.

It seems the mapping between device numbers and BIOS drive numbers happens in a few places in elks/arch/i86/drivers/block/doshd.c, elks/arch/i86/drivers/block/directhd.c, (my modifications of) elks/arch/i86/drivers/block/init.c, and (?) perhaps elsewhere. Also it seems there are some parts --- e.g. the drive_info[4]; definition in doshd.c --- that assume that there will only ever be 4 disk devices. I guess all these will have to be fixed, and tested.

(this also requires some changes in setup.S)

This is for /dev/fd1 being at a different minor for setting ROOT_DEV, right?

Currently elks/arch/i86/boot/setup.S only reads the disk geometry information for the first two fixed disks --- so this will have to be changed.

Thank you!

ghaerr commented 4 years ago

Hello @tkchia, thanks for your explanation.

It seems the mapping between device numbers and BIOS drive numbers happens in a few places...

I was thinking this might be complicated. I'll take a look in those areas, not sure I'm ready to jump into this enhancement just yet! Also don't have a way to test MBR partition recognition without importing various MBR disk images.

ghaerr commented 4 years ago

Hello @tkchia and @mfld-fr,

I have studied doshd.c, directhd.c and genhd.c and understand the general structure much better now. Adding the ability to have more than two BIOS HD devices and restructuring the minor device numbers as discussed doesn't appear too hard, but testing everything poses a challenge. I know there have been discussions for changing the way hd/fd I/O will be done, so I have some questions before jumping in:

If I can get an MBR boot image, and mount it later while booting ELKS from floppy, it would be possible to enhance doshd.c to allow 4 BIOS 'hard drives' and two floppies while being able to test it. After working, directhd.c could be changed, although not sure whether that would work for flash disk.

Is this worth doing?

Mellvik commented 4 years ago

@ghaerr, FWIW - I believe the 100M image can be fixed so that it boots in QEMU. I'll take a stab at that later this week, when the sash cmd history is fully operational (along with disk i/o benchmarking).

Further, the math for CHS disks seems off. The limit for PC/AT INT 13 is 528M. If the HW (and BIOS) is just a little newer, it's 8GB.

--Mellvik

BTW - I'm keeping a writeup on CHS & BIOS I/O that I found on the net a while back easly at hand in order to support my own memory ( http://www.uruk.org/orig-grub/PC_partitioning.txt) . Here's a clip:

Definitions

  1. mar. 2020 kl. 05:08 skrev Gregory Haerr notifications@github.com:

 Hello @tkchia and @mfld-fr,

I have studied doshd.c, directhd.c and genhd.c and understand the general structure much better now. Adding the ability to have more than two BIOS HD devices and restructuring the minor device numbers as discussed doesn't appear too hard, but testing everything poses a challenge. I know there have been discussions for changing the way hd/fd I/O will be done, so I have some questions before jumping in:

Will the BIOS HD driver be the preferred method for HD I/O (vs HW IDE through directhd.c)? A few concerns here: BIOS int 13h only allows CHS = 102463255 = ~16M sectors = 8MB max HD size. That portion will have to be rewritten to use LBA or another method for any normal-sized partitioned disk, right? How does this work now with our hd32-minix.bin 32MB image, let alone FAT MBR images? I think it doesn't, but haven't tested filing the image. I think I understand the reason DMA_OVR was brought back: we don't want to align the L1 buffers to 1k boundaries right now (due to future local heap mgr), nor do we have mechanism to do I/O into L2 buffers yet... but why is DMASEGSZ only 1k, thus slowing all I/O down to 2 sectors, with the last 1k always 1 sector (due to 720k floppy fix #39, even for HD I/O)? Is DMASEGSZ 1k just because the L1 buffer size is 1k, with the BIOS driver called from the buffer routines? Changing minor numbers require both BIOS and direct driver to be changed, since same major number. Does QEMU emulator properly emulator IDE controller? When a floppy and a hard drive are both specified to QEMU, using either -drive or -fda / -hdb, QEMU always tries to boot from HD. I would like to boot from FD and mount HD later. Can this be done? Does anyone have a bootable, partitioned MSDOS HD image for testing? The 100Mb image from @Mellvik won't boot under QEMU. If I can get an MBR boot image, and mount it later while booting ELKS from floppy, it would be possible to enhance doshd.c to allow 4 BIOS 'hard drives' and two floppies while being able to test it. After working, directhd.c could be changed, although not sure whether that would work for flash disk.

Is this worth doing? Many flash drives are larger than 8MB.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Mellvik commented 4 years ago

BTW - I'm not an QEMU expert, but I believe you can us the index= parameter with -drive, -fda etc. to tell it which sequence you'd prefer for booting.

tkchia commented 4 years ago

Hello @ghaerr,

When a floppy and a hard drive are both specified to QEMU, using either -drive or -fda / -hdb, QEMU always tries to boot from HD. I would like to boot from FD and mount HD later. Can this be done?

Does your version of QEMU have a -boot option? -boot order=a in combination with -fda and -hda is what I use to request QEMU to boot from the floppy drive.

Thank you!

tkchia commented 4 years ago

By the way, I had been trying to modify setup.S to retrieve the drive geometries for more than two hard drives --- specifically, to modify the parts for setting these setup variables:

!   0x80:   BIOS data for harddisk 0, 6 bytes
!           0x80, no. of heads, 2 bytes (0, no disk)
!           0x82, no. of sectors/track, 2 bytes
!           0x84, no. of cylinders - 1, 2 bytes
!   0x86:   BIOS data for harddisk 1, 6 bytes
!           0x86, no. of heads, 2 bytes (0, no disk)
!           0x88, no. of sectors/track, 2 bytes
!           0x8A, no. of cylinders - 1, 2 bytes
!   0x8c:   data for floppy disk 0, 6 bytes
!           0x8C, no. of heads, 2 bytes (0, no disk)
!           0x8E, no. of sectors/track, 2 bytes
!           0x90, no. of cylinders, 2 bytes
!   0x92:   data for floppy disk 1, 6 bytes
!           0x92, no. of heads, 2 bytes (0, no disk)
!           0x94, no. of sectors/track, 2 bytes
!           0x96, no. of cylinders, 2 bytes
!   ...
...
#ifdef CONFIG_HW_HARD_DRIVE
    call    gethd       // Get geometry for harddisk 0/1.
#endif
#ifdef CONFIG_HW_FLOPPY_DRIVE
    call    getfloppy   // Get geometry for floppy disk 0/1.
#endif

It seems though that this portion is closely coupled with the way the minor numbers are assigned in the kernel (bios_gethdinfo() in doshd.c). I am wondering if I should just remove this part of setup.S, and leave it to the kernel instead to retrieve the drive geometries.

Thank you!

ghaerr commented 4 years ago

Hello @tkchia,

I am wondering if I should just remove this part of setup.S, and leave it to the kernel instead to retrieve the drive geometries.

I noticed that too, and am wondering why all that code is in setup.S. My guess is that it was there in the early days because the BIOS function returns values in a bunch of registers. I think all of it should be moved to doshd.c, with a wrapper function in assembly used to collect the register variables to C.

The disk geometries only need to be known when first accessing the disk, in the disk driver, and all these kinds of things should be kept there. doshd.c could be cleaned up a bit, while adding this to it.

ghaerr commented 4 years ago

Hi @tkchia,

Another related issue is the kind-of mess between doshd.c, directhd.c, and especially floppy.c. Some discussion should be had as to which should be supported.

For instance, the direct floppy driver floppy.c is still wholly compiled with a big '#if 0' around the entire file. It should just be thrown out, IMO. Is the direct HD/IDE driver directhd.c supposed to operate without regard to the BIOS CHS values and just use config set values? I don't know.

ELKS is currently very dependent on having BIOS. Is there a point in keeping directhd.c as an option? If so, it should probably be rewritten to only operate doing low-level I/O outside the BIOS rather than trying to be a driver in its own right, which now complicates things by also trying to manage minor numbers, etc.

ghaerr commented 4 years ago

Hello @tkchia,

-boot order=a

Thanks, this allowed booting from a Minix floppy then mounting @Mellvik's 100MB MSDOS 6.22 HD on /dev/bda1 (still not bootable, but mounting works). After adding a printk in genhd.c::add_partition(), the boot console shows that image has an MBR at sector 1, and a 100MB partition starting at sector 63. This clarifies my understanding of ELKS MBR partition identification and thought I'd make a few comments clarifying some earlier statements.

Since /dev/bda and /dev/bda0 are the same (raw) device, /dev/bda0 should be removed and a /dev/bda4 added in the 'Make.devices' portion of the build, to properly handle up to 4 partitions per MBR image. I'll change this in a PR.

Secondly, if the 'setup.S' drive geometry retrieval is moved into the kernel, it will have to go prior to the call to setup_dev() in init.c::device_setup(), not into doshd.c as I previously stated. For the time being, it seems that partition identification will need to happen just after the device is initialized by genhd.c::setup_dev, like it is now.

A question then arises - do we want to support booting from MBR disks (FAT or Minix)? If the bootloader passed the partition number separately through boot identification then 'setup.S', it could be 'OR'd into ROOT_DEV in device_setup() and everything should work fine from there, since the partition identification would have already been done by setup_dev.

tkchia commented 4 years ago

Hello @ghaerr,

Thanks for your PR!

My guess is that it was there in the early days because the BIOS function returns values in a bunch of registers.

I am not so sure that is the reason --- the kernel's call_bios(.) routine is quite capable of returning multiple register values via its structure pointer. (Perhaps the setup.S code is yet another of those holdovers from older Linux/x86 kernels...)

ghaerr commented 4 years ago

Hello @georgp24,

PR #505 completes the work for this issue. When you have time after it is committed, please re-test this issue to confirm that FAT (or Minix) USB devices are mountable, and close this issue.

If an ELKS bootable image is copied onto a USB flash drive, that image should now also boot if you setup your PC BIOS for that.

georgp24 commented 4 years ago

I tested it now. When I posted this issue, I did not think about my disk layout and that this is much too complicated for ELKS. I was used that when I boot DOS from a CD it finds the first FAT32 partition (E:) and presents this as the C: drive. Here is my disk layout including one flash disk inserted (click to enlarge): Disk-Layout DOS 7.0 will find partition E: of the first disk (C:) and the USB flash disk (D:). Anyhow, now I booted ELKS from a 1.44 floppy disk (a USB floppy drive) and this finds both hard disks and the flash disk, so three hard disks altogether: ELKS-Boot However, I cannot mount the disks, including the flash disk, maybe because FAT32 is not working: ELKS-mnt When I committed the FAT support in 2017, it did include FAT32 support though: https://github.com/jbruchon/elks/blob/master/Documentation/text/fat_fs.txt

ghaerr commented 4 years ago

Thanks for testing. FAT32 works, you have mount the drive that matches the partition, this isn't too complicated for ELKS.

For the first picture, use /dev/bda1. For the second picture, use /dev/bdc1. Notice the 'Partitions:' scan at boot. Those numbers are the start sector and size of each partition, with the word 'skipped' on invalid partitions. See that bda1 starts at sector 2048, bdb1 is invalid, and bdc1 starts at sector 63.

mount -t msdos /dev/bda1 will work. You can't mount an MBR disk using /dev/bda, which is reserved for flat disks with no MBR.

georgp24 commented 4 years ago

I tried that, but /dev/bda1 did not work. However, I somehow managed to format the flash disk with FAT16 without MBR. Then I could mount the flash disk with /dev/bdc as you can see.

ELKS-mnt2

I wrote a FAT hd.bin image to the flash disk with dd.

ghaerr commented 4 years ago

I somehow managed to format the flash disk with FAT16 without MBR. Then I could mount the flash disk with /dev/bdc.

Good, glad to hear ELKS confirmed to support FAT flash drives now!

I tried that, but /dev/bda1 did not work

Send over the screenshot of the boot followed by a mount on /dev/bda1. I need to see the error messages.

ghaerr commented 4 years ago

What I meant to say is that I need to see the Partitions: line on boot and the FAT mount message on the same screen. Thanks, we're almost there.

ghaerr commented 4 years ago

@georgp24, Looking closely at your disk tables (thanks), it appears the reason why /dev/bda1 won't mount is that partition is an NTFS partition.

If that's incorrect, lets try running ELKS with just a single hard drive attached, and send the disk partition manager output for that disk to help me debug it.

ELKS will identify up to 4 primary partitions per disk.

Extended partitions are untested. If someone can send me an MBR disk image with extended partitions, that could also be tested.

georgp24 commented 4 years ago

Yes, /dev/bda1 is an NTFS partition, but ELKS should skip that then. I was not aware ELKS supports just 4 primary partitions per disk. Yes, the FAT32 partition is the fifth partition. As you can see, the third partition is not yet formatted. I could split that into a FAT32 and FAT16 partition for testing.

Here is the boot screen matching the FAT mount messages I posted above: ELKS-Boot2

ghaerr commented 4 years ago

was not aware ELKS supports just 4 primary partitions per disk.

It is supposed to support extended partitions, but I have never tested that, since I don't have any disk images with extended partitions on them.

I could split that into a FAT32 and FAT16 partition for testing.

For now, that would be great, since there has been little testing of the 4 primary partitions, which I'm thankful you are testing. Once we show that the 4 primary partitions working, we can worry about extended partitions. Ultimately, I am trying to get this issue fixed, which is booting from a FAT flash disk (primary partition).