Direct floppy driver enhancements

[Mellvik]

An important fix/enhancement update to the direct floppy driver primarily dealing with the 82077 advanced FDC chip and media change detection. Even though the 82077 its now supported, perpendicular recording (2880k floppies) is still unavailable, a conscious choice based on the fact that media and drives supporting the format are very rare.

Update summary:

• 82077A FDC support which enables the 16 bit FIFO and implied seeks. No big deal really - the FIFO makes DMA timings more flexible, but the slow systems we're typically running on don't stress memory cycles all that much anyway. The implied seeks are useful in that explicit seeks (and thus seek interrupts) are (almost) eliminated which may speed up things somewhat. OTOH, there aren't many vintage systems out there with this advanced chip anyway, so the advantage is limited.
• The activation of implied seeks delivered a couple of surprises.
  • A bug in QEMU that manifests itself when running 360k images in 1.2M drives and seeking past track 20, is eliminated by implied seeks (and QEMU always 'runs' the 82077 chip). OTOH, on a physical system, a 360k floppy in a 1.2M drive WILL NOT WORK with implied seeks. Both of these cases are handled in this version of the driver, which uses a global running_qemu flag set in the directhd driver and defined in blk.h. [The QEMU hack that was introduced in an early commit in this PR was removed when the implied seek trick was discovered.]
  • The need to change FDC configuration to accommodate the 360k case complicates the driver logic somewhat: When copying floppy-to-floppy and one of them is 360k medium in a 1.2M drive, the FDC must be (re)configured per transaction, a situation that may call into question the benefit of activating implied seeks in the first place.
• [Experimental] Like before the driver uses a 18 sector track buffer in low memory to speed up reads. New in this version (and somewhat alleviating the problem discussed above) is that the track buffer is now a cylinder buffer for smaller drives, which typically have 18 sectors per cylinder or less. However, the jury is still out as to whether this is advantageous or not. If a small floppy is the root FS, cylinder buffering is most likely a disadvantage, the same goes for file copying between floppies unless the copy buffer exceeds the stack capacity, while file copying from floppy to HD will definitely benefit. Your milage will vary wildly, and this capability will become configurable in the future. An interesting worst case scenario is running the cmp command - one file on a 360k FAT floppy in a 1.2M drive, the other on a 720k FAT floppy in a 1.44M drive. cmp reads 2k from each file and compares, then repeats: Every other read operation reads a full cylinder. A detailed benchmark is required in order to find out, preferably using a raw device, bypassing the buffer system. In a more general case, an important question is whether the FDC manages to read the 2nd track without losing a disk rotation. In any case, one rotation being ⅙ of a second is a steep price to pay if the buffered data is useful only half the time or less. BTW - this particular setting (low cap media in high cap drives) is also interesting from another vantage point: Underscoring the importance of keeping track of the FDC configuration, which needs to be changed for just about every operation.
• This version also takes a stab at cleaning up the MEDIA_CHANGE detection logic - in several steps.
  • An ifdef enables complete removal of the code dealing with the DIR register and media change. This is good for pre-AT systems which have no means of media change detection and the code is wasted. In fact any system will run just fine with the code commented out, but with the risk of compromising floppies if changed while mounted.
  • Further, if the media change code is included, it will disable itself if the system is pre-AT, which means the code will run as is on older systems.
  • What's still missing is the disabling of the media change support on physical 360k drives (CMOS type 1). These drives have no media change line.
  • Also missing is a test for the media change line being permanently on. Some 5.25in drives use the media change line (pin 34) as a ready signal, always on when the motor is spinning and head loaded. Such drives will now hang the driver and (probably) the system/OS.
• The code has been reorganized somewhat so that the configuration-related ifdefs appear together for easier apdaption to local requirements.
• The directhd driver has been updated to recognize 'QEMU' as part of the IDE drive serial number and set a global flag accordingly. Useful for the floppy driver to get around the 360k floppy bug, and possibly other kernel level subsystems. In order to benefit from they 'magic', the IDE ID string must be set in the qemu startup script, like this: -global ide-hd.serial=EQUMI_ED. Notice the byte swapped text. Only the first two bytes of the ID (EQ) are used for the test.

[ghaerr]

Exciting @Mellvik, good to hear about what you've learned with real hardware testing, and very informative!

the FDC must be (re)configured per transaction, a situation that may call into question the benefit of activating implied seeks in the first place.

"Reconfiguring" the FDC is seemingly very fast, it consists of sending just 3 bytes to the FDC. Do you think that somehow affects floppy read/write speed in any meaningful way? OTOH, I don't know how implied seeks work - one would think that waiting for a seek complete interrupt might be quite a bit slower. How does the FDC know when to proceed when using implied seeks, or are both the seek methods about the same speed (other than reset/recalibrate operations)?

82077A FDC support which enables the 16 bit FIFO and implied seeks. No big deal really - the FIFO makes DMA timings more flexible

Good to know the 82077 chip support works on real hardware. I had enabled that but never tested on a PC. Can you explain a bit more about how the FIFO affects DMA timings? I thought it only affected the speed with which one could write commands to the FDC.

OTOH, on a physical system, a 360k floppy in a 1.2M drive WILL NOT WORK with implied seeks.

You mean - one can't send double the track number to the FDC for this case? I had imagined that the issue relates to physically positioning the head at on a stepper that moves half the 360k step size due to the increased density of 1.2M.

In fact any system will run just fine with the code commented out, but with the risk of compromising floppies if changed while mounted.

I'm not really sure how the risk of compromised floppies changes at all - the kernel nor driver doesn't do anything at all if the media is changed, other than clearing the DIR flag and reseting the FDC, right? Any media change will remain entirely unnoticed by the kernel, along with a damaged filesystem on writes or mismatched/improper data on reading the floppy.

the track buffer is now a cylinder buffer

An interesting worst case scenario is running the cmp command - one file on a 360k FAT floppy in a 1.2M drive, the other on a 720k FAT floppy in a 1.44M drive. cmp reads 2k from each file and compares, then repeats: Every other read operation reads a full cylinder.

Nice work! It would be interesting to learn whether a cylinder buffer helps... what do you think of the idea of just using a stop-watch style timer and just measuring seconds while booting up and running some basic commands, is that much faster or slower?

Also, the cylinder buffer only works for 360k floppies, since the DMASEG region is only 9k, right? Or was that region expanded to work with 1.2M floppies? If the floppy is too large, does the fallback to track-only buffering work (e.g., for 1.2M floppies)?

In a more general case, an important question is whether the FDC manages to read the 2nd track without losing a disk rotation. In any case, one rotation being ⅙ of a second is a steep price to pay if the buffered data is useful only half the time or less.

Agreed. Can we use the 10ms jiffies timer for that or do we need higher resolution?

the directhd driver has been updated to recognize 'QEMU' Useful for the floppy driver to get around the 360k floppy bug

This only works if QEMU is passed an HD on startup, right? Does the 360k floppy track doubling bug come back if QEMU is started as boot from floppy only?

It sounds like from your research that TLVC should be configured for real hardware by turning implied seeks off, and always using the interrupt seek mechanism and track doubling sent to the FDC, right? So all we're talking about is various kluges (from my or your driver) for dealing with QEMU and 360k floppies in a CMOS type 1 (1.2M) drive.

Thanks for the detailed report of your findings. I also appreciate the source code variable comments; l will move them over to my version of the driver as well :)

[Mellvik]

Exciting @Mellvik, good to hear about what you've learned with real hardware testing, and very informative!

Hi @ghaerr - indeed it has been and interesting (and at times surprising) journey back down to the (almost) signal level of floppy drives :-)

the FDC must be (re)configured per transaction, a situation that may call into question the benefit of activating implied seeks in the first place.

"Reconfiguring" the FDC is seemingly very fast, it consists of sending just 3 bytes to the FDC. Do you think that somehow affects floppy read/write speed in any meaningful way? OTOH, I don't know how implied seeks work - one would think that waiting for a seek complete interrupt might be quite a bit slower. How does the FDC know when to proceed when using implied seeks, or are both the seek methods about the same speed (other than reset/recalibrate operations)?

The observation about reprogramming the FDC is just me being annoyed about introducing an extra step. AFAIK there is no real penalty except a few bytes of code to traverse. As to the implied seeks, the benefit is real - although not 'big'. With a lot of head movement there are interrupts and command routines saved leaving more time for the processor to do other things. The FDC always knew about the seeks anyway (but had us doing them), doing it on its own without 'disturbing' us is just great.

82077A FDC support which enables the 16 bit FIFO and implied seeks. No big deal really - the FIFO makes DMA timings more flexible

Good to know the 82077 chip support works on real hardware. I had enabled that but never tested on a PC. Can you explain a bit more about how the FIFO affects DMA timings? I thought it only affected the speed with which one could write commands to the FDC.

This is on a level (cycles and timing) at which I'm very rusty. My take is that the FIFO has minimal effect if any on old (slow) systems regardless - both the command sequences and the DMA. The added flexibility is good on faster systems (faster processors and peripherals) with some real contention for memory access and 2880k floppies (1MHz transfer rate). Again, my take - and bottom line, probably no noticeable benefit to us.

OTOH, on a physical system, a 360k floppy in a 1.2M drive WILL NOT WORK with implied seeks.

You mean - one can't send double the track number to the FDC for this case? I had imagined that the issue relates to physically positioning the head at on a stepper that moves half the 360k step size due to the increased density of 1.2M.

What I'm saying is that 1) implied seeks don't work when there are doble steps to be made, and 2) regular seeks don't work either. Very surprising and it took me quite some time to accept, not the least since it worked so well on QEMU. While researching the problem, I when to the step of using the READID command after a doble step seek to see which track it actually ended on, and it read back just fine. The head is where it should be, but still the FDC doesn't like it. Weird.

In fact any system will run just fine with the code commented out, but with the risk of compromising floppies if changed while mounted.

I'm not really sure how the risk of compromised floppies changes at all - the kernel nor driver doesn't do anything at all if the media is changed, other than clearing the DIR flag and reseting the FDC, right? Any media change will remain entirely unnoticed by the kernel, along with a damaged filesystem on writes or mismatched/improper data on reading the floppy.

This is only partly true. The kernel does not get notified (but could) about media change. but the driver notices and runs through a reset sequence to get rid og the media change (DIR) bit. This invalidates the track buffer which will reduce the damages of the change by making it visible sooner. Not saving the removed media but possibly the newly inserted media.

the track buffer is now a cylinder buffer

An interesting worst case scenario is running the cmp command - one file on a 360k FAT floppy in a 1.2M drive, the other on a 720k FAT floppy in a 1.44M drive. cmp reads 2k from each file and compares, then repeats: Every other read operation reads a full cylinder.

Nice work! It would be interesting to learn whether a cylinder buffer helps... what do you think of the idea of just using a stop-watch style timer and just measuring seconds while booting up and running some basic commands, is that much faster or slower?

I do plan to run some benchmarks on this. Sequential reads (floppy copy) vs. random reads (as in the cap example). Will get back to that.

Also, the cylinder buffer only works for 360k floppies, since the DMASEG region is only 9k, right? Or was that region expanded to work with 1.2M floppies? If the floppy is too large, does the fallback to track-only buffering work (e.g., for 1.2M floppies)?

The buffer is the same 18 sectors and the cylinder buffering is good for 720k and 360k, both of which have been part of the testing.

In a more general case, an important question is whether the FDC manages to read the 2nd track without losing a disk rotation. In any case, one rotation being ⅙ of a second is a steep price to pay if the buffered data is useful only half the time or less.

Agreed. Can we use the 10ms jiffies timer for that or do we need higher resolution?

No, 10ms res is actually fine for this testing.

the directhd driver has been updated to recognize 'QEMU' Useful for the floppy driver to get around the 360k floppy bug

This only works if QEMU is passed an HD on startup, right? Does the 360k floppy track doubling bug come back if QEMU is started as boot from floppy only?

This always works because the IDE ID is read at boot time.

It sounds like from your research that TLVC should be configured for real hardware by turning implied seeks off, and always using the interrupt seek mechanism and track doubling sent to the FDC, right? So all we're talking about is various kluges (from my or your driver) for dealing with QEMU and 360k floppies in a CMOS type 1 (1.2M) drive.

Close. First, CMOS type 1 is a 360k drive, not 1.2M. Detecting a (real) 360k drive in CMOS should disable all DIR/media change action, pretending it's not there. I'm not doing that yet, pending getting my Compaq Plus (8088XTclone) up and running later this year.

Second, I'm going to keep the implied seeks until more testing has been done - simply because I have the hardware supporting it. And most likely disable the cylinder buffer - again pending testing.

What it seems like though, is that disabling/ignoring 82077 support is the optimal thing to do for 'vintage systems' - saving RAM and complexity. That's where 99% of them are anyway.

Thanks for the detailed report of your findings. I also appreciate the source code variable comments; l will move them over to my version of the driver as well :)

This driver is incredibly complex - there just cannot be enough comments to remind us why all these 'weirdities' are there. Actually, commenting out all the DIR/media change stuff was kind of revealing in how much code went out.

[ghaerr]

Thanks for your comments @Mellvik. Agreed this is a very complicated driver!

What I'm saying is that 1) implied seeks don't work when there are doble steps to be made, and 2) regular seeks don't work either.

What is the mechanism that does work on real hardware for 360k floppies in a 1.2M drive, if regular seeks don't work, or is this comment concerning QEMU only?

What it seems like though, is that disabling/ignoring 82077 support is the optimal thing to do for 'vintage systems' - saving RAM and complexity.

From your hardware testing, will the driver work on all systems (new and old, but possibly less efficiently), with all 82077 and DIR support removed?

commenting out all the DIR/media change stuff was kind of revealing in how much code went out.

Yes, a large amount of unneeded code until the kernel is enhanced to do something on media change notification.

The kernel does not get notified (but could) about media change. but the driver notices and runs through a reset sequence to get rid og the media change (DIR) bit. This invalidates the track buffer which will reduce the damages of the change by making it visible sooner. Not saving the removed media but possibly the newly inserted media.

IMO, instead of "reducing damages sooner", perhaps flat force-call the equivalent of the close entry point would be best, for the DIR code to be actually useful? That would immediately stop all reads and writes until the floppy was re-mounted.

I remember somewhere you had mentioned the driver "kind of" tries to support 4 drives - but doesn't really - since an additional adaptor board would be required, and the driver doesn't attempt to handle the additional FDC I/O address, amongst other issues. Given there's no way for CMOS to even identify 4 drives, what do you think of the idea of removing "supposed" support for 4 drives to make the driver smaller? I have been vacillating on that, especially after reading an article stating that almost no IBM PCs supported four drives directly from the manufacturer (only the NEC PC-98 did, from what I've seen)?

[Mellvik]

@ghaerr - I some times get this feeling we're curators at some tech museum :-) maybe even archeologists trying to figure out how ancient societies lived and played.

What makes it less ancient though is the fact why we actually lived through the times when these artefacts were the hottest thing on earth. As evidenced by this copy of Byte Magazine I keep in my bookshelf:

Not a complaint, rather the opposite - interesting and fun, if frustrating at times.

Thanks for your comments @Mellvik. Agreed this is a very complicated driver!

What I'm saying is that 1) implied seeks don't work when there are doble steps to be made, and 2) regular seeks don't work either.

What is the mechanism that does work on real hardware for 360k floppies in a 1.2M drive, if regular seeks don't work, or is this comment concerning QEMU only?

Disabling implied seeks in the 82077 is the only way, just like enabling implied seeks is the only way on QEMU. This is the reason the FDC must be reconfigured all the time when one drives use implied seeks, the other doesn't . Not a big deal as commented before, but annoying - as in: 'why didn't they fix this' or 'what is it I haven't understood yet'?

From your hardware testing, will the driver work on all systems (new and old, but possibly less efficiently), with all 82077 and DIR support removed?

Yes. No 2880 support, but it isn't fully implemented anyway.

commenting out all the DIR/media change stuff was kind of revealing in how much code went out.

Yes, a large amount of unneeded code until the kernel is enhanced to do something on media change notification.

The kernel does not get notified (but could) about media change. but the driver notices and runs through a reset sequence to get rid og the media change (DIR) bit. This invalidates the track buffer which will reduce the damages of the change by making it visible sooner. Not saving the removed media but possibly the newly inserted media.

IMO, instead of "reducing damages sooner", perhaps flat force-call the equivalent of the close entry point would be best, for the DIR code to be actually useful? That would immediately stop all reads and writes until the floppy was re-mounted.

Yes, that's the logical thing to do. Either 1) forced umount, flush buffers, report 'catastrophe', or 2) immediately set RO, give the user a chance to recover (put the media back in). There may be other approaches too, of course it would have to be kept very simple...

I remember somewhere you had mentioned the driver "kind of" tries to support 4 drives - but doesn't really - since an additional adaptor board would be required, and the driver doesn't attempt to handle the additional FDC I/O address, amongst other issues. Given there's no way for CMOS to even identify 4 drives, what do you think of the idea of removing "supposed" support for 4 drives to make the driver smaller? I have been vacillating on that, especially after reading an article stating that almost no IBM PCs supported four drives directly from the manufacturer (only the NEC PC-98 did, from what I've seen)?

It's complicated (great term, isn't it?). The driver can easily support 4 drives on the same controller but such hardware is really hard to come by. The PC and XT supported 4 flpy drives, the XT/286 (I didn't even know it existed till last week) and AT did too - or could be made to, but at that point they started mucking around with the cables, using the drive select lines for drive 3 and 4 for other things. So in order to actually get most systems AT and up to run with 4 drives and one controller, you'd have to do some not-too-complicated-but-annoying work on cables. BTW - IIRC - the CMOS can actually indicate 4 drives, but not what they are (types for 3 and 4). IOW - the sane answer to your question is to change it and save the RAM. The less sane answer is 'what if we're running an original PC and want to add 4 drives?' I'm thinking sanity must rule ...

[ghaerr]

As evidenced by this copy of Byte Magazine I keep in my bookshelf

Nice! I think I have that issue archived somewhere... I'll have to look for it since I have two Compaq 386s :)

Disabling implied seeks in the 82077 is the only way, just like enabling implied seeks is the only way on QEMU.

I see, so the track doubling issue is only when running with an 82077, thanks. My driver will use an 82077 if present, but never turns on implied seeks. However, my QEMU hack involves not doubling the track number. I'll try your idea of enabling seeks instead for the QEMU fix instead, since its cleaner and QEMU always emulates the 82077, as you stated.

I'm thinking sanity must rule

Agreed. More important to get more people running and testing TLVC & ELKS first than worrying about every possible configuration - especially when we're now equivalently rewriting the floppy BIOS for every PC this might run on! As you've pointed out, the possible variations between machines is seemingly endless.

[ghaerr]

OTOH, on a physical system, a 360k floppy in a 1.2M drive WILL NOT WORK with implied seeks.

1) implied seeks don't work when there are doble steps to be made, and 2) regular seeks don't work either. Very surprising and it took me quite some time to accept, not the least since it worked so well on QEMU.

While researching the problem, I when to the step of using the READID command after a doble step seek to see which track it actually ended on, and it read back just fine. The head is where it should be, but still the FDC doesn't like it. Weird.

I'm now trying to understand what might be going on with regards to a real 82077 accepting implied seeks and moving to the correct cylinder, but still not working. Very strange, as you say! What error was the FDC giving for the transfer (since I assume there is no specific error on implied seeks)?

Also, above you say that "2) regular seeks don't work either". I thought that non-implied seeks did work on 82077 on real hardware.

Right now, both our driver versions use implied seeks only for QEMU when stretch is set, thereby using the older original (8272A) code path for everything else. That code path uses regular seeks for stretch floppies and ignores 82077 chip enhancements, unless I'm getting confused.

Agreed we can't use QEMU for testing this precisely. I think there's a pretty good probability that my 386 desktop has an 82077 but I haven't checked yet.

[Mellvik]

OTOH, on a physical system, a 360k floppy in a 1.2M drive WILL NOT WORK with implied seeks.

1. implied seeks don't work when there are doble steps to be made, and 2) regular seeks don't work either. Very surprising and it took me quite some time to accept, not the least since it worked so well on QEMU.

While researching the problem, I when to the step of using the READID command after a doble step seek to see which track it actually ended on, and it read back just fine. The head is where it should be, but still the FDC doesn't like it. Weird.

I'm now trying to understand what might be going on with regards to a real 82077 accepting implied seeks and moving to the correct cylinder, but still not working. Very strange, as you say! What error was the FDC giving for the transfer (since I assume there is no specific error on implied seeks)?

that's right, seeks report no errors regardless. The error is sector not found - which only tells us that the fdc expects something else than delivered.

Also, above you say that "2) regular seeks don't work either". I thought that non-implied seeks did work on 82077 on real hardware.

yes, but only if you turn off implied seeks (for drives that need double stepping).

Right now, both our driver versions use implied seeks only for QEMU when stretch is set, thereby using the older original (8272A) code path for everything else. That code path uses regular seeks for stretch floppies and ignores 82077 chip enhancements, unless I'm getting confused.

not quite. As is, the tlvc driver - if 82077 is enabled - will use implied seek for everything except 360k in 1.2M drive in phys hw. In disabled, qemu & tlvc & 360in1.2 just won't work. Which is fine.

Agreed we can't use QEMU for testing this precisely. I think there's a pretty good probability that my 386 desktop has an 82077 but I haven't checked yet.

my guess is no. By the time th3 82077 became available, pentium-something was the normal (probably 1991 or later). Let me know when th3 verdict is in!