I'd love to help with Mac 800k

[andrewferguson]

Avid collector of all things old-Mac here. Also an amateur digital archivist, which is how I stumbled across FluxEngine (your Reddit marketing (tm) worked).

I've ordered the board and got it all assembled, and have successfully imaged a couple of PC disks (getting 869 kB/s on the bandwidth-side so I'm living dangerously!). But what I'm really after is better support for 800k Mac images, both read and (in an ideal world) write.

I have setup a Macintosh Classic II which can read/write both 1.44MB "standard PC" and 800k (definetely not standard) disks. And 400k single-sided too. I also have a USB floppy drive and an iBook G4, allowing me to transfer files to the Macintosh Classic II. And, of course, I have a FluxEngine, allowing me to image disks.

So, basically, my question is: how can I help? I'm happy to image disks, test disks, etc...* if it will potentially lead to better support for these old Mac disks - as many as you like, for as long as you like**. I'd love to see a time where FluxEngine is able to both read and write 800k Mac images that are also compatible with the various emulators such as Mini vMac and SheepShaver (both of which I also have setup and available for testing).

*One thing that, right now, is more difficult for me to do is to take an 800k disk image and write it onto a disk. This is because Disk Copy (disk image writing program on older Macs) is very picky about what sort of disk images it handles, usually limiting it to ones it created that have the correct resource fork data (don't get me started). So while I can easily copy files onto 800k disks, making 800k disks on the Macintosh Classic II is difficult right now. Having said that, if that's what you need, then I'll try and work out some way of doing it!

**Subject to the hardware playing nicely! I went through several 800k disks before I found one that formatted successfully, and the Classic II plays up sometimes. But hey, that's part of the fun!

[davidgiven]

That's awesome! Glad to have your help.

FluxEngine does already technically support Mac 800kB disks (and probably 400kB ones too), but I've only successfully tested it on KryoFlux streams sent to me by someone else. I do have a few 800kB disks that were posted to me by someone but for some unknown reason they don't image properly. I don't know yet whether it's a problem with the disks or something at my end. I have suspicions that certain types of disks don't image properly. I've just received a logic analyser to investigate this but haven't done it yet. So if you could just try to read some disks and see what happens...

Write support needs some work; it's currently very much bodged together for the Brother and needs some proper refactoring. That's one of the next things to do.

The other thing which I don't know is how to produce or consume useful disk images. Mac disks are awkward as the tracks are all different sizes. What formats do Mini vMac and SheepShaver support?

[andrewferguson]

OK! I've imaged some stuff that should provide a starting point. All of these images are taken using the same 800k disk, which was erased each time using the "Erase" option in System 7.1.

There are three groupings of files:

1. "blank": an image of a disk taken immediately after doing the "Erase" option in System 7.1. Note that it is not truly blank, and does contain a folder called "Desktop Folder" which System 7.1 hides.
2. "minisystem": a bootable System 6 disk image. Very minimal, containing only what is needed to boot. I've checked on my Macintosh Plus and this software on the disk I imaged does boot it.
3. "textfile": an erased disk with a single text file containing the string "this is a test". The name of the text file is "fluxengine"

So, as for the actual files I've attached: each "grouping" described above consists of three files:

1. A ".img" file. This is created using "sudo ./fluxengine mac", then subsequently renaming the file to match the name of the grouping.
2. A ".image" file. This is created on the Macintosh Classic II using the "Disk Copy 4.2" software.
3. A ".log" file. This is the final summary that FluxEngine prints after creating the ".img" file.

I've tested the ".image" files (the ones made by Disk Copy 4.2) and they all work in both the Mini vMac and SheepShaver emulators. Sadly none of the images made by FluxEngine work in the emulators.

From looking at the output summary by FluxEngine, I think it is apparent that the 800k disk I'm using is not in the best of shape, with quite a lot of errors. That being said, System 7 formats it fine (and refuses to format many of my other 800k disks, so it is a bit choosy), and when loaded up with the Minisystem it works fine as a System Disk. It's probably a typical example of a fairly-well looked after Mac disk!

Let me know if there's anything else I can do! Now for the files (I had to use a zip as GitHub didn't like the images files).

[davidgiven]

Thanks for those; that does sound like the behaviour I'm seeing with my disks, which suggests that there is a problem with the imaging. Curses. Although the pattern of errors in the log files looks kinda weird.

Can you run the fluxengine read mac stage with --write-flux=something.flux? Given the error rate you probably also want --retries=0 to prevent the drive from spending all of its time grinding. The --write-flux option will generate files containing the raw magnetic flux data, which can be decoded again without needing access to the physical disk. If you can send me those I can try to decode them.

[andrewferguson]

OK! (Sorry for the delay, sleep then work took up my time!). Attached is a new zip, files2.zip, containing the exact same groupings (minisystem, blank and textfile) as last time, and a new .flux file for each of the groupings. Note that the other files are not identical to the files sent yesterday: although the same in purpose, I've recreated them all today to ensure an equal test.

And thanks for the tip of --retries=0. Not only did it stop all the grinding noises, it cut the time by about 2/3.

Let me know how else I can help! files2.zip

[davidgiven]

There is something wrong with the sampler and it's dropping pulses. See #75. I'm investigating.

[andrewferguson]

It's not a bandwidth issue on my end, is it? Two days ago I was fairly close to the 850 kB/s limit, and I didn't bother checking again yesterday (although I may have had a USB memory stick plugged in as well).

That being said, I was able to make a perfectly working copy of an IBM 1.44MB disk yesterday as well, so I don't think the bandwidth I was getting was too bad. Any idea how much 800kB Mac disks need?

[davidgiven]

No, if it were bandwidth you'd get a hard error and the read would stop. This is something altogether more subtle. I have no idea why it doesn't show up on IBM disks --- it's extremely weird.

[andrewferguson]

When I get back tonight I'll try formatting the disk as 400kB single-sided and seeing if there is any difference. (Unfortunately I don't think I have a brand new, unused DD disk to try).

Is it worth me trying to generate a flux image using another software decoder (ie: "ibm" rather than "mac") or is the flux code shared between all the software decoders?

[davidgiven]

The flux images should all be the same; the only difference is that the decoder is used to check whether the image was read correctly (floppy disks suffer from perfectly normal transient errors, which need to be retried, but you need to know the data encoding in order to check for these).

IIRC the 400kB format is just the same as the 800kB one but just uses one side, so I don't think there'll be any difference there.

[davidgiven]

So, er, turns out that the sampling error could well have been fixed for a while now. You just need an extra command line parameter.

$ ./fluxengine read mac -s /tmp/tosend2/blank.flux:s=0 --bit-error-threshold=0.3
H.SS Tracks --->
0. 0 ...........................................................B....................
0. 1 ................................................................................
0. 2 .....................B..........................................................
0. 3 .............................................................B..................
0. 4 ......................B.........................................................
0. 5 ................................................................................
0. 6 ................................................................................
0. 7 ................................................................................
0. 8 ..............................B.................................XXXXXXXXXXXXXXXX
0. 9 ................................................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
0.10 ................................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
0.11 ................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Good sectors: 795/960 (82%)
Missing sectors: 160/960 (16%)
Bad sectors: 5/960 (0%)
80 tracks, 1 heads, 12 sectors, 524 bytes per sector, 491 kB total

The Bs represent bad sectors, which will probably go away on real hardware if you let it retry failed reads; the Xs represent sectors which aren't present, which is perfectly normal on Mac disks because the inside has fewer sectors than the outside.

Could you give that a try and let me know if it works? You may want to update from git (and rebuild and reprogram the firmware) first. Thanks!

[andrewferguson]

OK, updated from git and reprogrammed the PSoC. Unfortunately it still didn't work. The exact command I ran was sudo ./fluxengine read mac --bit-error-threshold=0.3. I see in the example command you gave about you had ":s=0", which I presume means only do the first side. For you, this created a disk image 491kB in size, which is about right. However I didn't do the ":s=0", and yet I still got a disk image 491kB in size. Perhaps this is the reason it didn't work? I only did "textfile" and "blank" this time, and didn't make Disk Copy images (can do easily, just wanted to save time to let you know about this ASAP).

After writing the above I realised I had forgotten to recompile the tool after updating from git. So I did that, but it still didn't work. Also, the output is now.... weird. See the 'afterupdate.log' and 'afterupdate.img'. This is of the "blank" disk.

files3.zip

[davidgiven]

What it's done is that it's read both sides, but it's found sectors on side 1 which claim they're from side 0, which is a conflict. These are the C sectors in the map. FluxEngine always trusts the sector header to get the sector ID rather than the physical location (which is sometimes wrong).

The weirdness is because there's a bit error in a sector header on side 1 so it's claiming to be sector 255. I've put in a sanity check for this as it's clearly wrong. (In git branch 'mac'.)

A bit more investigation with my new tools has revealed the multiple reads of a Mac disk are definitely losing pulses. I've reopened #75 (more info there)...

[davidgiven]

I'm still thinking about this!

The good news is that I've rewritten most of the reading and writing firmware, and it's producing much better results than before. My disks are still not great but the difference is astounding. Given that your disks were mostly generating better results, I'd be interested to know if this helps you at all if you're interested in giving it a try.

You'll need to rebuild the client and reinstall the firmware.

Thanks!

[andrewferguson]

Terribly sorry for not responding sooner - end-of-year uni exams followed by Christmas limited my free time.

I've re-flashed the new firmware and rebuilt the software, but frustratingly I'm getting the Error: cannot find the FluxEngine (is it plugged in?) message when I try fluxengine rpm.

I've done the usual, trying different USB cables, checking that ninja-build, libusb-1.0-0-dev and libsqlite3-dev are all installed. I've also re-flashed the old firmware to see if that made a difference (it doesn't). Frustratingly I didn't check to see if everything still worked before I rebuilt the client / flashed the updated firmware. I've also verified that lsusb detects the board as Bus 001 Device 010: ID 1209:6e00 Generic, which matches the PID / VID in protocol.h.

I don't want to turn this issue into teh n00b can't do USB so I'm happy to open a new issue if that would make troubleshooting easier. (Or I welcome the immediately-obvious answer as to what I'm doing wrong that will make me go d'oh! and look stupid :)

[davidgiven]

No worries --- I'm now on holiday and away from my hardware...

It is possible I screwed things up. I don't know why it would be showing up with the right USB ID but still uncontactable, though. Do you need sudo to give the client access to USB? I have one machine where I do and one where I don't and I keep getting it wrong.

If that's not it, then I'd suggest forgetting about it until January and I get home to my hardware. Currently I'm about 1500km away from it...

[andrewferguson]

Well silly me. I wasn't doing sudo.

I've been distracted by other things for the past few weeks, but I'll try and get some Mac disks imaged with the new firmware / client by the end of this week.

[Icekhaos]

@davidgiven Is your user perhaps in the dialout group on one machine and not the other? This is a common cause of this kind of issue for serial connections under Linux.

Also, status on this?

[andrewferguson]

On my side, the status is Eek, I'd completely forgotten about this. I cannot honestly remember where I got last with this - so I'll assume nothing and re-flash the latest firmware / use the latest software before trying to image some disks.

[davidgiven]

Yeah, I, uh, got busy with other things. (I wrote a compiler! http://cowlark.com/cowgol)

At my end: someone contributed some drastically bugfixed sampler code, which is giving me much cleaner Mac disk reads on my hardware. Also, someone else contributed a semi-improved decoder. When this works, it works very well, but it also frequently makes things worse. I'm still working on that one and it's not merged.

If you want to try imaging some disks, that'd be awesome, but I don't need the whole set --- I know it takes forever.

[Icekhaos]

Nothing about the dialout group? Damn.

Also that is a perfectly acceptable use of time! I might actually DM you... somewhere? about this.

[andrewferguson]

So after a slight .... delay when I couldn't get the Mac to boot (I really need to replace the capacitors on the logic board!), I have made an image of the disk. The disk used is a blank 800k disk (freshly formatted in the mac).

The command I used was fluxengine read mac --bit-error-threshold=0.3. I've attached both the resulting disk image and the log produced, in a zip file (since GitHub doesn't like disk image files). (I check to make sure the setup was working correctly by imaging a regular IBM disk, which worked fine).

The output was a 490kB disk image... which is around half the size I would expect for an 800k disk. On the positive side of things, there was only a very small number of bad sectors (18 out of 960) and the 490kB disk image compresses very well in the zip, which I would expect for a blank disk.

I did try the disk image in the Mini vMac emulator, and it wasn't recognised, so evidently there are still some issues.

As before, I'm happy to do whatever imaging combinations are needed to help you debug this!

mac-read.zip

[Icekhaos]

Would the flux reading not be more useful?

[andrewferguson]

@Icekhaos That's a very good point. I can certainly do so.

@davidgiven what would you prefer? (And any specific command-line options I should pass?)

[davidgiven]

Actually, it's the log that's the really useful bit.

What I see is that the sectors on side 0 have the same IDs as the sectors on side 1, but other than that it's a completely clean read. This could mean it's a 400kB disk, or it could mean that I'm wrong about where the side bit is stored (see decode_size() in arch/macintosh/decoder.cc). I'm thinking it's more the latter. The conflicting sectors are because FluxEngine can see two copies of the same sector which are different (the ones on each side). The rest of the sectors won't conflict because the two sides store the same data --- the most likely explanation is that they're empty.

I think I may have a disk here which exploits the same symptoms... I'd always thought it was a 400kB disk. Let me get back to you on that one.

[davidgiven]

Whoops, I fixed this but attached the message to the wrong bug... turns out I was testing the wrong bit in the side byte, which is highly embarrassing. Current head of git should have a fix for it (no firmware upgrade needed). Judging by the rest of your log, I think that should give you clean reads.

[andrewferguson]

Aha, running with the new client does produce disk images of a much more sensible size - 982kB. (And like before, the disk image compresses very well (down to 2.1kB), suggesting the blank disk is being read successfully.

Unfortunately, the disk isn't being recognised by either the Mini vMac or SheepShaver emulators (they recognise it as a disk image, but not as a Macintosh-formatted disk). I'm guessing this is where things get a bit more complicated, with checking of disk image specs, and a "spot the difference" between "genuine" Mac images and the ones the FluxEngine produces.... I admit I'm completely out of my depth when it comes to these sorts of things, but I can pass along the following links describing how the Mini vMac emulator deals with disk images: How to work with Disk Images Emulated Hardware - Floppy Drives

In particular, the following passage in the second link caught my attention:

The preferred disk image format for Mini vMac is a file containing just the image data, with no header, trailer, or resource fork (and therefore without checksums or tag data). Mini vMac will accept disk images in formats with trailing information and no header, except that it will pass an incorrect disk size to the emulated computer, which affects programs such as Disk First Aid.

Oh, and here's the output from the latest read: mac.zip

[davidgiven]

Good news. The next stage is to emit disk images which emulators can understand. I'm not at all surprised the existing ones don't work --- the triangular disk image shapes are a pain.

There's actually an open bug to support DiskCopy 4.2 images (#85), and it looks like vmac supports these according to the link, so I'll take a look at those.

[kristomu]

I have a flux file of an 800k floppy with an attendant .img file produced by another flux imager. Linux is able to mount this .img file, but not the one FluxEngine produces.

I suspect the format is simply each track in order, each sector up to the liimit that can be put on the disk (no spacing for invalid sectors), all without metadata. But I'll send you the image by mail so you can take a look.

While I'm reasonably sure there's very little private data on the disk, it's from a computer that's not strictly speaking mine, so I don't feel like I should put it up here in public.

[andrewferguson]

Seemingly GitHub doesn't send email alerts when PRs are mentioned in an issue, so I didn't realise that you'd patched this until today.

I've now imaged several Mac disks successfully, and they work great in the various emulators that I have (Mini vMac and SheepShaver). Thanks so much for spending the time to get this to work - it'll be a game changer as I no longer need to rely on old hardware to image disks!

A couple of thoughts on potential extensions (no doubt "possible extensions" is the last thing you want to hear):

• It would be nice to support the "raw" image format that the emulators use, as unless compiled with the options enabled, Mini vMac (and I believe SheepShaver as well) cannot write to DiskCopy images (quite sensibly, it's a storage / preservation format, rather than a in-use-disk format). I'm wondering if it would be as simple as stripping off the DiskCopy headers/footers to expose the raw disk image? (For comparison I can create an 800k 'raw' disk image that works in Mini vMac using hfsutils on Linux using the commands dd if=/dev/zero of=disk.raw bs=1k count=800 && hformat disk.raw ).
• What do you think the chances are of ever having write support for Mac disks in FluxEngine? Being able to read and write mac disks would be unbelievably cool (many folks in the Vintage Mac community have a "later-era" Mac used solely for transferring files from 1.44MB IBM formatted disks to the 400k / 800k disks required by earlier Macs - if FluxEngine could write to disks these separate Macs would no longer be necessary).

A huge thanks again for adding support for Mac disks, and all the time and effort it's taken to do this. Feel free to close this issue now and work on the above ideas later (or never - no obligations!). I'm always happy to try out things needing "real" Mac hardware, if you ever need it.

Thanks again!

[davidgiven]

Fantastic --- I'm glad that is working for you!

Regarding raw disk formats: Mac disks are weird. There's that extra 12 bytes of metadata per sector, which has to go somewhere. DiskCopy files put all the normal data first, and then there's a chunk at the end of the file for the metadata, so by stripping off the header you effectively just get the 512-byte sector data. I believe that's what you get with dd too. I know that some applications need it and some don't, but don't know the details...

Regarding writing: yes, that's perfectly possible. However I'd need some hardware to test on to make sure the disks are readable. I'm actually hunting for an old Mac of some description but they're not cheap.

[andrewferguson]

If it's a help, the dc2dsk command line tool can convert from DiskCopy images to raw images. (It was written for use with the Floppy Emu floppy drive emulator, since although the Floppy Emu can read DiskCopy images, it cannot write to them - much like Mini vMac and SheepShaver).

The source for dc2dsk is 3-clause BSD (and is only 122 lines, 72 with the license and comments stripped), but I completely understand if you don't want to start including BSD code in the MIT-licensed FluxEngine. In that case, might I suggest a link to dc2dsk in the docs for reading Mac disks?

[davidgiven]

I've just merged a change which adds write support for Mac 800kB floppies --- it works fine given the extensive testing I did (I tried it a couple of times on a PowerBook 150). I can download a bootable DiskCopy System 6 floppy from archive.org and then boot from it, at least up until the point where it hangs because the 150 doesn't support System 6. It's also got read and write DiskCopy 4.2 file support.

Reading is still a little suspect, and I don't know why. Disks formatted on the 150 still have some bad sectors. However, disks written on the FluxEngine, written to by the 150, and then read back again seem to work absolutely fine. There's still an edge case to fix somewhere, I think.

[andrewferguson]

Wow! This is incredible! I've only played around with it for an hour or so, but it seems to be working seamlessly. I've written back some disks I'd made images of using FluxEngine, and they work great. I also downloaded a boot disk (System 3.2, it's what I had handy) and my Macintosh Plus booted perfectly off that.

This is going to be a real game changer for how I deal with getting software to/from my Macintosh Plus. I'm so unbelievably grateful to you for getting this working - thank you!!!

Now that this is working reliably, it would be interesting to see how dependent it is on the floppy drive used. When I have time, I'll try and go through my box of floppy drives, documenting which ones work and which ones have issues. (As a single data point, the drive I randomly grabbed from the pile is a Mitsumi D359T5).

[davidgiven]

Excellent --- I'm glad that works! (Also, booting of a CLV disk plays all sorts of interesting tunes on the floppy drive as it speeds up and slows down.) It was actually much easier to write the encoder than the decoder; I should have done it way sooner.

I'm still searching for a surface scanner for MacOS, BTW. FluxEngine doesn't do any kind of verification. It does seem capable of reading its own disks, but checking this on the Mac would be a good idea.

[davidgiven]

BTW, I wrote a tool for actually measuring how well a given drive copes with non-PC-format pulse intervals. This should reveal problematic drives.

http://cowlark.com/fluxengine/doc/driveresponse.html

[ladyada]

for analysis reference, here's a sony drive scp that verifies correctly

sonymac800.zip

[ladyada]

here's one with 3 revolutions and bad high sectors badsectors.zip