search

davidgiven / fluxengine

PSOC5 floppy disk imaging interface
MIT License
356 stars 69 forks source link

Reading ND (Norsk Data) format 17b floppies fails #46

Closed tingox closed 5 years ago

tingox commented 5 years ago

So I'm trying to read a ND aka Norsk Data format 17b floppy (the floppy itself is a normal 5.25 inch DS, HD 1.2MB floppy) using the fe-readibm command so that I can capture the flux information into a file. I'm using this command: $ ../.obj/fe-readibm --output nd-525-210523g02-xx-02d.img --write-flux=nd-525-210523g02-xx-02d.flux but it fails with an error 46.0: 162 ms in 102144 bytes (629 kB/s) 1.00 us clock; 20279 bytes encoded; 32 records terminate called after throwing an instance of 'std::out_of_range' what(): byte access out of range Aborted the floppy drive I am using is a Teac FD-55GFR floppy, and I have verified that it reads IBM formatted floppies ok. ND format 17b has 1024 bytes per sector, 8 sectors per track, 77 tracks and is DS/DD. When I read such a floppy on a FreeBSD machine, I set parameters with the fdcontrol command first, # fdcontrol -f 1232 /dev/fd1 view settings # fdcontrol -vF /dev/fd1 /dev/fd1: 1232 KB media type Format: 8,1024,0xff,0x35,77,500,2,0x74,1,0,+mfm Sector size: 1024 Sectors/track: 8 Heads/cylinder: 2 Cylinders/disk: 77 Transfer rate: 500 kbps Sector gap: 53 Format gap: 116 Interleave: 1 Side offset: 0 Flags <MFM>

then read with the dd command like so $ dd if=/dev/fd1 of=./test.image bs=1024 1232+0 records in 1232+0 records out 1261568 bytes transferred in 230.765575 secs (5467 bytes/sec)

Back to the fluxengine. Is this the correct way to capture the flux file of an unknown format?

davidgiven commented 5 years ago

Yup, just add --just-read and it'll skip all the analysis.

This is clearly a bug at my end; if you can send me the flux file I'll fix it. Email me at dg@cowlark.com (and of course I promise to delete the file one I'm done with it).

tingox commented 5 years ago

ok, with '--just-read' it doesn't fail: tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm --just-read --output nd-525-210523g02-xx-02d.img --write-flux=nd-525-210523g02-xx-02d.flux Reading from: :d=0:s=0-1:t=0-79 Writing a copy of the flux to nd-525-210523g02-xx-02d.flux ... skipped ... 79.1: 162 ms in 91264 bytes (561 kB/s) --just-read specified, halting now I'll send you the file.

davidgiven commented 5 years ago

There were a couple of nasty buffer overrun bugs, which I've fixed and pushed. Thanks.

...but the image you sent me won't decode. Thing is, I actually have a USB floppy drive which generates those 1232kB disks, and they read just fine. But with your image, I can see it correctly detecting the clock, there's a nice clean histogram, but the results are complete garbage. If I look at the raw data I can see fragments of mangled English text so it's almost getting the format right, but there's a bit error every few bytes.

You said you verified that it could read a IBM floppy --- was that a HD or DD floppy? I don't have any HD floppies to test with, and it's possible that I need to explicitly tell the drive it's HD or it won't read properly. 3.5" drives autodetect this so it's not a problem there.

tingox commented 5 years ago

IBM floppies - the 5.25 inch IBM floppies I have read (successfully) so far are physically HD, but the format has been 360k, like this $ ../.obj/fe-readibm --output 525-brief-2.1.img ... 40 tracks, 2 heads, 9 sectors, 512 bytes per sector, 360 kB total I have some 5.25 inch IBM format floppies that won't decode this way, but I haven't done more research on that yet.

tingox commented 5 years ago

I've tried to set the clock rate manually (based on a histogram) $ ../.obj/fe-readibm --just-read --output nd-525-210523g02-xx-02d_manual.img --write-flux=nd-525-210523g02-xx-02d_manual.flux --manual-clock-rate-us=3.75 I'll send you that image in case it helps.

davidgiven commented 5 years ago

Hmm. Maybe the drive setting only makes a difference for HD data, and the pulses are far enough apart on a 360kB drive that they're not necessary? What about the disks? Having a hub ring is usually a sign of a DD disk, and not having one is a HD disk. Mine are both (I only have two!) DD.

Also, the clock rate's only used during the decode phase and flux files don't use them. You can actually use a flux file as a source and try decoding with different clocks without wear and tear on the disk.

tingox commented 5 years ago

the disks have no hub ring; they also have the 3M brand label with the serial number, and the text

high density formatted DS, HD IBM AT 1.2 MB

and I picked them out of a sealed 10-pack of 3M floppies back when I wrote them, so they are genuine hmm - scratch that - what I wrote above is correct for the ND format floppies, not the IBM format floppies. I'll go back and check them.

the clock rate - good to know.

davidgiven commented 5 years ago

So this was slightly more complex than I thought --- it turns out I had gotten about half way through adding high density mode, except it defaulted to enabled...

Could you have a go with the hd branch? This should actually fix the support; there's now a --high-density (or -H) flag which sets the drive to high density mode when reading, and the default is double density as it should be. This should be set based on the media type, not the format. But I don't have any disks to test with!

You'll have to rebuild the firmware and flash, I'm afraid.

tingox commented 5 years ago

So, with firmware and client from the hd branch I can read and decode 5.25 inch HD floppies that are MS-DOS formatted, se my comment in #48 . However, ND formatted floppies always fail with "USB underrun". example:

tingo@kg-u35jc:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -o 525-CIS-format-f1232-number-3.img -f 525-CIS-format-f1232-number-3.flux
[..]
  4.0: 160 ms in 143232 bytes (891 kB/s)
       0.96 us clock; 20951 bytes encoded; 16 records
       8 sectors; 
       logical track 4.0; 8192 bytes decoded.
  4.1: Error: USB underrun (not enough bandwidth)

it doesn't matter is I use the '-H' switch

tingo@kg-u35jc:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -H -o 525-CIS-format-f1232-number-3_2.img -f 525-CIS-format-f1232-number-3_2.flux
[..]
  4.0: 160 ms in 143232 bytes (891 kB/s)
       0.96 us clock; 20951 bytes encoded; 16 records
       8 sectors; 
       logical track 4.0; 8192 bytes decoded.
  4.1: Error: USB underrun (not enough bandwidth)

I don't understand why this fails when MS-DOS format works? FWIW, this is what fe-rpm gives on this machine

tingo@kg-u35jc:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-rpm
Rotational period is 166 ms (361.446 rpm)

and fe-testbulktransport

tingo@kg-u35jc:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-testbulktransport
Transferred 1048576 bytes in 1096 (933 kB/s)

I've tried 15 different ND formatted floppies, none of them completed the reading, all failed with "USB underrun". The machine I ran this test on runs Xubuntu 18.04.2 LTS with kernel

tingo@kg-u35jc:~/personal/projects/psoc/fluxengine/tmp$ uname -a
Linux kg-u35jc 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

the reason for using that machine is that my preferred laptop at home, which runs Debian 9.8 and kernel

tingo@z30b:~$ uname -a
Linux z30b 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux

failed on reading anything (even DD formatted floppies) with "USB underrun". Not sure it is relevant.

tingox commented 5 years ago

FWIW, I also tested the 'hd' branch on my work machine, but it also has the "USB underrun" problem. Example:

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -o nd-525-210523g02-xx-02d.image -f nd-525-210523g02-xx-02d.flux
[..]
 42.1: 161 ms in 128256 bytes (795 kB/s)
       0.96 us clock; 21025 bytes encoded; 16 records
       8 sectors;
       logical track 42.1; 8192 bytes decoded.
 43.0: Error: USB underrun (not enough bandwidth)

Do you want any of the flux files? Are they of any use to you (even if they don't contain a full image)?

davidgiven commented 5 years ago

USB underrun means that you're out of USB bandwidth. Each pulse on the disk uses up two bytes.

One of my 1232kB 3.5" disks takes 150kB to scan, at 785kB/s bandwidth. That's a 300rpm disk. 5.25" disks spin at 360rpm, which is 1.2x the speed, so I'd expect a 1232kB 3.5" disk to use 942kB/s of bandwidth, which looks like it's beyond your limit. Your 360kB disks are working because they have fewer pulses on the disk, and therefore use less bandwidth.

I suspect I'm going to need to reduce the bandwidth use, somehow. I can think of a few tricks but I just hope the board has enough CPU to manage it. You could try using lsusb -t and trying to find a connection as close to a high-speed hub as possible --- try a USB2 port if you've got one.

Re the high-density flag: I'm very confused. Previously the drive was stuck in a single mode. Unless I've managed to completely fluff the flag implementation, I would expect your disks to work with one setting but not the other...

tingox commented 5 years ago

One of my 1232kB 3.5" disks takes 150kB to scan, at 785kB/s bandwidth. That's a 300rpm disk.

Yes, no worries about that.

5.25" disks spin at 360rpm, which is 1.2x the speed, so I'd expect a 1232kB 3.5" disk to use 942kB/s of >bandwidth, which looks like it's beyond your limit. Your 360kB disks are working because they have >fewer pulses on the disk, and therefore use less bandwidth.

Well, the point I was trying to make was that my machines are apparently fast enough to capture 1200 kB 5.25 inch (in other words, ibm or ms-dos formatted) floppies without USB underrun, while 1232 kB 5.25 inch (ND formatted) floppies fails - every single of them. It could be that the limit is somewhere between 1200 and 1232, but it still "smells suspicious" to me. Anyway, if you manage to reduce the required bandwidth that would be great, currently it is hard to use the FluxEngine for 5.25 inch HD floppies. I'm guessing buffering more data on the PSOc wouldn't help?

I've already checked with lsusb -t, for some reason the FluxEngine always attaches at full speed (12Mbps) no matter which port (usb 2.0 or usb 3.0) I connect it to. Looking at my notes, I haven't documented lsusb -t output, I'll do that tomorrow (my FluxEngine is currently in my office).

About the high-density flag; alas, I have no clues. I can only report what I observe.

davidgiven commented 5 years ago

Oh, I'm sorry --- I thought you had 360kB 5.25" disks. Yes, that's really weird and extremely suspicious.

Sadly the Cypress board is a 12Mb/s device, so it won't get USB 2 speeds. However, USB 2 ports appear to be better at maxing out the bandwidth of a USB 1 connection, so you may get slightly better performance that way. If it's mostly working but marginal then that might make the difference.

I suppose I'll start thinking about really quick data compression...

tingox commented 5 years ago

I've opened issue #51 to document usb related issues (mainly ports, connections and bulktransport test results).

davidgiven commented 5 years ago

If you look in the 'crunch' branch there's an extremely experimental version using a compressed datastream from the device to the PC. Bandwidth requirements are about 500kB/s, so should be comfortably within your margin. PTAL?

Incidentally, the -H flag has been renamed --hd (note the two dashes); there was a name conflict in one of the tools.

tingox commented 5 years ago

report on 'crunch' branch testing in #53.

tingox commented 5 years ago

ok, testing with latest firmware and client from master branch, since 'crunch' is no more. Trying to decode a 5.25 inch, HD, ND format 17b floppy (written from the same image as the one I sent you):

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -s :t=0-76 --retries=0 -o nd-525-211024-c01-xx-01d-hd.image -f nd-525-211024-c01-xx-01d-hd.flux --hd
Reading from: :d=0:s=0-1:t=0-76
Writing a copy of the flux to nd-525-211024-c01-xx-01d-hd.flux
  0.0: (56840 bytes crunched) 162 ms in 90944 bytes
       1.04 us clock; 19534 bytes encoded; 16 records
       0 sectors;
       0 bytes decoded.
  0.1: (54080 bytes crunched) 162 ms in 86528 bytes
       1.04 us clock; 19556 bytes encoded; 16 records
       0 sectors;
       0 bytes decoded.
[..]
 76.0: (55521 bytes crunched) 162 ms in 88832 bytes
       1.04 us clock; 19546 bytes encoded; 16 records
       0 sectors;
       0 bytes decoded.
 76.1: (53721 bytes crunched) 162 ms in 85952 bytes
       1.08 us clock; 18801 bytes encoded; 3 records
       0 sectors;
       0 bytes decoded.
H.SS Tracks --->
No sectors in output; skipping analysis
0 tracks, 0 heads, 0 sectors, 0 bytes per sector, 0 kB total

for some reason, it doesn't want to decode anything. If I try without the '--hd' flag:

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -s :t=0-76 --retries=0 -o nd-525-211024-c01-xx-01d.image -f nd-525-211024-c01-xx-01d.flux
[..]
 76.0: (78560 bytes crunched) 161 ms in 125696 bytes
       0.92 us clock; 21995 bytes encoded; 16 records
       8 sectors;
       Failed to read sector 0 (bad checksum);
       Failed to read sector 1 (bad checksum);
       Failed to read sector 2 (bad checksum);
       Failed to read sector 4 (bad checksum);
       Failed to read sector 5 (bad checksum);
       giving up
       logical track 76.0; 8192 bytes decoded.
 76.1: (78560 bytes crunched) 161 ms in 125696 bytes
       0.92 us clock; 22003 bytes encoded; 16 records
       8 sectors;
       logical track 76.1; 8192 bytes decoded.
H.SS Tracks --->
0. 0 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB.B.B..B.B..B.BBBB.B......BBBBB.BBBBBBBBBBB.B
0. 1 BBBBBBBBBBBBBBBBBBBBBBBB.BBB.BBB..B.B.......X.BBBB.B......BXBBB.BBBBBBBBB.B.B
0. 2 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB.BB.....BB.BBBB.B......BBBBB.BBBBBBBBBBB.B
0. 3 BBBBBBBBBBBBBBBBBBBBBBBBXBBB.BBB..BBB.....BBB.BBBB.B......BBBBB.BBBBBBBBBBBB.
0. 4 BBBBBBBBBBBBBBBBBXBBBBBBBBBBBBBBB.BBBB......X.BBBB.B......BBBBB.BBBBBBBBXBBBB
0. 5 BBBBBBBBBBBBBBBBBBBBBBBBBBBB.BBB..B..B.B...BX.BBBB.B......BBBBB.BBBBBBBBBBBBB
0. 6 BBBBBBBBBBBBBBBBBBBBBB.BBBBBBBBBBBB....B....B.BBBB.B......BBBBB.BBBBBBBBBBBB.
0. 7 BBBBBBBBBBBBBBBBBBBBBBBBBBBB.BB.BBB.B..B....B.BXBB.B......BBBBX.BBBBBBBBBBBB.
1. 0 BBBBBBBBBBBBBBBBBBBBBB.BB...................B.BBB.BB...BBBBBBBB.BB...........
1. 1 BBBBBBBBBBBB.BBB.BBBBBBBB...................B.BBB.BB...BBBBBBBB....BB........
1. 2 BBBBBBBBBBBBBBBBBBBBBB.BBBBBBB..............B.BBB.BB...BBBBBBBB.B..B.........
1. 3 BBBBXXBBBBBBBBBBBBBB.BBB.B...BB.............B.BBB.BB...BXBBBBBB.BB.B.........
1. 4 BBBXBXBBBBBBBBBBBBBBBB.BB.B...BX............B.BBB.BB...BBBBXBBB.B..BB.B......
1. 5 BBBBBBXBBBBBBBBBBBBBBBBB..BB.BB.............B.BBB.BX...BXXBBBBB.B.XB.........
1. 6 BBBXBXBBBBBBBBBBBBBBBB.BBBBBB...............B.BBB.BB....BBBBBBB.BBBB..B......
1. 7 BBXBBBBBBBBBBBBBBBBB.B..B..B................B.BBB.BB...BBBBXBBB....B..X......
Good sectors: 420/1232 (34%)
Missing sectors: 26/1232 (2%)
Bad sectors: 786/1232 (63%)
77 tracks, 2 heads, 8 sectors, 1024 bytes per sector, 1232 kB total

this looks better. Let me try to use ndfs-tool (from Tor Arntsen) to see if the image can be read:

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ~/bin/ndfs -t nd-525-211024-c01-xx-01d.image
Directory name            : FF&&.f.%+
                                     &.(
Object file index pointer : 134217791 SI: 0x2 (sub-indexed)
User file index pointer   : 134217791 SI: 0x2 (sub-indexed)
Bit file pointer          : 38 SI: 0x3 (reserved for future extension)
No. of unreserved pages   : 1073741824
Error reading page 4142923776: Success

not a total success, but something is readable here. The correct output from the 'ndfs' command looks like

tingo@kg-bsbox:/zs/tingo/doc/ND/software/211024c01$ ~/bin/ndfs -t 211024C01-XX-01D.image
Directory name            : 211024C01-XX-01D
Object file index pointer : 508 SI: 0x1 (indexed)
User file index pointer   : 510 SI: 0x1 (indexed)
Bit file pointer          : 306 SI: 0x0 (contiguous)
No. of unreserved pages   : 1
Files:
  0   0: I       89 pages    184320 bytes 1987-09-21 12:21:10 (FLOPPY-USER)S3-CONFIG-C00:PROG
  0   1: I       16 pages     32448 bytes 1987-05-18 08:39:11 (FLOPPY-USER)DDBTABLES-E07:VTM
  0   2: I       51 pages     86016 bytes 1987-09-21 12:21:27 (FLOPPY-USER)NOTS-SERVICE-C00:PROG
Directory size: 611 pages
Bit file size : 1 page

I hope this helps.

tingox commented 5 years ago

Perhaps something is marginal? Increasing revolutions seems to help. Example:

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -s :t=0-76 --retries=0 -o nd-525-210523g02-xx-02d.image -f nd-525-210523g02-xx-02d.flux
[..]
H.SS Tracks --->
0. 0 .B..BB...BB.................................BBBBBB.B....B....................
0. 1 BBB.B.BB.BBBB.......B.......................BBBBBB.B.B..BBB.B.B...B..........
0. 2 ..B....B.......B............................BBBBBB.B.BB.BBB..................
0. 3 B.BB..BB....................................BBBB.B.B.BB.BBB..B...............
0. 4 .BBB.BB...B...B.............................BBBBBB.B.BB.B...B................
0. 5 BB...BB.B....B..............................BBBBBB.B.BB.B....................
0. 6 BB...BX.....B...............................BBBBB..B.BB.B.B..................
0. 7 BBB.B...BB..................................BBBBBB.B..B.....B................
1. 0 BBBB.B..BB..................................B.B.BB...BB.B....................
1. 1 BBBBBBBBBBBBB..B............................B.B.BBB.BBB.BB...B...............
1. 2 BXBBBBBB.B.B.B..............................B.B.BBB.BBB.BBB..B...............
1. 3 BBBBBBBBBBB..B..............................X.B.BBB.BBB..BB..B...............
1. 4 BBBBBXBBBBBB.B..B.B.........................B.B.BBB.BBB.BB...................
1. 5 BBBBBBBB.BB..B....B.........................B.X.BBB.BB..BB...................
1. 6 B.BBBB..B.B.................................B.B.BBB..B...B...................
1. 7 BB.BB.B.B..B................................B.B.BBB.BBB...B..................
Good sectors: 937/1232 (76%)
Missing sectors: 5/1232 (0%)
Bad sectors: 290/1232 (23%)
77 tracks, 2 heads, 8 sectors, 1024 bytes per sector, 1232 kB total

with '--revolutions=7':

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../.obj/fe-readibm -s :t=0-76 --retries=0 --revolutions=7 -o nd-525-210523g02-xx-02d.image -f nd-525-210523g02-xx-02d.flux
[..]
Good sectors: 1023/1232 (83%)
Missing sectors: 0/1232 (0%)
Bad sectors: 209/1232 (16%)
77 tracks, 2 heads, 8 sectors, 1024 bytes per sector, 1232 kB total

Hope this helps.

davidgiven commented 5 years ago

That's really interesting. You're using the same drive that you used to write the disks, right? That means that drive alignment isn't an issue. The fact that more revolutions produces a better result suggests some kind of random noise.

The very clear stripe at track 44 is extremely suspicious, but only appears if you don't use --hd. I think this is related to the reduced write current needed past track 44 --- 8" drives needed this supplied by the disk controller, but it's automatic with 5.25" and 3.5" drives.

I have had a thought about whether I have the right pullup resistor at the FluxEngine end. Let me go and try something...

tingox commented 5 years ago

You're using the same drive that you used to write the disks, right?

Unfortunately, no. My media conversion machine (has 3.5 inch and 5.25 inch floppies and more) isn't easily accessible (for disassembly). However, I have used the drives in it to write a lot of floppies (both 3.5 and 5.25 inch) which I then has read in other drives, so I don't think the drives in my media conversion machine has alignment or other issues.

davidgiven commented 5 years ago

I spent some time fiddling with the firmware.

This is weird.

Have I missed anything?

My previous working hypothesis was that the drive pulses were too brief to be sampled... but it turns out the pulses are at least 150ns, and my sample rate is 12MHz which is 83ns per pulse, and I tried running them through a pulse conditioner first which samples at 64MHz, so that's not it.

One thing I have noticed is that if I push the clock rate on the FluxEngine up to 72MHz, the quality of the data gets slightly better, although that could be coincidence as I was testing. But... why would this only affect 5.25" disks?

I'm going to have to buy a logic analyser and find out exactly what's coming off the disk, which will take a bit of time, but in the mean time is there any chance you could format a 1200kB PC disk on your hardware and then try to read it with FluxEngine? Sorry to keep asking you to do stuff...

tingox commented 5 years ago

one thing you have perhaps missed: '--hd' flag doesn't work for me, but it does for you.

tingox commented 5 years ago

yesterday was a field trip to a museum storage facility. I brought my laptop, a 5.25 inch floppy drive and a FluxEngine. Luckily, I was allowed to read (sample) some of the ND floppies in their collection there. My intention was to read single-sided (ND format 0b) floppies, but most of the 5.25 inch floppies was in ND format 17b. So I tried reading some of those also. TL;DR - I now have around twenty images (and the corresponding flux files) which are "good", that is good enough that i can do a listing of them with the ndfs-tool, example:

tingo@kg-core1$ ndfs -t 10333f00-3_s.image
Directory name            : 10333F00-3
Object file index pointer : 150 SI: 0x1 (indexed)
User file index pointer   : 152 SI: 0x1 (indexed)
Bit file pointer          : 72 SI: 0x0 (contiguous)
No. of unreserved pages   : 1
Files:
  0   0: I        9 pages     17607 bytes 1984-11-28 11:57:47 (FLOPPY-USER)SWAPPER-F00:PSEG
  0   1: I      134 pages    274405 bytes 1984-11-28 11:58:23 (FLOPPY-USER)SWAPPER-F00:DSEG
Directory size: 149 pages
Bit file size : 1 page

if it helps, I can send you some (or all) of the flux files for the good images?

davidgiven commented 5 years ago

Sorry for the delay in getting back to you --- good to know it's being useful to you in its current state. Did you have any problems with the 0b floppies?

I don't think the 17b flux files will be of any use to me at this point; what's happening is that the sampling's going awry and the FluxEngine is reading bad data --- I think it's dropping bits. I still don't know why. My suspicion is that the 5.25" drives in HD mode aren't behaving electrically like other drives.

Accordingly, I've ordered a logic analyser, and now I'm waiting for it to arrive. That'll let me independently capture the output of the drive so that I can compare it with what the FluxEngine hardware is producing. It'll take a few weeks to arrive, unfortunately.

If it does turn out that the drive is behaving weirdly, there are a few things I can do; the PSoC5 has some ability to do software configurable analogue electronics so I may be able to hook up an opamp or something, but this is getting increasingly esoteric and I don't know much about this area, so I'm crossing my fingers.

I'll let you know when it arrives...

tingox commented 5 years ago

No, I didn't have any problems with the format 0b floppies (caveat: the only testing done on each image was listing the contents via the 'ndfs' tool).

Hmm, the "reading bad data" part is probably bad news for me as well - it probably means that there isn't any parameters or "tweaks" I can use on the flux files I captured to improve the resulting images?

Today I tested a format 17b image by writing it to a physical floppy (using my media machine, not the FluxEngine). Then I put the floppy in a real ND machine and attempted to load it. Unfortunately, it stopped after loading the floppy monitor (think of it as a boot loader) and wouldn't show any files on the floppy. So the image is not good enough to be working.

davidgiven commented 5 years ago

I haven't forgotten about this!

I've managed to confirm that this is a sampler bug. See #75.

davidgiven commented 5 years ago

I've managed to confirm that it's not a sampler bug, or at least not a current one. I think I actually fixed this a while ago with another change and never noticed. Certainly, I can read your 17b disk absolutely fine now.

Could you update your firmware and client and have another go and see if it works? Thanks!

tingox commented 5 years ago

I tested on a small sample set (7 floppies, ND format 17b) It is much better now:

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../fluxengine read ibm -s :t=0-76 --retries=0 -o test-nd-525-250305m05-xx-03d.image
[..]
H.SS Tracks --->
0. 0 .......?.......?..........?..................................................
0. 1 ...............X..........X..................................................
0. 2 ...............X..........X..................................................
0. 3 ............?..X..........X..................................................
0. 4 ...............X..........X..................................................
0. 5 ...............X..........X..................................................
0. 6 ...............X..........X..................................................
0. 7 ...............X..........X..................................................
1. 0 .........................X.................................?.................
1. 1 ...........................................................X.................
1. 2 ...........................................................X.................
1. 3 ...........................................................X.................
1. 4 ...........................................................X.................
1. 5 ...........................................................X.................
1. 6 ...........................................................X.................
1. 7 ...........................................................X.................
Good sectors: 1210/1232 (98%)
Missing sectors: 22/1232 (1%)
Bad sectors: 0/1232 (0%)
77 tracks, 2 heads, 8 sectors, 1024 bytes per sector, 1232 kB total

for most floppies I can get improvements by adding '--revolutions=2', like this

tingo@kg-bsbox:~/personal/projects/psoc/fluxengine/tmp$ ../fluxengine read ibm -s :t=0-76 --retries=0 --revolutions=2 -o test-nd-525-250305m05-xx-03d_r2.image
[..]
H.SS Tracks --->
0. 0 .............................................................................
0. 1 .............................................................................
0. 2 .............................................................................
0. 3 .............................................................................
0. 4 .............................................................................
0. 5 .............................................................................
0. 6 .............................................................................
0. 7 .............................................................................
1. 0 .............................................................................
1. 1 .............................................................................
1. 2 .............................................................................
1. 3 .............................................................................
1. 4 .............................................................................
1. 5 .............................................................................
1. 6 .............................................................................
1. 7 .............................................................................
Good sectors: 1232/1232 (100%)
Missing sectors: 0/1232 (0%)
Bad sectors: 0/1232 (0%)
77 tracks, 2 heads, 8 sectors, 1024 bytes per sector, 1232 kB total

as you can see, I'm not using the '--hd' flag. Out of the seven, only one image can't be read by the 'ndfs' tool (and that might possible be caused by a different filesystem format)

davidgiven commented 5 years ago

Oh, good --- yes, the --revolutions=2 is probably because on this format the sectors aren't lined up with the index hole and one sector overlaps it. You need two revolutions to be able to read a complete copy of it. The whole row of Xs is a little bit strange, however. The error rate is probably low enough that you can use automatic retries now.

I'm going to close the bug as fixed --- let me know if you have any problems...