Further Format issue on BBC 8271 with Disc Utilities

[kgl2001]

BBC Micro, 8271 controller, DFS1.20, FF 0.9.14a, index-suppression = no

Following on from ticket https://github.com/keirf/FlashFloppy/issues/57, I have been able to generate a format error when running a disc test with Disc Utilities. The format part of the test starts at track 80 and works down to track 1. It failed at track 19:

.hfe attached:

Test-A1.zip

[keirf]

Interesting data thank you!

[keirf]

I think what's needed here is a fakier index signal to keep the Beeb happy. With the current behaviour it's really hard to reason about whether index pulses will be sufficiently regular to avoid timeouts.

[keirf]

Here is the plan.

BBC (and more broadly other Acorn systems such as A5000, based on 82C710) uses INDEX in the following ways:

1. Trigger sector-not-found during sector id lookup -> must see every sector id between each pair of INDEX pulses.
2. Start and end of track format -> INDEX pulse expected to align with virtual 'index hole'.
3. Regular pulses indicate disk is present and spinning. Absence indicates "drive empty". -> Must see INDEX pulses no more spread than 200ms + delta (delta not known, probably reasonably generous).

Most systems make use of (1) and (2), and can be conveniently dealt with by driving INDEX only when: (a) We are at the virtual index hole; and (b) RDATA or WGATE active

This however breaks use (3) on Acorn systems. The three uses are in fact mutually antagonistic on an emulated floppy system with unpredictable write-to-flash latencies, and I haven't thought of a way to perfectly square the circle.

Proposal is therefore: I. Free-run INDEX timer at 200ms+epsilon (epsilon = 1-2ms) II. When WGATE deasserts, disable INDEX timer. III. When RDATA restarts post-writeback, reset the INDEX timer and trigger immediate INDEX pulse.

This deals with Acorn uses 1-3 as follows:

1. Between each pair of INDEX pulses, in the absence of seeks, every sector id will be seen. Even if pulse occurs in the middle of a sector id (aka IDAM) the +epsilon means it will be fully seen before next INDEX pulse.
2. Doesn't deal well here. Write may be too long, hence need overwrite-avoidance in HFE handler. Write may not be aligned to track start. Could fix that up for track-long writes to HFE? Or live with it for now and see if it matters. None of this affects sector image formats (ADF, SSD, DSD, etc)
3. Wins here: except after WGATE INDEX pulses are guaranteed every 200ms+epsilon. WGATE deassert stretches this by worst-case writeback-to-flash time. We hope with a decent stick to keep this below say 100ms quite conservatively. And therefore hope that delta in (3) is >= 100ms.

Caveats:

• How much does alignment of track-format writes matter?
• Does it matter if INDEX pulses occur during a sector (during sector id? during sector data? between sector id and sector data?). Hypothesise (hope) that INDEX only causes timeout during sector-id search.

Alternatives: Replace II and III with: II.b Immediate INDEX pulse and reset timer on any deactivation of RDATA (SIDE change, STEP assert, WGATE deassert). Needs to be quick enough (especially on SIDE change) that sector-id-search hasn't started at time of immediate INDEX pulse.

[kgl2001]

Sounds very complicated! Very happy to continue testing, but I'm still waiting for my logic analyser to arrive in the event that you need any specific timing details. Is there any further testing / logging I can do with the current firmware?

[keirf]

More complicated than I'd like, shouldn't be that hard to implement and then we can see if it actually works, for this issue and #69.

No need to do any further testing until I have a new firmware for you.

[keirf]

Okay here is a firmware with new implementation of index-suppression=no for you to try. Normal non-debug build only for now. This is from Github branch fake-index:

ff_67_3.zip

[PeterBBCUser]

FF_67_3 with index-suppression=no in the FF.cfg The Test failed on the verify, I then reformatted using the format program on the welcome disc, it came up with Trace 09?. When I tried to verify the disc I get disk fault 08 at 00/00 also if i try to catalog the disk. which is :- Error 08 (clock error) During a disc read operation a clock bit was missing. Peter.

[keirf]

Ugh, well here's another to try. Perhaps try some read commands to start with on a known good HFE, then try the format+verify test. If this fails then I have some more ideas to try...

ff_67_4.zip

[PeterBBCUser]

Well reading did work on another image as I used 2 different ones to load the Acorn format and the disc test program its only the disc i had formatted/used for testing that came up with the error 08. But will give the new one a try.

[PeterBBCUser]

okay it got through the format and verify plus the wrote 1st test pattern but then failed on the reading of it, but also tried a verify of a untouched disk, a couple actually and they come up with the odd verify error, it's certainly better than the 67_3 version which btw made a right mess of the disc when I looked at it with HxCFloppyEmulator.

[kgl2001]

Starting with ff_67_3 on DFS0.98, I'm able to read without any issue. Using the Disk Utilities, the format runs without any problem, but then fails the verify. The attached failed at track 63 (counting downwards from track 80). Looking at the hfe image, I'm not sure why it didn't fail before that!?

Test1-ff_67_3.zip

[keirf]

@kgl2001 the red sectors in the HxC track analyser are actually okay but flagged because they straddle the index pulse. This would definitely be expected in 67_3 and 67_4.

[PeterBBCUser]

If you look at the disc layout everything is allover the place.

[keirf]

@PeterBBCUser On format it would be.

Since neither 67_3 nor 67_4 works anyway, perhaps it is time to shelve this idea.

[kgl2001]

ff_67_4 reads and formats ok, but fails more catastrophically during verify.

Test1-ff_67_4.zip

On inspection, for me ff_67_3 is definitely the better of the two.

[keirf]

Okay thanks for today's testing. Not a success ;)

Tomorrow I will try a different strategy: keep index pulses in correct places during read/write, then insert fake ones at safe places when needed and when safe to do so (during seek, and immediately after a write). And we'll see how that goes.

[PeterBBCUser]

O kgl2001 my experience was exactly opposite of yours that ff_67_4 was better, I .E. i got further with the test and with a normal format/verify.

[keirf]

Ok, here's a new approach. It reverts to v0.9.15a behaviour, except that:

1. INDEX pulses are disabled post-write, while flushing to usb stick. This avoids read/verify timeouts.
2. An opportunistic INDEX pulse is inserted immediately that the host write ends (WGATE deasserted). This avoids drive-not-ready timeouts during long flush to usb stick.
3. Post-flush, read head is repositioned near end of track so that a true INDEX pulse is quickly triggered. This also avoids drive-not-ready timeouts.

Please test!

ff_67_5.zip

[keirf]

And here's another which skips (3) above. Does not reposition read head. Might be good enough, and will make multi-sector writes faster if so. Worth testing this one too :)

ff_67_6.zip

[PeterBBCUser]

67_5 formatting a disk verifying (using acorn format verify program on welcome disc) worked fine i saved files till the disc was full did a compress all worked fine, disc looks good even after that (attached) when I tried to run the diagnostic it formatted okay but failed on the 1st track (Trk 79) of the verify. If I run a separate verify on the disk formatted by the diagnostic it verify fine so no idea why the diagnostic fails. The image is after writing data till the disc was full. test_ff_67_5_ssd.zip

67_6 can read disc's sometimes comes up with an error on trk 0 when trying to verify a already formatted disc. I tried to format a disk it failed on the 1st track of the format image looks okay does not seem it even over wrote the previous data on it from the last test. test_ff_67_6_ssd.zip

In my opinion 67_5 works well, as I can format verify and use a disc without a problem as long as I don't try to do any crazy things like run any diagnostics because that seems to fail every time. Peter.

[keirf]

@PeterBBCUser I'm surprised 67_6 is so much worse than 67_5. Would you say that 67_5 seems to be better than v0.9.15a (most recent official release)? If so, please test 67_5 more particularly with small writes. Check how fast/slow it seems compared with v0.9.15a.

Additionally, here is another update which also inserts spurious index pulse on every head step operation. That's another safe place to insert them, and might help with that verify-after-format on the Diagnostics:

ff_67_7.zip

[keirf]

@kgl2001, when you have a chance would be useful to get your test feedback on ff_67_5, ff_67_6, ff_67_7. Thanks! :)

[PeterBBCUser]

67_7 does indeed seem to fix the problems with running diagnostics, both the Watford and other disc test run through 100% with no error on this version. Speed tests Verify disc:- 67_5 = 32.42 sec 67_6 = 32.29 sec 67_7 = 32.20 sec

Disc doctor Format/verify 67_5 = 43,67 sec 67_6 = 79 sec (lots of format errors but no verify errors) 67_7 = 43,54 sec. Peter.

[keirf]

@PeterBBCUser Thank you very useful. Is there a multi-sector write test you could time? Like the "write tracks 0..80" step in Watford Diag? That's the thing that really risks being slow in 67_5 and 67_7.

What I'm really interested in is speed of 67_7 (most reliable firmware for you) relative to the v0.9.15a baseline (most recent official release).

[PeterBBCUser]

the other diagnostic I have is in basic so i can add a time at start and end and see how long the whole test takes, or each segment for the two versions.... back in a bit with results. Peter.

[PeterBBCUser]

okay some speed results : - 67_7 Format 36,95sec, verify 46,53sec random access write 45,38sec random access read 51,79sec Pattern B has same read write speeds results.

9.15a Format and verify failed so no times for those. random access write 181,62sec random access read 351,74sec about the same for Pattern B

As you can see 67_7 is a lot faster.

[keirf]

Wow, 9.15a was very slow then. I'm also interested in sequential write performance, can you trigger a multi-sector write, either sector-by-sector or as an explicit multi-sector command to the 8271 controller? I guess don't put in too much effort as 9.15a clearly has horrible perf for you anyway!

The main thing is to get wider testing feedback from @kgl2001 and from @drdpj (on issue #69)

[kgl2001]

Yes. I'll be able to do some testing and provide feedback shortly. I need to get home from work first...

[kgl2001]

ff_67_5 with Disk Utilities - I get formatting errors (not getting as far as verify stage). With the attached .hfe I got a formatting error at track 24 (stepping down from 80). I can't actually see anything wrong, though.

Test1-ff_67_5.zip

More tests to follow.

Edit 1: Added a short time delay stepping between tracks during track format commands (which also delays stepping between tracks during track verify commands) and had two successful disk formats / verifies. Now getting occassional error during write - track 39 of Test2. Note that I've not introduced any time delay into the track stepping during these writes:

Test2-ff_67_5.zip

[kgl2001]

ff_67_6 with Disk Utilities - I still get formatting errors (not getting as far as verify stage). With the attached .hfe I got a formatting error at track 20 (stepping down from 80). As with ff_67_5, I can't actually see anything wrong, though.

Test1-ff_67_6.zip

Will move onto 67_7 now...

[kgl2001]

Hmmm. ff_67_7 with Disk Utilities - I still get formatting errors :( Tried with Watford Utilities and getting similar results. These last 3 tests were all with DFS 0.98 and index-suppression=no

Edit: Moved over to DFS1.20. Same problem with formatting error at track 33.

Test1-ff_67_7.zip

[PeterBBCUser]

doing 5x random writes/read with random length sector 100x ff_67_7 took 332sec (Avg for the 5 tests) 9.15a took 446sec (Avg for the 5 tests) I have only been using DFS 1.21 for all the tests.

[drdpj]

DFS0.98 is buggy :( But it's odd that it's not behaving :-/

On 13 March 2018 at 20:50, Peter notifications@github.com wrote:

doing 5x random writes/read with random length sector 100x ff_67_7 took 332sec 9.15a took 446sec I have only been using DFS 1.20 for all the tests.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/keirf/FlashFloppy/issues/67#issuecomment-372813737, or mute the thread https://github.com/notifications/unsubscribe-auth/ACdNPB9vXCcmTq6fc73a03rn5TauWuEcks5teDEQgaJpZM4SdsLW .

[kgl2001]

Ah! Let me try 1.21...

Edit: Nope. With DFS1.21 and ff_67_7 I'm still getting verify errors with Disk Utilities. I've also tried Disc Doctor and I'm getting occasional formatting errors with that too:

Test2-ff_67_7.zip

After a failed format, this one gives me a Disk Fault 0C at 00/01 when I try to read the catalogue

[PeterBBCUser]

Odd that its the 1st version that I can test everything without running into any problems. Here is the write multi-sector test it does 5 passes each with 100 write/reads and gives a Avg time it took to do the tests. This also works fine on my system.

10REM ------------------------------ 20REM DEMONSTRATION 30REM ------------------------------ 40REM OSWORD write data multi-sector 50REM ------------------------------ 60DIM tm(5) 70mc%=&900:wdata=&4000:rdata=&5000 80osword=&FFF1 90FOR pass% = 0 TO 2 STEP2 100P%=mc% 110[OPT pass% 120\ write parameter block 130.wparm EQUB 0 \drive 0 140EQUD wdata \write buffer 150EQUB 3 \no. of parms 160EQUB &4B \command 170.trk1 EQUB 79 \log track 180EQUB 0 \log sector 190.nsec EQUB 1 \1 sector 200.wres EQUB 0 \result 210\ read parameter block 220.rparm EQUB 0 \drive 0 230EQUD rdata \read buffer 240EQUB 3 \no. of parms 250EQUB &53 \command 260.trk2 EQUB 79 \log track 270EQUB 0 \log sector 280.nsec1 EQUB 1 \1 sector 290.rres EQUB 0 \result 300.raddr EQUW rparm \addr of rparm 310.waddr EQUW wparm \addr of wparm 320 330.start 340LDX#10:LDY#0 350LDA#0:STA&70:LDA#&40:STA&71 360.loop 370LDA&FE68 380EOR &FE69 390STA(&70),Y 400INY 410BNE loop 420INC&71 430DEX 440BNE loop 450LDA #&7F \function 460LDX waddr \point X & Y 470LDY waddr+1 \at wparm 480JSR osword \write a sect 490LDA wres \get result 500AND #&1E \extract error bits 510BNE end \if bad 520.read LDA #&7F \function 530LDX raddr \point X & Y 540LDY raddr+1 \at rparm 550JSR osword \read back 560.end RTS 570] 580NEXT pass% 590CLS:TIME=0 600PRINT"Randon track and sector length" 610PRINT"write and read test with random data" 620FOR tm% = 1 TO 5 630FOR Z%=1 TO 100 640T%=RND(70)+9:?trk1=T%:?trk2=T% 650X%=RND(10)+&20:?nsec=X%:?nsec1=X% 660PRINTTAB(0,5)"Trk "T%;" #Sec ";X%-&20;" " 670CALL start 680NEXT 690tm(tm%)=TIME/100 700PRINTTAB(0,6+tm%)tm%;" Time ";tm(tm%);" Sec" 710NEXT 720PRINT''"Avg Time =";(tm(1)+tm(2)+tm(3)+tm(4)+tm(5))/5 730PRINT"write result =&";?wres;" read result = &";?rres 740END Here is the disc image I have been using the above test program is on there called test1, its save to run it on this disc as it only writes from track 10 to 79. RND Write test_ssd.zip

[keirf]

@kgl2001 Test2-ff-67_7.hfe has those 12us+ bitcells again. I wonder if there is something up with your test rig. I don't know why the HxC analyser is marking all sectors as green, would need to investigate. It does however show many out-of-spec bitcells on the y axis, and on a quick initial test my own disk analyser finds many invalid sectors.

EDIT: But Test1-ff-67_7.hfe is clean....

What USB stick are you using? Have you tried any others?

[kgl2001]

I'll check if I have another BBC with an 8271 controller, and test this with a different Gotek. I'll do that tomorrow evening (actually, this coming evening - as it's past midnight!) and report back.

Edit: I'm using the same Sandisk stick that I've been using for all tests. I'll try a different stick tomorrow.

[kgl2001]

Logic Analyser has arrived, so (assume I can work out how to use it!) I can start getting some logic traces too.

[drdpj]

Ken - use a floppy lead with two sockets. How many channels have you got?

[keirf]

@drdpj - Is BBC/8271 a test you can do? Another data point on 67_7 would be helpful, as it will point the finger further at something about @kgl2001's setup (or not, as the case may be).

@kgl2001 since we have a successful run from @PeterBBCUser the obvious thing to do now is drill into what is different in your setup.

We must be pretty close to putting this to bed. :)

[drdpj]

I can, I'll have to shift it across from the A3010 and wipe some eproms. Are we concentrating on DFS 1.2, not 0.98?

(the good reasons for keeping with 0.98 have all been cracked long ago...).

d.

On 14 March 2018 at 08:28, Keir Fraser notifications@github.com wrote:

@drdpj https://github.com/drdpj - Is BBC/8271 a test you can do? Another data point on 67_7 would be helpful, as it will point the finger further at something about @kgl2001 https://github.com/kgl2001's setup (or not, as the case may be).

@kgl2001 https://github.com/kgl2001 since we have a successful run from @PeterBBCUser https://github.com/peterbbcuser the obvious thing to do now is drill into what is different in your setup.

We must be pretty close to putting this to bed. :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/keirf/FlashFloppy/issues/67#issuecomment-372941212, or mute the thread https://github.com/notifications/unsubscribe-auth/ACdNPGW9jWzXZZLy1DIVbqbQdzGQIJgMks5teNSogaJpZM4SdsLW .

[keirf]

@drdpj The main interest is in DFS 1.20 I believe. Thanks! @PeterBBCUser Were your tests on DFS 1.20?

[kgl2001]

@drdpj - It's a cheap 8 channel unit from ebay China. The reason I still use 0.98 sometimes is because some programs I have don't work with DNFS, but let's not get bogged down with that here. Happy to test with DFS1.20, which is part of DNFS. @keirf - I'll get set up this evening for more tests with different hardware (BBC, Gotek & USB stick). I believe @PeterBBCUser has been using DFS1.21 for all his tests.

[keirf]

Oh I would much rather we were testing 1.20 than 1.21, as the canonical official final released version, no hacks.

By the way those cheap analysers are good because they are Saleae clones and the Saleae software is pretty nice to use.

[PeterBBCUser]

My tests where with DFS 1.21 but will give it a test with 1.20 if you prefer to use that one. With DFS 1.20 and ff_67_7 on my Issue 7 BBC model B Keyboard switches 3&4 on (fast step) My random sector length write/read test 5 passes x 100 random seeks Average time 335Sec ----------- Passed. Disc diagnostic test------------- Passed. Watford disc diagnostics ------- Passed. Disc doctor format & verify ---- Passed. Acorn format & Verify ----------Passed. ADT format & verify ------------Passed.

[PeterBBCUser]

Don't know what other hardware details you want that are relevant, Also have a modern ROM/RAM board (from ifelsales) SCSI Hard drive SD card (no ROM on while testing) MMC drive with active rom (own modded DFS ROM fitted). Apart from that its a standard BBC.

[PeterBBCUser]

mmm just thinking maybe the gotek is different, not sure how I can identify the one I have. and not using a very fast USB stick at all, in fact its quite a slow one.

[PeterBBCUser]

This is the gotek I am using, very recently purchased.

[keirf]

All Goteks behave the same. Sometime 'slow' USB sticks are actually fast in this use case, which is very small I/O requests. But I don't think @kgl2001 has a USB stick problem, though can't rule it out until he tries another.

[kgl2001]

Just getting prepared for testing. Found another beeb with 8271. Need to get DFS1.20 into it, but that shouldn't take too much work (it's got SWRAM). Got another USB stick and other gotek too, so give me an hour or so, and I should be able to feed back some results.