Odd behaviour with multiple partitions

[Eraph]

Hello, firstly I want to thank you for putting this together, having a hard disk on an old ST is a dream come true!

I'd like to make use of multiple partitions on my SD card. I use Linux and would like to be able to transfer files between machines so a sector size of 4096 and partition size of 256MB works well here.

What I'm seeing is that when I set up more than one partition on the SD card I start to get errors when creating/deleting directories on any of the partitions.

Creating a directory gives me this error:

This disk does not have enough room for this operation.

Deleting a directory (dragging to Trash) sometimes gives me this error:

Invalid copy operation.

Closing the window for the partition and then opening it again shows that the directory is indeed created, but it will never let me delete it.

Using Quick Partition to set up multiple partitions gives me the same problem. My biggest worry here is the reliability of the partitions if I'm getting errors like these when performing what should be simple file operations.

Using the partition tool to remove all partitions but one restores normal behaviour.

I have a 1040 STE with 2MB RAM, TOS version 1.62 and GEM version 0.17.

Cheers!

[retro16]

Hi Phil, thanks for the feedback it is always appreciated.

This looks like filesystem corruption, it's really bad news. I had this kind of issue during driver development, I thought I removed all these errors before releasing the stable version but that thing is hard to program for (the TOS filesystem is weird and full of bugs) !

You have the exact same machine as mine, so it's not your machine.

First things first, could you try running a self-test from the ACSI2STM setup menu ? Can you give me a rough idea of the amount and size of the files you try to copy before it fails ?

Honestly I have little time to work on this project right now, so don't expect a quick fix in the coming weeks. In the meantime maybe you could another driver if you need reliable writing operations (ICD seems to work on partitions created by the partitioning tool).

JM

[Eraph]

Thank you very much for your quick response and your suggestion.

I have tried the self-test a number of times, it has always passed successfully.

File sizes aren't particularly big, it seems to especially struggle with creating and copying directories. I've ended up with a few corrupted directories (filenames all garbled, file sizes either 0 or up to 1.4GB, other times a system crash)

I will try the ICD driver and let you know how it goes.

Does the size of the SD card matter? I've tried 16GB and 64GB (overkill I know, but they're all I had handy)

[Eraph]

Thanks for the suggestions, I had other issues with the ICD driver and tried P Putnik's driver, which had its own issues.

So what I've done is taken a look at the compile options and adopted some of the slower options. After a flash of the STM32 and a fresh format of the SD card, the situation is improved a great deal, but eventually I run into some corruption. This is what it looks like when opening the partition on a PC:

You can see the filenames are almost intact; SKTOP IN.F was DESKTOP.INF, for example. But I'm pretty sure it wasn't 1.4GB ;)

I'm trying now to set all the compile options to the slowest I can get them. I think this is probably what will help, put it down to my dodgy soldering or unscientific wire clipping. I will keep you updated.

For the record, I have modified the following options:

#define ACSI_CS_FILTER 3
#define ACSI_FAST_DMA 0

[Eraph]

I met up with some local Atari enthusiasts today, they reckon it's the specific STE model I have. Apparently it's notorious for corrupting partitions.

I'd seen the warning about certain STE models on Jookie's page, attributing it to a dodgy DMA chip (C025913-38 in my case - the page says this is bad). One of the blokes I saw today reckons a lower power CPU will generate less signal interference, so he's going to help me out with that. I will let you know how it goes.

We did try the ACSI2STM with another ST, there were occasional funnies (TOS Error #06 or something like that) but nothing like corrupted partitions so I'm hopeful we can fix it on the ST itself.

[retro16]

Thanks for the very detailed feedback, it helps a lot. There are indeed suspicions of bad DMA chips, there is also the issue of buffered/unbuffered chips. I am far from being an ST expert, I only test on my 3 machines which seem to have the "good" chips.

I think you have 2 separate issues: TOS/GEM bugs and a DMA transfer issue.

• I also have filesystem corruption on my machine when copying too many folders or files from GEM, I believe this can happens even on legit hardware, I have no solution for that, maybe installing fixes like FOLDRXXX.PRG may help. This leads to the "Invalid copy operation" message at some point, but no file is corrupt except the last being copied.

• Your screenshot above looks like DMA issues. The way file names look clearly shows that the DMA data transfer is skipping bytes when writing to the STM32. I had similar issues with buggy code in earlier versions but I can't reproduce the issue anymore with 3.x code on my hardware. This may be an issue on the STM32 side, including a bad STM32 chip or yet another very subtle bug.

I just found out that the blue pill board has pull-ups on PA11 and PA12 (for USB operation), this is really bad because it happens to be the DMA synchronization pins. On strongly buffered chips this works usually well, but it may fail on weaker chips.

If you don't mind losing the USB port of your blue pill, could you remove R9 and R10 on the back ? You can either unsolder them, or if you don't mind more expeditive solutions you can just destroy them with small cutting pliers, then clean everything up with a cutter.

About compilation options:

• You can revert ACSI_CS_FILTER back to its default value, this setting is not involved in data transfers (only commands).
• You could try playing around a bit with ACSI_ACK_FILTER, even though this is only for reading the SD card, it may solve some issues.

[Eraph]

Hello again,

I've installed a low-power 68HC000 processor and it looks like the corruption problem has gone, so that's a great result!

With a single partition I don't seem to have any problems but with two or more partitions I get those "invalid copy operation" errors when copying folders onto any of the drives. Removing the two resistors as suggested does not fix this but hopefully helps to ensure overall stability. FOLDRXXX.EXE (renamed to FOLDR500.EXE) did not help. I have also reverted all the ACSI2STM options. Could there be a bug in the code when handling multiple partitions from the same SD card? I have my partitions laid out like this with a sector size of 4096 for Linux compatibility.

Partition	Format	Size (MB)
1	FAT16	250
2	FAT16	250
3	FAT16	250
4	FAT16	250

By the way, using your partition tool it is possible to choose a partition type of FAT12/FAT16, but it looks like this is totally separate from how a partition is formatted, for example I can format a type 2 partition to FAT16. What does type mean in this case?

[retro16]

Wow that is very valuable information you provide there. I never had a first hand confirmation that these DMA issues were real and impacting this project, you put so much effort to confirm this. Thank you so much !

About "Invalid copy operation" I still need more time to work on this. It may be a GEM/TOS bug but I still need to do some more tests in a controlled environment (comparing with EmuTOS on real hardware, comparing with other GEM/TOS drivers in Hatari, and comparing with TOS 2.06 which is much better quality).

I can already reply about partition type: my driver simply ignores partition type entirely. My driver just senses partition types based on their content, just like what the Linux driver does. Other operating systems/drivers do use the type field to determine the type of the partition (especially some Microsoft operating systems and some other Atari HD drivers) so it's better if it is set correctly.

[Eraph]

Hello again,

I've been mucking about with EmuTOS and it handles multiple partitions much better, so could well be a TOS bug but may give you an idea of where to look. Running EmuTOS has its own issues, I can't get it to boot from SD when running as a .PRG (it hangs on the loading screen but the STM32 shows a flashing activity light) so I've been running it from floppy for now.

Probably unrelated but may be useful for you to know, I tried writing the EmuTOS Bootloader image to an SD card, this boots straight into EmuTOS bypassing the ACSI2STM init screen but as soon as I add another partition it no longer boots automatically. Even removing the new partition to revert it to its original state doesn't help.

[retro16]

Thanks again for testing that. Now I need to make the self-testing tool better to try to reproduce a similar issue without using TOS, thanks to your testing I know exactly what to test. If my tests reproduce any kind of bug I may be able to fix this issue.

Thanks also for the EmuTOS bootloader feedback. It's indeed another issue related to the partitioning tool (which is far from perfect). it's worth investigating and it looks easy to reproduce on an emulator. It seems that the boot checksum is somehow damaged during partitioning.

I still don't have a lot of time to work on the project so there will be a few weeks of latency before I make some significant progress, so thank you in advance for your patience.

[Eraph]

Thank you for all your effort! I'd love to see this succeed, it's a great little project. I wish I had the hardware/assembly know-how to help contribute but my domain is more web application development unfortunately! Happy to help where I can even if it's just reporting on problems.

[retro16]

Okay I had more time than expected during lunch break and I could write a small "disk surface test" utility doing raw access with my driver (running with no TOS/GEMDOS dependency, 100% bare metal code). Results: the driver is 100% stable on my machine while scanning a whole 64MB partition. Then, based on my GEM/TOS bug suspicion and your encouraging EmuTOS results, I fixed my dual TOS STE. The test I did was the following:

• Copy my whole C: partition to another partition.
• The source partition contains about 80 directories and 9MB data.
• Copy in one big operation.
• The source and destination partitions are on different SD cards, but this doesn't change anything from a TOS standpoint based on how the acsi2stm driver works.
• Tested on a 30MB target partition (512 bytes sectors) and a 250MB target partition (4096 bytes sectors). All formated with the acsi2stm setup tool.

Results on my machine:

• On TOS 2.06: works perfectly for both 30MB and 250MB partitions. All files look identical to the original. Starting a few programs from the copy is okay.
• On TOS 1.62: First attempt on the 30MB partition ends up with TOS 6 error. Second attempt on the 250MB partition results in "Invalid copy operation" very quickly.

My final recomendation is: do big file operations on a PC and avoid TOS as much as possible. Saving/loading small documents from within ST applications works well (... at least it works as poorly as it did in 1992). Modifying the STE to support ROM switching between 1.62 and 2.06 is heavily recommended. There are disk-loaded TOS images floating around on the internet, some are okay-ish, others are full of bugs.

Will have a look at this EmuTOS bootloader issue later.

[Eraph]

Thanks so much for all your efforts so far! I have booted EmuTOS directly from floppy and it seems that I can use all partitions without any problems. That pretty well confirms we're dealing with issues in early implementations of TOS, unfortunately.

Using the floppy to boot into a workable environment is a decent compromise, I'd love it if you could get it to play nicely with booting from the hard drive. I will see about getting alternative EPROMs and making them switchable.

[retro16]

Had a quick look at this pre-made EmuTOS images, it doesn't look very standard to me:

Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x89ff0000

Device           Boot Start        End    Sectors   Size Id Type
sdcard-acsi.img1 *       64     499967     499904 244,1M  6 FAT16
sdcard-acsi.img3          0 1296515584 1296515585 618,2G  0 Empty
sdcard-acsi.img4          0          0          0     0B  0 Empty

Partition table entries are not in disk order.

I'm more focusing on the EmuTOS installer right now (INSTALL.PRG provided in the same zip file).

[Eraph]

I'm not gonna pretend I understand what's wrong with what you showed me there!

I neglected to consider the .PRG installer, I'll have to give it a go, could be the solution to all my problems.

[Eraph]

I recently saw this article which sheds some light on the corruption problem with early STEs, it proposes a solution that can be baked into EmuTOS, but I wonder if you can take any value from the analysis and bring it in to your drivers?

https://www.chzsoft.de/site/hardware/new-atari-ste-bad-dma-investigation/

[retro16]

Thanks a lot for bringing this very insightful document to my attention.

As you may have noticed, I drastically changed my approch for the future 4.0 release (still a lot of work unfortunately), but this document is still relevant anyway.

[retro16]

Version 4.00 was extensively tested against drive swaps. There is even a unit test for swapping SD cards in GemDrive mode. GemDrive mode is immune to TOS filesystem bugs due to the way it works. Closing the issue since I don't think it is relevant anymore. Feel free to reopen if you still have the problem or if I misinterpreted.