List directory content and copy files to and from images

[henrikstengaard]

My hst amiga library used for accessing Rigid Disk Blocks and formatting Fast File System and PFS3 partitions has recently been expanded with support for reading and writing files to and from both Fast File System and PFS3.

The idea is to add basic file system commands like dir, copy, rename, delete and move. Copy will support source set to a physical drive or image file and destination to a folder like d:\copy-to-amiga and vice-versa.

Example of listing partitions in Rigid Disk Block:

hst.imager.exe fs dir D:\test.vhd rdb

[20:32:59 INF] Hst Imager v0.1.0 (12/08/2022 15:31:48)
[20:32:59 INF] Henrik Noerfjand Stengaard
[20:32:59 INF] [CMD] fs dir D:\test.vhd rdb

Entries:

Name |  Size | Date | Attributes
-----|-------|------|-----------
DH0  | <DIR> |      |
DH1  | <DIR> |      |
DH2  | <DIR> |      |

3 directories, 0 files
[20:33:00 INF] Done

Example of listing Super Frog WHDLoad contents in Rigid Disk Block DH1 partition:

hst.imager.exe fs dir D:\test.vhd rdb/dh1/whdload/games/s/superfrog

[16:33:20 INF] Hst Imager v0.1.0 (12/08/2022 15:31:48)
[16:33:20 INF] Henrik Noerfjand Stengaard
[16:33:20 INF] [CMD] fs dir D:\test.vhd rdb/dh1/whdload/games/s/superfrog

Entries:

Name                 |      Size | Date                | Attributes
---------------------|-----------|---------------------|-----------
Disk.1               |    960 KB | 2006-09-08 06:47:30 | ----RWED
Disk.2               |    960 KB | 2006-09-08 06:48:00 | ----RWED
Disk.3               |    960 KB | 2006-09-08 06:48:30 | ----RWED
igame.iff            |   30.3 KB | 2021-09-29 09:30:28 | ----RWED
Manual               |   13.7 KB | 2001-05-03 09:22:50 | ----RWED
Manual.info          |      2 KB | 2021-07-29 04:08:26 | H---RWED
ReadMe               |      7 KB | 2021-07-29 04:07:18 | ----RWED
ReadMe.info          |      2 KB | 2021-07-29 04:08:26 | H---RWED
Superfrog.highs      | 160 bytes | 2001-05-03 09:51:30 | ----RWED
Superfrog.info       |   18.3 KB | 2021-07-29 04:08:30 | ----RW-D
Superfrog.slave      |    6.6 KB | 2021-07-29 04:07:20 | ----RWED
SuperfrogIntro.info  |   19.6 KB | 2021-07-29 04:08:36 | ----RW-D
SuperfrogIntro.slave |    3.4 KB | 2021-07-29 04:07:28 | ----RWED

0 directories, 13 files
[16:33:21 INF] Done

[roc-eab]

Hi Henrik,

This set of commands is going to be quite useful.

For instance, currently after the image is created I invoke a WinUAE session where an ADF is activated with a AOS script to copy the file into the image. Before Hst-Imager it used also to format in PFS3 as well, now not needed any longer.

When this is implemented, I can happily retire the below WinUAE part altogether

Write-Host "Copying the files in WinUAE" Measure-Command { Start-Process -NoNewWindow -FilePath $Path_WinUAE_Exe -ArgumentList "-f $Path_WinUAE_Config_WB130","-G" -Wait } | Out-Default

[henrikstengaard]

I'm making progress and have parts of it working. I can copy between directories, images (.img/.hdf/.vhd) and physical drives. I also added a extraction of .lha, .adf and .iso, which can extract to directory, image (.img/.hdf/.vhd) and physical drive.

It's currently in a testing phase stress testing FFS and PFS3 file systems fixing error occuring when reading or writing files.

For first version of this I'm limiting this functionality to following commands:

• fs dir list directory in images/medias (.img/.hdf/.vhd,.lha,.adf,.iso) and physical drives.
• fs copy copy files and directories from directories, images (.img/.hdf/.vhd) and physical drives to directories, images (.img/.hdf/.vhd) and physical drives.
• fs extract extract files and directories from medias (.lha,.adf,.iso) to directories, images (.img/.hdf/.vhd) and physical drives.
• adf create create blank adf file and optionally format it.

I'm not sure if it's worth making a prerelease as there are too many issues that needs fixing first with these new commands.

[henrikstengaard]

Most of the critical errors are fixed now and I have run a large stress test copying ~5.7GB whdload demos, games and magazines, which successfully copied all ~277000 files from a .hdf image file to a .vhd image file.

It took almost 12 hours and I noticed it was fast in the beginning and then over time became more slow (see attached screenshot of end result). So there is room for improving what ever is making it slower maybe with a profiler.

I also ran pfsdoctor experimental from pfs3aio v3.1 final forum thread at EAB and it reported 54 errors like:

• ERROR:Anode #lx is double allocated
• ERROR:Unknown extrafield flags found (ignored)
• ERROR:Dangling rollover
• ERROR:Corrupt directory entry

This is most likely related to how dirblocks are handled and how dir entries within dir blocks are managed when writing a lot of files to one or more directories. I have already done some fixes for this, so I have a good idea about what goes wrong. Fixing the errors requires creating a preferably small test case to reproduce the errors, so I can use those to examine and fix the errors.

For smaller tasks like extracting superfrog whdload .lha to a image file works fine and doesn't result in any errors when running pfsdoctor, so even with errors still present in pfs3 read and write functionality it feels like a good time to make a new prerelease.

[henrikstengaard]

I started fixing the errors discovered and solved many of them, but still a few errors remain reported by pfsdoctor experimental:

• 2 corrupt dir entries.
• 1 reserved block nut used but allocated.
• 17 anodes not used but allocated.

Even with the errors, the image works and reads fine in WinUAE, so they are not super critical ones but still errors and therefore writing to PFS3 is still in an experimental phase. Reading seems to work fine.

I also did some fixes to caching used by PFS3 to ensure the cache gets flushed properly over time copying many files. The caching used linked list to store cached blocks read from disk. These have been replaced with dictionaries making cached block look up go from potentially have to go through each cached block in the lists to find out if the cache contains it to now have a directly look up using cached block no. or seq no.

This resulted in huge speed improvements and the same test as I did before went from ~12 hours to ~3 hours, so roughly 4 times faster for copying 5.7GB.

[henrikstengaard]

Lot of bugfixes and testing done over the last few months and it's now very stable. I haven't encountered any errors for the last couple of weeks and decided now is the time to release it.

v1.0.335 is available for download now and it contains with following filesystem commands:

• fs dir: list directory content in archives, disk images or physical drives.
• fs copy: copy between local directories and disk images or physical drives.
• fs extract: extract archives (.zip/.lha/.lzx/.z) to local directories, disk images or physical drives.

The remaining commands filesystem rename, delete, and move will be part of a future release.

Screenshots showing 5300+ whdloads being extracted from .lha and .lzx to a .vhd disk image, which took approx. 50 minutes for 6.4gb of files.

Screenshot showing same 6.4gb files and directories copied from one disk image to another, which took 1 hour and 6 minutes

I'm closing this issue now.