keirf / greaseweazle

Tools for accessing a floppy drive at the raw flux level
The Unlicense
984 stars 98 forks source link

Python code to decode IBM sectors #6

Closed hpingel closed 4 years ago

hpingel commented 5 years ago

Hi,

As a user of FluxEngine and AmigaArduinoFloppyDiskReaderWriter I stumbled upon Greaseweazle and find this a very interesting project. By coincidence, I have started working on Python scripts myself a couple of weeks ago to interface AmigaArduinoFloppyDiskReaderWriter via serial connection to decode disks with IBM-style sectors.

If you are interested, you can find my python code here: https://github.com/hpingel/pyaccess1581

The code contains an IBM sector parser in Python - I don't know if this is in the scope of your project. I also do CRC calcs and am not using crcmod any more but instead binascii.crc_hqx as I need binascii anyway and it doesn't need to be installed via pip.

The most interesting file for sector decoding should be: https://github.com/hpingel/pyAccess1581/blob/master/access1581/imager.py

Serial communication is done here: https://github.com/hpingel/pyAccess1581/blob/master/access1581/arduinointerface.py

Disk formats are described here: https://github.com/hpingel/pyAccess1581/blob/master/access1581/diskformats.py

Maybe this is interesting for you.

Cheers, Henning

keirf commented 5 years ago

Oh yes this is much better than my Python. This is the sort of place where I want Greaseweazle to get my Python skills to.

Unfortunately our licenses are currently incompatible. I'm PD (actually unlicense, which I may change to CC0).

keirf commented 5 years ago

May I ask what sort of things do you use binascii for?

Also, I only use crcmod for CCITT calculations, are you saying binascii can supplant that too?

hpingel commented 5 years ago

Thank you, but don't expect me to be a Python expert! I coded the whole thing in PHP first and after I realized that pyserial is that good I rewrote the whole code in Python and never regretted it.

From binascii I only use crc_hqx and unhexlify. (Example: https://kite.com/python/docs/binascii.unhexlify). I wanted to use more tools from binascii but sometimes it takes a while to match the own code to some library method.

I get the bitstream from disk and once I know the byte boundary I can allow to store the sector data as hex encoded string. Later when I want to write a disk image or I want to calculate the CRC I need the data as binary again - there I use unhexlify.

For documentation of my thoughts about CRC please have a look here: https://github.com/hpingel/pyAccess1581/blob/master/access1581/imager.py#L140

To be honest, as I learned that David Given will try to adapt Fluxengine software to the STM32 based floppy interface, I kind of thought my Python code will be obsolete before it reached a release state. At the moment I am relaxed about license stuff.

What is the scope of your project? Are you working on the same features as David in Fluxengine? Or are you more focussed on writing flux to disk to let it be processed by other tools? Maybe I didn't read the right parts of your documentation but it seems that your project is in a very dynamic development stage (which is impressive).

keirf commented 5 years ago

The scope is to include analysis and transcoding within the project. FluxEngine style (it appears). But using Python for rapid development. Both FluxEngine and the snappily-named AAFDRW seem to be using C++ which I hate anyway, and I don't see the need for in this instance.

I have done significant analysis and transcoding in the past. See my Disk-Utilities github project. It was originally Amiga focused and can do a lot of Amiga custom track formats/protections.

Volutar commented 5 years ago

This project definitely gonna need a folder of different image converting/capturing tools (obviously written in python). Though SCP is the most precise one, in 99% of cases it's overbloated, excessive and practically not needed. I believe it was chosen because it's pretty straight, supported by HxC tool, and can be converted to number of formats without much of an effort. Cheers for that. :) But depending on GUI application without chance to "advance" it is not that pretty.

I think the first "usable" format support via convertion (or perhaps "gw.py" updating) should be made with HFE. Just because it's the most "universal" for using with FlashFloppy. Though v1 version doesn't support for fuzzy bits, v3 is kinda supports that. TD0 format also would be nice.

Well, it's the whole new area of making analyzing, converting and perhaps recovering tools, for using with FlashFloppy or emulation software.

As for cpp vs python - in this pair I'm more of a cpp guy, but I aware that this project already uses python approach - it's initially multiplatform and these additonal tools can evolve pretty quickly. It's not something that needs for lots of processor powers and intense reatime calculatons, so the choice is pretty obvious.

By the way, I found this small snippet, that might help to remove the need of finding port and using extra parameter in the command line.

import serial.tools.list_ports

s=''
for n, (port, desc, hwid) in enumerate(serial.tools.list_ports.comports()):
    if "Greaseweazle" in desc: s=port

if s!='': print('Port: ',s)
else: print("Greaseweazle is not connected")
Volutar commented 4 years ago

I'm currently finishing MFM decoding function in py to use with converting-saving to different formats, including IBM images. It gathers all necessary info from raw track to properly process different protections too (including sectors within sectors) and CRC check. The first converter I'm going to make with it is HFE2UDI (udi format is kind of hfe, without logical sectors).

Also I'm working on raw SCP data processing, which is able to adjust to floating speed.

dpaschal commented 4 years ago

Kierf,

So I have a SuperCard Pro from Jim. I have your Disk Utilities installed on Linux and they work great for dumping my old Amiga disks. I want to dump my old C64 disks on Linux. I was wondering if any one had made a Python program to interface with the SCP device and use it to dump Amiga and C64 disks and that lead me to here. Any reason why the C64 disks have not been implemented yet? I didn't know if there was some technical hurdle.

keirf commented 4 years ago

I have no prior C64 knowledge. I may add raw C64 support (g64?) directly into GW. Perhaps with verification of unprotected (dos?) disks.

keirf commented 4 years ago

Not much more to say on this ticket I think.