Hi-MD support?

[benguild]

I'm curious about Hi-MD support, and if it's potentially to be supported at some point ever or if there are currently obstacles that make this unrealistic for the project's lifespan.

Hi-MD provides filesystem access, but according to linux-minidisc it seems like there's a special SCSI interface for some other features: https://wiki.physik.fu-berlin.de/linux-minidisc/himddiskformat

I'm mainly interested in organizing music into groups, but also converting some tracks to ATRAC3Plus would be nice. Doing this manually or in a Windows VM isn't ideal but it works! Would prefer something more modern if it's potentially roadmappable, but as mentioned, I'm curious about blockers.

[asivery]

@benguild I am currently working on adding HiMD support. It might not be possible to add AT3+ encoding (I am still not sure), but metadata editing works already :)

[GrayzerGray]

Seeet. Thank you for your effortsSent from my iPhoneOn 25 Dec 2022, at 14:57, asivery @.***> wrote: @benguild I am currently working on adding HiMD support. It might not be possible to add AT3+ encoding (I am still not sure), but metadata editing works already :)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[benguild]

@benguild I am currently working on adding HiMD support. It might not be possible to add AT3+ encoding (I am still not sure), but metadata editing works already :)

Great! Can't wait to try it. — AT3+ support would be great, but I'm not sure if it's supported by any tools.

Currently I use a Windows VM for managing things, and it'd be nice to have something more modern.

[asivery]

@benguild I got all HiMD functions working - AT3+ / AT3 / MP3 upload included. I will release the new version in the next few days.

[benguild]

oh hell yes. can't wait to try it out! I need to update some tracks on a disc so it's perfect timing.

[Invictaz]

This is mega cool. I hope these patches get ported to Platinum-MD

[Invictaz]

@benguild I asked for it in 2019. Wow. It only took four years

[asivery]

@Invictaz It wasn't as simple back then. Getting it working in Chrome requires temporarily opening the normal MD mode, then exploiting the firmware and switching back. It's impossible without that, because WebUSB doesn't let you connect to mass storage device. To get around that, I load up an exploit and patch the USB descriptor, so that the OS misidentifies the HiMD device.

[benguild]

exploiting the firmware and switching back

is that going to work on Mac? 👀 are you exploiting the Hi-MD device, or the host system?

[asivery]

@benguild I'm exploiting the device, and I have confirmed it works on Mac, Windows and Linux

[benguild]

@benguild I'm exploiting the device, and I have confirmed it works on Mac, Windows and Linux

that's hilarious, what an interesting discovery. is it a permanent exploit or just temporary? how does it work? very interesting!

[asivery]

@benguild It's all temporary, when you unplug power from the device, or the device goes to sleep, it reverts back. Feel free to look through a repo called 'netmd-exploits' on my Github, if you'd like to learn more about its inner workings.

[benguild]

@benguild It's all temporary, when you unplug power from the device, or the device goes to sleep, it reverts back. Feel free to look through a repo called 'netmd-exploits' on my Github, if you'd like to learn more about its inner workings.

nice! I'm checking it out. Is there a way to force the player into charging but not data mode until it's rebooted? that'd be great for playback while charging from a PC/Mac.

[asivery]

@benguild I might be able to do it using exploits, for now it's not possible

[benguild]

@benguild I might be able to do it using exploits, for now it's not possible

ah okay. was just wondering! right now I often do playback using a wall charger to avoid cycling the battery. (I just leave the battery out)

would be great to be able to use my Mac to run the player instead without it kicking over to data. I tried a USB-C data filter but that confused the player for some reason.

[Invictaz]

@asivery are Hi-MD transfers to computer (uploads) encrypted like in Sonicstage? Or decrypted? And how about downloads (from PC to device) ?

You mentioned AT AT3+ and MP3 but there is also lineair PCM.

[asivery]

@Invictaz At the time when I was typing that message, PCM wasn't working. Now it is. The uploads and downloads indeed depend on SonicStage's keys, but they work on unencrypted files. All the encryption happens in the app and on the devices.

[Invictaz]

But how to get those keys? You need SonicStage installed? I wonder why nobody reverse engineered the Sony system or cracked the 3DES encryption. It's considered inadequate by today's security standards.

[asivery]

@Invictaz They are hardcoded in the app

[Invictaz]

So there is a masterkey?

[asivery]

Yes - here it is: https://github.com/thp/linux-minidisc/blob/minidisc-ffwd/libhimd/encryption.c#L131

[asivery]

All other keys are generated on upload, then either XORed with this one, or DES encrypted / decrypted with it

[Invictaz]

What about decrypting older .OMA files that you lost the encryption keys from?

[asivery]

That's still impossible

[Invictaz]

Where are the keys stored on Sonicstage? Somewhere in %appdata% ?

[asivery]

Not sure, but here:

why nobody reverse engineered the Sony system or cracked the 3DES encryption

you're wrong. I managed to add support for HiMD only thanks to the original linux-minidisc developers' notes, and by contacting them on the MD wiki discord. Nobody before re-implemented the DRM, but it had been reverse engineered long ago.

[Invictaz]

I am not wrong. You admitted that older .OMA tracks encrypted with 3DES are still not able to be cracked. That is something different than implementing DRM. I know of the efforts of the encryption algorythms and the QHiMDtransfer software did transfer a lot of Hi-MD functions already years ago.

[asivery]

Ah, I misunderstood you then, sorry. Indeed the 3DES keeping the tracks encrypted has not been cracked yet.

[Invictaz]

No worries you did a great job.

The thing is that we need to solve this matrix

If we have filled in the blanks the linux-minidisc project is complete. I know they forked their project in a new branch called linux-minidisc-ffwd and are backporting or cherry picking fixes which are coded in their main repo. But that's very very slow.

The Sony system how they generate the keys is also not understood. The .dll for AT3 encoding is from my understanding closed source and never reverse engineered.

Some of the DRM comes from OpenMG which is still available from Sony

https://www.sony-latin.com/en/electronics/support/downloads/W0001399

[asivery]

Per-track key generation is simple. Also, we solved the problem of a good quality ATRAC3 encoder with the help of the remote encoder. This is how the table looks like with this new version of Web Minidisc:

Edit: NetMD audio upload via USB works only on Sony portables, and I did omit the MZ-RH1 from having full support for all the ATRAC himd formats on purpose - it's one of the 2 unsupported devices - the MZ-RH1 and the MZ-DH10P.

[Invictaz]

The RH1 that is technically superior and should have all formats up and down? Why is it unsupported?

[asivery]

It's unsupported because my HiMD mode depends on firmware exploits. They patched the simple exploits in that device (they actually changed the whole main chip), and even though it has been exploited recently using another attack vector, I am not sure weather or not it is safe to include it in the main version of Web Minidisc. The MZ-RH1 is a very rare and expensive device - I don't want to brick any of them. Edit: As far as the camera goes - that one is unsupported because I don't own one, and I don't even know what chips and firmware it uses

[benguild]

The MZ-RH1 is a very rare and expensive device - I don't want to brick any of them.

womp womp. I have this player :(

[asivery]

@benguild I own an RH1 - I'll try to add support for it, but it might not work out in the end... In any case, on Linux it's possible to get the full HiMD mode working without any firmware exploits (just with ElectronWMD and not WMD in Chrome), so that's always a way out...

[benguild]

@benguild I own an RH1 - I'll try to add support for it, but it might not work out in the end... In any case, on Linux it's possible to get the full HiMD mode working without any firmware exploits (just with ElectronWMD and not WMD in Chrome), so that's always a way out...

I might be interested in that if I can spin up a VM and burn tracks more quickly than with Windows. The Sony software is awful and tends to lock up when I virtualize a Windows system on an ARM machine just to use it.

[asivery]

The problem with the MZ-RH1 is: The normal NetMD / HiMD units have (among others) these commands:

• Open / close mapped / eeprom memory for reading / writing / both
• Write previously open memory
• Read previously open memory
• Write DRAM (anti-shock memory)
• Read DRAM (anti-shock memory)

The MZ-RH1 does away with the first 3. @Sir68k managed to reproduce the behavior of the read / write memory commands using the DRAM ones, but it works only for some sections of the device's memory map (You can access the DRAM and peripheral segments, not the device RAM or ROM). In theory, I only need access to the peripheral segment to get the required firmware patches running, but I haven't tested it.

[Invictaz]

First I am not a fan of doing exploits to please a Chrome environment for a device that should do it natively. That's why I bought it. And yes I understand that the sending of unencrypted tracks is different from encrypted ones using SonicStage. The whole pesky encryption is a pain in the eye. Exploiting a very rare, very expensive and sadly fragile device worth also emotional value to me is troublesome.

I have seen the repository with exploits and the chipset of the RH1 needs to be investigated more. The RH1 had only 1 firmware v1.0 and I wonder why there were no updates. There is a factory mode but if you use it wrong you end up with a laser which does not work anymore or a brick.

[asivery]

@Invictaz The whole problem with these HiMD devices (and the reason why no one wrote a program that would upload ATRAC3 / ATRAC3+ / PCM) is that the devices need the disc to be signed - there's an integrity check value, that's generated from some files on the disc, and later gets written to a region outside of the HiMD filesystem using Sony vendor SCSI commands.

Linux lets me disconnect the kernel mass storage driver, even after it had already been loaded. That is not the case with Windows (to use devices with WebUSB, you need to change the driver to a libusb-type one, with something like Zadig - you most likely could get it to work with ElectronWMD on there, but you'd lose the ability to access the files directly, since Zadig would override the base mass storage driver, at least from what I understand). I don't know how that would work on Mac. When I tried to disconnect the kernel driver on there, nodejs got terminated by the OS. Granted, all my Mac testing was done on Catalina, but I'm not sure the added an ability to detach an already loaded kernel driver.

The only thing my exploit does, is it rewrites the USB VID and PID pair, and changes the interface class from 0x08 subclass 0x05 (mass storage device using SCSI-over-ATAPI) to 0xFF subclass 0x00(vendor specific). This makes it possible to: a) Use SonicStage alongside Web Minidisc Pro - the SonicStage driver looks for a device with a different VID/PID b) Use the device within Chrome (WebUSB doesn't let you attach to devices which declare themselves as mass storage) c) Let you still see the devices' filesystem, after the exploit clears itself

[Invictaz]

Having the Zadig swap already has its constraints but I prefer changing a pc based driver over an exploit.

Regarding "the devices need the disc to be signed - there's an integrity check value, that's generated from some files on the disc, and later gets written to a region outside of the HiMD filesystem using Sony vendor SCSI commands."

More protection problems... sigh.

If you could document the vendor specific SCSI commands we would be further along the way.

[asivery]

@Invictaz I still need to do tests on how Windows behaves - everything I've said in this thread are purely just my expectations. I am developing all of this on Linux, because it's just a lot easier, and don't really have a Windows PC on hand to test everything on (VMs have problems with the device switching from NetMD mode to HiMD mode).

I'll do my best to document the SCSI commands, but most of that has already been documented over at the linux-minidisc wiki.

Edit: The only thing I tested on windows is the exploit method with my MZ-NH700, and it worked.

[Invictaz]

If we would have an RH1 donor board it would be much easier

[asivery]

Why? We already have the MZ-RH1's firmware

[GrayzerGray]

Superb, can’t wait to get trying that. Great efforts providing this feature, I know it’s been alluding you for a while. Well done and thank youGraySent from my iPhoneOn 8 Apr 2023, at 00:16, asivery @.***> wrote: @Invictaz It wasn't as simple back then. Getting it working in Chrome requires temporarily opening the normal MD mode, then exploiting the firmware and switching back. It's impossible without that, because WebUSB doesn't let you connect to mass storage device. To get around that, I load up an exploit and patch the USB descriptor, so that the OS misidentifies the HiMD device.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[GrayzerGray]

Thankfully I have the RH1 in mint other than the famous OLED failure (Not brave enough to do the fix yet). And also have an MZ-RH910 so all good. If anyone is on the hunt for an RH1 boxed NOS will set you back £1000. Battery most likely shot and no guarantee the OLEDs will not be good. Mine still cost £400 in the condition it was in and factored in an OEM battery from AliExpress. But thank you again for getting Hi-MD functionality out of itSent from my iPhoneOn 8 Apr 2023, at 02:47, Ben Guild @.***> wrote:

The MZ-RH1 is a very rare and expensive device - I don't want to brick any of them.

womp womp. I have this player :(

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[asivery]

@benguild I got the full HiMD mode running on the MZ-RH1

[Invictaz]

@asivery If we would have a donor board we could test without sacrificing a good working MZ-RH1.

How did you manage to get it fully working?

[asivery]

@Invictaz Recently, @Sir68k managed to dump the firmware from his MZ-RH1. I just used the same exploit to dump mine, then reverse engineered where the firmware keeps the values I need to change, and patched them as I would in a normal HiMD device.

[Invictaz]

Great work. What has changed since last week?

[asivery]

I just thought that if I can implement that functionality, I should. There will be an appropriate warning in the app before running the exploit code. If someone decides to continue, then bricks their device, that's on them.

[benguild]

I just thought that if I can implement that functionality, I should. There will be an appropriate warning in the app before running the exploit code. If someone decides to continue, then bricks their device, that's on them.

Is there a non-exploity way to get this running on Linux? I might just spin up a VM versus brick risk. The Windows NetMD software is terrible, slow, and unstable and if there's any modern option I imagine it has to be better.

My device is from Japan so the firmware is likely different as it displays characters in Japanese on some screens.