[FR] Support for cso/gzip/chd compressed ISOs

eadmaster commented 4 years ago

CSO (aka CISO) can be converted with maxcso.

This could have some advantages:

save space on USB and HDD storage devices due to compression;
no need to remove padding files on ISOs to save space;
speedup loading from slow storage devices like USB, since less data needs to be transferred.

Any file format supported by PCSX2 will be fine for me, so i won't have to keep multiple copies of my collection.

JoseAaronLopezGarcia commented 2 years ago

python ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"
  File "C:\Users\user\Desktop\OPL_ZSO_CACHE\ziso.py", line 292
    except GetoptError, err:

ziso.zip try this one

mrchrisster commented 2 years ago

python ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"
  File "C:\Users\user\Desktop\OPL_ZSO_CACHE\ziso.py", line 292
    except GetoptError, err:

you are using python 3. Try python2 ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"

orcutt989 commented 2 years ago

python ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"
  File "C:\Users\user\Desktop\OPL_ZSO_CACHE\ziso.py", line 292
    except GetoptError, err:

ziso.zip try this one

Perfect! Looks like you made it work with Python3, thanks!

Any idea what I should do with this?

Align error, you have to increase align by 1 or CFW won't be able to read offset above 2 ** 31 bytes

Do I need -a 1 ?

python ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"
  File "C:\Users\user\Desktop\OPL_ZSO_CACHE\ziso.py", line 292
    except GetoptError, err:
you are using python 3. Try python2 ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"

Yes that's what I figured, but @JoseAaronLopezGarcia provided one above that works with python3.

mrchrisster commented 2 years ago

Quick test of compressing 25 games: ISO 52.8GB ZSO 39.0GB

JoseAaronLopezGarcia commented 2 years ago

Difference between plain ISO and ZSO on USB devices:

ISO: https://drive.google.com/file/d/1O7EGH1cO09SexxgyfadjHJrUvIR0kkAu/view

ZSO: https://drive.google.com/file/d/15E_9sWO_eNHl52XDjOpBIueADcZ9_ZwY/view

Difference is night and day. One stutters like crazy, the other is smooth.

mrchrisster commented 2 years ago

I made it super straight forward for people to use in Windows.

Download ISO-ZSO Compressor.zip
Unzip and click on python-3.10.4-amd64.exe to install python3 or enter python in a powershell window and install from app store
Drop any ISO on ISO-ZSO Compressor Drop Files on here.bat and it will process all ISO's in the folder.

orcutt989 commented 2 years ago

I opened an issue for @israpps on HDL-Batch hopefully we can get support for loading files onto a hard drive. https://github.com/israpps/HDL-Batch-installer

Cengobro commented 2 years ago

Ability to split files larger than 4gb for python script and will there support for OPL to read the split file ?

malvarenga123 commented 2 years ago

Would someone mind pushing a fix to that app (maxcso), so that it doesn't cut off the first 16 sectors???

What do you mean? Compressing an iso with maxcso to zso and then decompressing it back to iso generates an identical file.

sionisaac commented 2 years ago

I have tried a few games compressed with mixed results. In theory if compressed Iso's can be used from the internal drive will it eliminate the video and audio stutter? Can all games be compressed? I noticed that Shadow of the Colossus will not work at all from your opl mod it errors about fragmentation. Would it be possible to make a compression tool that links to an online db so it will only compress games that work flawlessly? I am willing to put in some hours testing compressed games. I have around 1800 games already in iso format (every exclusive PS2 game favouring USA region without duplicates for other regions and pretty much every other English language game worthy of being in my collection).

JoseAaronLopezGarcia commented 2 years ago

I have tried a few games compressed with mixed results. In theory if compressed Iso's can be used from the internal drive will it eliminate the video and audio stutter? Can all games be compressed? I noticed that Shadow of the Colossus will not work at all from your opl mod it errors about fragmentation. Would it be possible to make a compression tool that links to an online db so it will only compress games that work flawlessly? I am willing to put in some hours testing compressed games. I have around 1800 games already in iso format (every exclusive PS2 game favouring USA region without duplicates for other regions and pretty much every other English language game worthy of being in my collection).

The fragmentation error is not a problem with OPL, it literally means the file is fragmented on the filesystem. Use window's defragmenter to fix that. I've tested Shadow of the Colossus on USB and it worked flawlessly with no stutter.
Of course this won't eliminate stuttering for all games, it mostly depends on how big the FMVs are, but there is quite a big list of games with zero stutter.
On internal HDD it should work fine, there's reports of people getting it to work, but I haven't tested it myself.
Edit: ZSO confirmed working perfectly on HDD. Just as good as an ISO.
All games can be compressed, but for the time being there's no support for dual layer games.

JoseAaronLopezGarcia commented 2 years ago

IMPORTANT INFO:

Lots and lots of people (myself included) have had issues generating ZSO files with maxcso. If your ZSO file is not detected by OPL or crashes, try compressing the ISO with the python script provided.

JoseAaronLopezGarcia commented 2 years ago

Guys, the latest build is now reported to work fine with SMB and internal HDD: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111

Only missing feature so far is support for DVD9 games.

orcutt989 commented 2 years ago

Guys, the latest build is now reported to work fine with SMB and internal HDD:

https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111

Only missing feature so far is support for DVD9 games.

How does one get the ZSO games on the HD @JoseAaronLopezGarcia ?

INDRAPhilip commented 2 years ago

Guys, the latest build is now reported to work fine with SMB and internal HDD: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111 Only missing feature so far is support for DVD9 games.

How does one get the ZSO games on the HD @JoseAaronLopezGarcia ?

For usb, just install python 2.7 and lz4 (install it by terminal itself) and then use this command in terminal: "sudo apt install python2 curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py python get-pip.py python -m pip install lz4" to convert the game below! python ziso.py -c 9 "input.iso" "output.zso" for internal HD, wait for programs like HdlDump and HDL-Batch to update

israpps commented 2 years ago

for Internal HD, wait for the update of programs like HDL-Batch I'm waiting for cso support in upstart HdlDump

JoseAaronLopezGarcia commented 2 years ago

DVD9 (dual layer) games now work perfectly. https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111

mrchrisster commented 2 years ago

You're killing it, great job! Just curious after you got familiar with the PS2 SDK if you still see a chance for CHD support down the road?

CHD is way too big to fit on a PS2 imo, CSO is much more simpler and requires very basic code.

JoseAaronLopezGarcia commented 2 years ago

You're killing it, great job! Just curious after you got familiar with the PS2 SDK if you still see a chance for CHD support down the road?

CHD is way too big to fit on a PS2 imo, CSO is much more simpler and requires very basic code.

Not a chance. I already had to do lots and lots of cleanup and make sure I use the smallest possible memory to fit ZSO (which is the smallest and fastest possible format you can have) and have it work well with all devices, specially SMB (which already takes up lots of RAM and CPU). CHD would never fit into the RAM we have and even if it does, don't expect anything above a few frames per second. We are talking about a format made for PCs and newer consoles where they can read huge chunks of the format and have extremely complex readers and drivers and whatnot.

mrchrisster commented 2 years ago

Makes sense. Thanks for all your work on this and the explanation!

sionisaac commented 2 years ago

If I get this error when compressing Iso's "Align error, you have to increase align by 1 or CFW won't be able to read offset above 2 ** 31 bytes", does it mean that the compressed iso will not work properly? I've batch compressed a lot of Iso's and some have thrown that error but when I try to play the games they appear to work fine. Just wondering if I need to recompress those games or if I can continue using them as is.

synrais commented 2 years ago

That seems to be a PSP CFW warning, adding -a 2 will make the error disappear for those titles but I'm unsure if this hurts other titles at all, it does increase their size by a miniscule amount. You may of noticed the iso that error do so before completion, so its safe to say those ones are borked. It is happening because they are ending up larger than 4.2GB, the fat32 file size limit. (2^32)

More info here. https://www.wikiwand.com/en/.CSO

For now I'm logging the error titles with these, then reprocessing with -a2 for just those titles. https://drive.google.com/file/d/1dhaeIQkNIVfWWoVQdxnJtuh5Ua4K8rEv/view?usp=sharing Alone in the Dark The New Nightmare.iso Champions of Norrath.iso Critical Velocity.iso Final Fantasy X.iso Final Fantasy X2 International Last Mission.iso Forbidden Siren 2.iso God of War I.iso God of War II.iso Haunting Ground.iso Kessen III.iso Kessen.iso Killzone.iso Kingdom Hearts II Final Mix.iso Metal Gear Solid 2 Substance.iso Metal Gear Solid 3 Sub (Disc 3).iso Midnight Club 3 DUB Remix.iso Rogue Galaxy.iso Sakura Wars So Long My Love.iso Shadow of Rome.iso Shin Megami Tensei Digital Devil Saga 2.iso Star Ocean Till the End of Time (Disc 1).iso Star Ocean Till the End of Time (Disc 2).iso Suikoden V.iso Syberia II.iso Tekken 5.iso Wild Arms Alter Code F.iso Xenosaga Episode I.iso Yakuza 2.iso

I tried decompressing but keep getting the same error? 0 block: 0x0004EFDC 156 Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 48

JoseAaronLopezGarcia commented 2 years ago

That seems to be a PSP CFW warning, adding -a 2 will make the error disappear for those titles but I'm unsure if this hurts other titles at all, it does increase their size by a miniscule amount. You may of noticed the iso that error do so before completion, so its safe to say those ones are borked. It is happening because they are ending up larger than 4.2GB, the fat32 file size limit. (2^32)

More info here. https://www.wikiwand.com/en/.CSO

For now I'm logging the error titles with these, then reprocessing with -a2 for just those titles. https://drive.google.com/file/d/1dhaeIQkNIVfWWoVQdxnJtuh5Ua4K8rEv/view?usp=sharing Alone in the Dark The New Nightmare.iso Champions of Norrath.iso Critical Velocity.iso Final Fantasy X.iso Final Fantasy X2 International Last Mission.iso Forbidden Siren 2.iso God of War I.iso God of War II.iso Haunting Ground.iso Kessen III.iso Kessen.iso Killzone.iso Kingdom Hearts II Final Mix.iso Metal Gear Solid 2 Substance.iso Metal Gear Solid 3 Sub (Disc 3).iso Midnight Club 3 DUB Remix.iso Rogue Galaxy.iso Sakura Wars So Long My Love.iso Shadow of Rome.iso Shin Megami Tensei Digital Devil Saga 2.iso Star Ocean Till the End of Time (Disc 1).iso Star Ocean Till the End of Time (Disc 2).iso Suikoden V.iso Syberia II.iso Tekken 5.iso Wild Arms Alter Code F.iso Xenosaga Episode I.iso Yakuza 2.iso

I tried decompressing but keep getting the same error? 0 block: 0x0004EFDC 156 Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 48

yeah we're having issues with the decompression routine. Hopefuly someone will eventually make a better compressor/decompressor with GUI, blackjack and hookers.

sionisaac commented 2 years ago

Can you explain the command line for adding -a2 . Also what is the command line for decompressing?

JoseAaronLopezGarcia commented 2 years ago

Can you explain the command line for adding -a2 . Also what is the command line for decompressing?

The alignment (-a 2 or -a 3) is used to allow for 32 bit systems to address compressed ISO files above 4GB. So when a file is too big then the alignment needs to be adjusted. The command to decompress would be -c 0, but it seems the decompression function is broken for the python script.

sionisaac commented 2 years ago

I am still getting mixed results with this compression. Usb fat32 drives work with most < 4gb games but I've found some that do not work. Samba shares have far less compatibility even with games that work from usb. I've compressed around 1000 games and used the error logs to recompress the games that gave the align error. Cannot get FFX to work from Samba or usb. I've tried multiple samba setups. Windows/Linux/raspberry pi 0,2,4,400 and the setup makes no difference. Some of the compressed 007 games hang at black screen from Samba, work fine from usb yet the uncompressed Iso's work fine from Samba. I challenge the reports that internal HDD works fine with all compressed games since this is an extremely new way to play games. OPL probably needs a complete overhaul to be ready for it. And we need a proper compression and decompression program that takes into account correct alignment for iso size.

I love that compressed games are possible. But we are no where near a situation where all games can be compressed and played. Please tell me I'm wrong.

JoseAaronLopezGarcia commented 2 years ago

Hey guys, new release here: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111 PS2 Logo is now working for ZSO files on all devices!

On a side note, many users are experiencing a black screen when playing ZSO games on internal HDD. We have come to the conclusion that ZSO actually reads data at a faster rate than plain ISO and this introduces many timing issues. Most games can easily be fixed by using Mode 1 and/or Mode 2, and maybe even lowering the DMA speed and HDD Cache.

JoseAaronLopezGarcia commented 2 years ago

I am still getting mixed results with this compression. Usb fat32 drives work with most < 4gb games but I've found some that do not work. Samba shares have far less compatibility even with games that work from usb. I've compressed around 1000 games and used the error logs to recompress the games that gave the align error. Cannot get FFX to work from Samba or usb. I've tried multiple samba setups. Windows/Linux/raspberry pi 0,2,4,400 and the setup makes no difference. Some of the compressed 007 games hang at black screen from Samba, work fine from usb yet the uncompressed Iso's work fine from Samba. I challenge the reports that internal HDD works fine with all compressed games since this is an extremely new way to play games. OPL probably needs a complete overhaul to be ready for it. And we need a proper compression and decompression program that takes into account correct alignment for iso size.

I love that compressed games are possible. But we are no where near a situation where all games can be compressed and played. Please tell me I'm wrong.

A few things to consider:

The SMB driver seems to have an issue in the latest OPL. Try different cache settings (8 seems to work best).
For games that give you a black screen, try using Mode 1 and 2.

rickgaiser commented 2 years ago

We have come to the conclusion that ZSO actually reads data at a faster rate than plain ISO

Can you provide some proof to this claim?

The HDD is very fast on the IOP, but the speeds also vary greatly depending on the requested block sizes. So when measuring speed of ISO compared to ZSO it is important to also take the effect of the newly introduced cache into account. The HDD speeds I measured are:

1 sector (2k) = 7MB/s
2 sectors (4k) = 12MB/s
4 sectors (8k) = 20MB/s
8 sectors (16k) = 32MB/s

The same applies to other devices. link to measurement results

At the same time I highly doubt the IOP can decompress 30MB/s @ 30MHz. That would mean 4 bytes in 4 cpu cycles.... I know lz4 is fast, but this seems impossible.

Can you put the cache changes into a separate PR? I think they are more easy to review, test and merge.

sionisaac commented 2 years ago

Hey guys, new release here: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111 PS2 Logo is now working for ZSO files on all devices!

On a side note, many users are experiencing a black screen when playing ZSO games on internal HDD. We have come to the conclusion that ZSO actually reads data at a faster rate than plain ISO and this introduces many timing issues. Most games can easily be fixed by using Mode 1 and/or Mode 2, and maybe even lowering the DMA speed and HDD Cache.

How are these people transferring the zso images to internal drive?

AKuHAK commented 2 years ago

How are these people transferring the zso images to internal drive?

https://github.com/AKuHAK/hdl-dump/actions/runs/2295246292 cli version will work just specify the main elf file.

sionisaac commented 2 years ago

How are these people transferring the zso images to internal drive?

https://github.com/AKuHAK/hdl-dump/actions/runs/2295246292 cli version will work just specify the main elf file.

?? What?? Where is the command line version? The git has no releases. Does this have to be built from source?

AKuHAK commented 2 years ago

Artifacts section

AKuHAK commented 2 years ago

JoseAaronLopezGarcia commented 2 years ago

We have come to the conclusion that ZSO actually reads data at a faster rate than plain ISO

Can you provide some proof to this claim?

The HDD is very fast on the IOP, but the speeds also vary greatly depending on the requested block sizes. So when measuring speed of ISO compared to ZSO it is important to also take the effect of the newly introduced cache into account. The HDD speeds I measured are:

1 sector (2k) = 7MB/s

2 sectors (4k) = 12MB/s

4 sectors (8k) = 20MB/s

8 sectors (16k) = 32MB/s

The same applies to other devices. link to measurement results

At the same time I highly doubt the IOP can decompress 30MB/s @ 30MHz. That would mean 4 bytes in 4 cpu cycles.... I know lz4 is fast, but this seems impossible.

Can you put the cache changes into a separate PR? I think they are more easy to review, test and merge.

I've linked to videos showing the difference on USB and it's clear that it reads data faster (reading one ZSO sector usually translates to reading more than one ISO sector). Also the many games that now have zero cinematic lag on USB.

This is also evident on HDD were many games need Mode 1 for ZSO to work, due to timing issues introduced by the faster transmission of data.

As for games itself, the IOP doesn't need to process 30MB/s, it needs to process the data rate of a DVD at 1X-4X speed (depending on the game), most games will crash at higher rates (hence the need for Mode 1 as mentioned above).

The thing about the cache is that it is essential for ZSO to perform correctly, so I can't really get rid of it or turn it into another feature and it's best to let users configure its size (or even disable it), that way it's memory consumption is dynamic (and unused for ISO).

sionisaac commented 2 years ago

Changing the SMB cache to 8 seemed to work for the few games I was having difficulties with. Final fantasy X is now working (yeah). Final Fantasy X2 will not work no matter what I try. Looking at the compatibility list for OPL it says the American version doesn't need any game modes except 6 disable igr. I've tried this and it doesn't work just black screen. I've also tried modes 1 and 2 and both and mode 3 because the list says to use mode 3 for pal and jap versions. And still no go. The game works fine on both my PS2 fats with internal uncompressed iso. Does anyone have a working config for SMB?

What are the optimal cache settings for USB and internal HDD and DMA(this can only be changed for games on internal drive).

I think we need to start a db of game configs for compressed games on HDD/USB/SMB. I'm working through the games right now alphabetically but I'm concentrating on SMB since all my internal games are working fine. Once it becomes easy to transfer zso images to HDD I'll start replacing games and going through configs for that.

AKuHAK commented 2 years ago

I dont think that I will do something with hdl-dump, so you can freely use the version from artifacts, its proved that it is working fine. Also, for non-working games you can try to repack them with different threshold (python script )

AKuHAK commented 2 years ago

For example -t 80 or -t 90. It will increase a size but maybe solves other problems.

orcutt989 commented 2 years ago

I copied Star Wars - Jedi Starfighter (2.10).zso with HDL DUMB gui to my internal HDD. I had to manually enter the Startup and Name fields. The startup value I am pretty sure was correct, but I just made up the Name field.

I copied @JoseAaronLopezGarcia's OPNPS2LD.ELF to my memcard and booted it, but it did not recognize the ZSO file.

EDIT: Nevermind, the name of the game in OPL was the made up name I put into the HDL DUMB Name field.

UPDATE: Getting black screen but will try this https://github.com/ps2homebrew/Open-PS2-Loader/issues/225#issuecomment-1129939521

UPDATE2: No more black screen with MODE1/MODE2 enabled for game settings.

AKuHAK commented 2 years ago

HDL DUMB gui

Thanks for confirmation that gui also works

malvarenga123 commented 2 years ago

For anyone interested, @AKuHAK found out that the issue with maxcso was the block size used to create ZSO files. So to create compatible files just use: maxcso --block=2048 --format=zso input.iso

mrchrisster commented 2 years ago

For anyone looking for the latest relaese: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader (Check under OPL archive section at the bottom for a link)

rickgaiser commented 2 years ago

ZSO has been merged into OPL. It's part of the 1.2.0 Beta builds now. Latest releases can be downloaded here: https://github.com/ps2homebrew/Open-PS2-Loader/releases

VisionR1 commented 1 year ago

I have try it some game zso format with OPL v1.2.0-Beta-1940-d58e721 seem to work until now, the game Disney's Treasure Planet i try it with zso not play only the iso format, and the Jackie Chan Adventures (Europe) not play with all format and mode, i try it and cso format but not show it inside in opl i dont know if support, what else format can try it?

ps2homebrew / Open-PS2-Loader

[FR] Support for cso/gzip/chd compressed ISOs #225