[FR] Support for cso/gzip/chd compressed ISOs

eadmaster commented 5 years ago

CSO (aka CISO) can be converted with maxcso.

This could have some advantages:

save space on USB and HDD storage devices due to compression;
no need to remove padding files on ISOs to save space;
speedup loading from slow storage devices like USB, since less data needs to be transferred.

Any file format supported by PCSX2 will be fine for me, so i won't have to keep multiple copies of my collection.

TnA-Plastic commented 5 years ago

There would be a disadvantage however...

The support would need code to run on the IOP (to decompress it) and the filesystem-support in-game on USB was already stripped (hence defragmenting ISOs and VMCs), due to it needing too much code and causing incompatibilities with a lot of games.

I would like to have support for it as well tho'.

AKuHAK commented 5 years ago

http://psx-scene.com/forums/f19/%5Bhelp%5D-loader-compressed-isos-103805/

rickgaiser commented 5 years ago

I really like this idea and have already been playing with lz4 since the decompression requires so little resources. I didn't know about the ZSO and CSOv2 formats that also use lz4 (and deflate) for compression. Perhaps I'll do a quick experiment to see what's possible with the iop's low speed and lz4...

eadmaster commented 5 years ago

The reason why i suggested CSO it's because it is supported by PCSX2 as well, so we could have a good archival format for both real hardware and emulation. Gzipped isos would be another option.

TnA-Plastic commented 5 years ago

SMB could also benefit a bit, since it wouldn't be un-/decompressed on the host-side (given someone uses something like NTFS-Compression) [before the transfer], thus the transfered game-data is compressed there as well, unlike a filesystem-based compression on the host.

It depends if it would dive bandwidth more down than it can yield in compression, or not.

AKuHAK commented 5 years ago

I really like this idea and have already been playing with lz4 since the decompression requires so little resources. I didn't know about the ZSO and CSOv2 formats that also use lz4 (and deflate) for compression. Perhaps I'll do a quick experiment to see what's possible with the iop's low speed and lz4...

According to this lz4 (default compression level) decompression is only 3 times slower than simple memcpy so maybe PS2 will be possible to handle this. Some tips:

it will be possible to compress most of the titles less then FAT32 limit, so no need to split image for USB storage
@rickgaiser if you do testing maybe it will be useful to not compress media files. CISO plus has the feature to not compress multimedia files: Leave PMF and AT3 files alone.

TnA-Plastic commented 5 years ago

Well, the PS2 uses AT3 sometimes I suppose, but others like PSS are more common!

It would be neat to selectively choose the files which should not be compressed and even more so would 'profiles' for games (which files to compress and which shouldn't be compressed, if it shouldn't compress the entire ISO) be a cool thing, IMO.

sp193 commented 5 years ago

Long ago, PS2ESDL used to support compressed disc images. Not CSO, but its own format. It used LZO and compressed only sectors that could be sufficiently compressed. I eventually removed it because it seemed like nobody wanted to use a different format.

Speed was never really improved. Sometimes it did, but it almost never helps FMVs (a benchmark by some people) and so on because videos are usually compressed (and compressing compressed data is usually a bad idea). So I think we should not consider a speed improvement as a condition for considering whether to add this feature.

AKuHAK commented 5 years ago

Speed was never really improved. Sometimes it did, but it almost never helps FMVs (a benchmark by some people) and so on because videos are usually compressed (and compressing compressed data is usually a bad idea).

@sp193 Yeah, it is why I mention the ability to skip movies. As I know CSO also has the ability to skip sectors that are not compressable (for example if the compression ratio is less then 95%). I don't think that this will improve speed, but it can save a lot of space (maybe about 20% can be saved even with the lowest/fastest compression ratio).

This can be very useful for internal HDDs cause they are very fast but sometimes has not so much free space.

eadmaster commented 5 years ago

I expect this to speedup loading times of vector and textures data of the disk (if they are not already compressed).

Also i could skip removing zero-filled/padding files from the Isos, because they are easily compressed.

gingerbeardman commented 4 years ago

This is really exciting!

G-BOT12 commented 3 years ago

So if the idea has been renamed and reopened is this going to be an actual thing? Anyone? That would kind of be convient for the FAT32 limit problem that doesn't have a quick solution.

AKuHAK commented 3 years ago

This feature needs a LOT of work, cause it is very low level and should be always alive while game is running.

eadmaster commented 3 years ago

little update: added CHD to the list of image formats to consider, since pcsx2 is now getting support for it

In the end any compressed format also compatible with pcsx2 would be fine for me.

malvarenga123 commented 3 years ago

Just thinking out loud, but is it possible for the decompression to be done server-side? For example, a program like OPL-Server reads the chd, decompresses it and feeds OPL just the raw uncompressed data?

eadmaster commented 3 years ago

Just thinking out loud, but is it possible for the decompression to be done server-side? For example, a program like OPL-Server reads the chd, decompresses it and feeds OPL just the raw uncompressed data?

Prolly easier than native compression support on the console, but i'd prefer to load these isos from USB.

Also, data needs to be decompressed before sent over the net, so no speed gain this way...

orcutt989 commented 2 years ago

Would definitely love this feature!

INDRAPhilip commented 2 years ago

It would be amazing another file conversion format besides iso, since pcsx2 is compatible or opl-server even if there is no speed improvement, there will be more spaces for all devices allowing to add more games or even more space for USB defragmentation! Because that's what I was observing on pcsx2 considering that many emulators support these formats, such as even retroarch which is zip, even though it is not recommended to use it very often for the retroarch core for ps2.

mrchrisster commented 2 years ago

CHD support would save a lot of space on my HDD. Would love to see this being a thing!

JoseAaronLopezGarcia commented 2 years ago

Hello guys! I've recently finished working on ARK-4 CFW for PSP and one of the features I've worked with was full support for all known compressed file formats (CSO, ZSO, JSO, CSOv2 and DAX), all of this using the same base algorithm that can be easily compiled on PC and other platforms. I thought maybe this project could benefit from it.

https://github.com/PSP-Archive/ARK-4/blob/5ee890712cb336b041d84012efd5eae5026a1af2/core/inferno/isoread.c#L251

This is the core Inferno Driver reader I made. It has all the IO speed improvements which means that most calls will have at most 2 IO (seek+read) calls.

Out of all the compressed formats, for PS2 I recommend either JSO or ZSO. They are both extremely fast and lightweight. ZSO is faster and more lightweight but it has worse compression.

JoseAaronLopezGarcia commented 2 years ago

For a lightweight and cross-platform LZ4 implementation: https://github.com/PSP-Archive/ARK-4/blob/main/core/systemctrl/src/lz4.c

For LZO: https://github.com/PSP-Archive/ARK-4/blob/main/core/systemctrl/src/minilzo.c

malvarenga123 commented 2 years ago

If CHD had LZ4/LZ5/Lizard support, it would be ideal. High decompression speed with a unified format for most discs.

JoseAaronLopezGarcia commented 2 years ago

CHD is way too big to fit on a PS2 imo, CSO is much more simpler and requires very basic code.

mrchrisster commented 2 years ago

I think eventually CHD will become the standard for preservation of all disc based consoles. That would be too bad if the PS2 won't be able to handle it.

JoseAaronLopezGarcia commented 2 years ago

Ok so I've read the source code for OPL's isofs module and I know exactly how to add support for ZSO. It should be simpler than I initially thought since it requires only implementing sector-level reads instead of arbitrary reads like on PSP. Can anyone link me to a PS2 SDK I can use to compile OPL?

Wolf3s commented 2 years ago

Ok so I've read the source code for OPL's isofs module and I know exactly how to add support for ZSO. It should be simpler than I initially thought since it requires only implementing sector-level reads instead of arbitrary reads like on PSP. Can anyone link me to a PS2 SDK I can use to compile OPL?

Here is the link of the ps2sdk: https://github.com/ps2dev/ps2sdk

However if you prefer the easy mode then git clone this: https://github.com/ps2dev/ps2dev This will automaticly install the whole environment.

Good luck

JoseAaronLopezGarcia commented 2 years ago

Guys I got it working!

Here's the first release: https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r000

Tested with USB stick. FMV stuttering is not fixed but you do get some games really shrunken, gameplay is fine.

Edit: I've implemented a sector cache on block devices (i.e. USB) which helps reduce FMV stuttering and load times. https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111

orcutt989 commented 2 years ago

This is amazing!! Can anyone link to a tool for creating ZSO files? Googling sends me to blog posts from 2006.

EDIT nvm

https://github.com/unknownbrackets/maxcso

JoseAaronLopezGarcia commented 2 years ago

There's a python script included with the OPL_ZSO build to generate zso files.

orcutt989 commented 2 years ago

There's a python script included with the OPL_ZSO build to generate zso files.

Awesome. Is there a good way to load them onto the internal HD?

JoseAaronLopezGarcia commented 2 years ago

There's a python script included with the OPL_ZSO build to generate zso files.

Awesome. Is there a good way to load them onto the internal HD?

It would require new tools to install them. I guess someone will make one eventually.

JoseAaronLopezGarcia commented 2 years ago

Updated the release with an important bug fix that would prevent games from showing up in the menu list.

JoseAaronLopezGarcia commented 2 years ago

Games I've tested so far:

Aggressive Inline: intro FMV stutters. Gameplay is perfect.
Ape Escape 3: works perfectly.
ATV Offroad Fury 2: takes a while to bootup, intro FMV stutters, otherwise works perfectly.
Black: very minor FMV stuttering though FMV audio is perfect and cutscenes can be watched. Everything else perfect.
Bully: intro FMV stutters, rest of the game is perfect.
Call of Duty Finest Hour: works perfectly, zero FMV lag.
Call of Duty 2 Big Red One: some minor stuttering in the intro FMV, otherwise plays perfect.
Dragon Ball Z Budokai: works perfect with no lag.
Dragon Ball Z Budokai 3: intro FMV stutters, rest of the game is perfect.
Dragon Ball Z Budokai Tenkaichi 3: intro FMV stutters, rest of the game is perfect.
FIFA 14: works perfectly.
Grand Theft Auto 3: works perfect. No FMV lag at all.
Grand Theft Auto Vice City: works perfectly. No FMV lag at all.
Grand Theft Auto San Andreas: intro FMV stutters, gameplay and in-game cutscenes work perfectly with no lag.
Grand Theft Auto Liberty City Stories: minor stutter in the intro FMV, rest of the game is perfect.
Grand Theft Auto Vice City Stories: works perfectly, zero FMV lag.
Gun: intro and in-game FMV stutters, rest of the game is perfect.
Lord of the Rings - Return of the King: works perfectly. Zero FMV lag.
Metal of Honor Frontline: works perfectly, no FMV lag at all.
Metal Gear Solid 2 Sons of Liberty: intro FMV stutters. Gameplay is perfect and in-game cutscenes have no lag.
Metal Gear Solid 3 Subsistance: works perfectly. No FMV lag at all.
NBA Street Vol. 2: very minor intro FMV stutter, otherwise works perfectly.
Need for Speed Undercover: intro and cutscenes FMV stutters, some popping of buildings during gameplay, otherwise works perfectly.
Tekken 4: intro FMV cuts off, rest of the game is perfect.
Tekken Tag Tournament (converted to DVD): intro FMV stutters. Gameplay is perfect.
Scarface: intro FMV stutters, rest of the game works perfect.
Silent Hill 2: very minor stutter.
Silent Hill 3: intro FMV stutter, in-game cutscene are perfect.
Simpsons Hit and Run: heavy stuttering on FMVs, gameplay is fine.
Soul Calibur 3 Arcade Edition: intro FMV stutters. Gameplay is perfect.
Shadow of the Colossus: works perfectly, no FMV lag at all.
Splinter Cell: works perfectly.
Ratchet and Clank: intro FMV and pre-rendered cutscenes stutter.
Red Dead Revolver: works perfectly.
Resident Evil 4: works perfectly, no FMV lag at all.
The Godfather: works perfectly, zero FMV lag.
Transformers: intro and in-game FVM stutters, game crashes after first call.
Psi-Ops the Mindgate Conspiracy: works perfectly, zero FMV lag.
WWE Smackdown vs RAW 2011: works perfectly.

I will keep updating this post as I test more games.

Tested on PS2 Fat with USB drive formatted in FAT32.

Generally speaking, in-game cutscenes don't have stuttering, only pre-rendered cutscenes have. Load times are also very acceptable.

To convert ISO games to ZSO you can use the provided Python 2 script: python2 ziso.py -c 9 "input.iso" "output.zso"

Or you can use maxcso: https://github.com/unknownbrackets/maxcso/releases maxcso --format=zso "input.iso"

mrchrisster commented 2 years ago

Amazing! Which games would be over 4GB?

Masamune3210 commented 2 years ago

Dual Layers, Probably

mrchrisster commented 2 years ago

This would be the list then I guess https://everipedia.org/wiki/lang_en/List_of_PlayStation_2_DVD-9_games

sp193 commented 2 years ago

The maximum size for a DVD is over 4GB. There are a handful of such games that end up being over 4GB. Some games may be padded to push the content towards the outer tracks, for performance.

TnA-Plastic commented 2 years ago

Awesome work @JoseAaronLopezGarcia! 🙂

sionisaac commented 2 years ago

This is great. Can we please have a tool for transferring to internal HDD though. USB sucks. I have 2 fat ps2's both with 2tb internal drives that are full. I would love to extend my PS2 library by compressing some games.

TnA-Plastic commented 2 years ago

This would be the list then I guess https://everipedia.org/wiki/lang_en/List_of_PlayStation_2_DVD-9_games

Not necessarily! Some games above 4GB compress below it! A lot actually, as it seems!

TnA-Plastic commented 2 years ago

This is great. Can we please have a tool for transferring to internal HDD though. USB sucks. I have 2 fat ps2's both with 2tb internal drives that are full. I would love to extend my PS2 library by compressing some games.

The tools haven't been adapted for internal HDD yet. Internal HDD is also not yet tested.

INDRAPhilip commented 2 years ago

I'm not having success to create the zso files by the scrip in python by the terminal in windows, in maxcso it doesn't work to read the files in the opl so how do I create the files properly?

TnA-Plastic commented 2 years ago

Would someone mind pushing a fix to that app (maxcso), so that it doesn't cut off the first 16 sectors???

mrchrisster commented 2 years ago

python script seems to work well for me over here on debian buster. I had to install lz4 with pip install lz4 using python 2.7.

Edit: For people having issues with ziso.py, here is how to get it working on Windows WSL (and probably Ubuntu as well)

sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py
python2 -m pip install lz4

You can now start ziso.py with python2 ziso.py For easy transcoding I wrote a quick one-liner called ziso.sh: python2 ziso.py -c16 -a2 "${1}" "${1%.*}.zso" Now you can transcode by simply typing: ./ziso.sh Tekken 5.iso for example

JoseAaronLopezGarcia commented 2 years ago

I've updated the release (https://github.com/JoseAaronLopezGarcia/Open-PS2-Loader-ZSO/releases/tag/r111) with a new build that eliminates the 4GB limitation. All ZSO files of all sizes should work now.

INDRAPhilip commented 2 years ago

@mrchrisster I do not understand why is giving this? python ./ziso.sh -c16 -a2 Grand Theft Auto San Andreas.iso python ./ziso.py -c16 -a2 ${Grand Theft Auto San Andreas.iso} ${A}.zso ziso-python 1.0 by Virtuous Flame Can't open Grand Can't open {Grand

JoseAaronLopezGarcia commented 2 years ago

@mrchrisster I do not understand why is giving this? python ./ziso.sh -c16 -a2 Grand Theft Auto San Andreas.iso python ./ziso.py -c16 -a2 ${Grand Theft Auto San Andreas.iso} ${A}.zso ziso-python 1.0 by Virtuous Flame Can't open Grand Can't open {Grand

just do this: python ziso.py -c 9 "Grand Theft Auto San Andreas.iso" "Grand Theft Auto San Andreas.zso"

mrchrisster commented 2 years ago

You need to quote the filename like so "Grand Theft Auto San Andreas.iso"

orcutt989 commented 2 years ago

python ziso.py -c 9 "Midnight Club 3 - DUB Edition Remix (1.00).iso" "Test.zso"
  File "C:\Users\user\Desktop\OPL_ZSO_CACHE\ziso.py", line 292
    except GetoptError, err:

hardlevel commented 2 years ago

hi, any chances to bring this zso support to opl 1.0 or 1.1.0?

I have problems with the 1.2.0 betas, I can´t use any device in USB that I already use in 1.1.0 on 1.2.0, even eth isn´t working

ps2homebrew / Open-PS2-Loader

[FR] Support for cso/gzip/chd compressed ISOs #225