search

RainbowCookie32 / rusty-psn

A GUI/CLI tool for downloading PS3 and PS4 game updates
MIT License
379 stars 17 forks source link

Multipart PS4 packages merging in egui release #256

Closed sarpt closed 1 month ago

sarpt commented 1 month ago

Since PS4 packages are split into parts it might be necessary for them to be merged into a single file depending on where they are used. There are already tools available that merge the part packages into one usable package (for example https://github.com/Tustin/pkg-merge), so I've thought that rusty-psn can also have this feautre added so the user doesn't have to switch tools.

mpv-shot0001

This PR is only a bare minimum necessary to support this feature. As such this PR only adds support for parts merging in egui version of the tool since this PR already got quite big and I have to ponder a bit how to add it to cli release in such a way that it's a good user experience there. Currently merging also only creates a new file on disk - ideally I would like there to be an option to create the merged version in place by merging N-1 parts into the first part and getting rid of already merged parts, so the disk usage is reduced to minimum, but I plan for that to be a future improvement.

In case an user decides to merge files again on already merged parts, the merge process will start from the beginning and will start overwriting already merged package according to pkg offset values - as such it's non-destructive and it's not prohibited since the same bytes will be written at the same offsets. One of the next improvements planned is to use already merged package for a "verifying" phase when "download all" is clicked, so the parts that are already merged will not be downloaded again, just their hashes checked according to offsets on a merged package, and the whole merging again part will be skipped too.

After merging the merged file is in the same path on disk where part packages are, but it's filename is stripped of part parts, so for example for part file EP9000-CUSA02171_00-0011223344556677-A0117-V0100_0.pkg the merged filename will be EP9000-CUSA02171_00-0011223344556677-A0117-V0100.pkg.

A video presentation of the process:

https://github.com/user-attachments/assets/aa307763-8064-41b7-b386-1566481172ef

I cross-referenced sha256 hashes of output files with merged files using mentioned above pkg-merge tool and the examples I checked seemed to be exactly the same.

RainbowCookie32 commented 1 month ago

Hello there, thanks for the PR! I intended to test and review today, but I'm kinda short on sleep and I don't trust my capabilities right now lol. I'll take a look tomorrow for sure

RainbowCookie32 commented 1 month ago

I have to ponder a bit how to add it to cli release in such a way that it's a good user experience there.

Maybe a --merge-parts flag that merges everything when needed.

Currently merging also only creates a new file on disk - ideally I would like there to be an option to create the merged version in place by merging N-1 parts into the first part and getting rid of already merged parts, so the disk usage is reduced to minimum, but I plan for that to be a future improvement.

I think you should be able to do this by opening the first file, resizing it with set_len to the final size, and then using seek to move to the end of the first file's bytes and start writing the contents of the others, in order.

sarpt commented 1 month ago

I have to ponder a bit how to add it to cli release in such a way that it's a good user experience there.

Maybe a --merge-parts flag that merges everything when needed.

That's a good idea. I guess if the flag is provided but there's nothing really to merge (only single-part PS4 updates or PS3 updates downloaded) there could be a warning. There's also a case when not all parts are selected for download for a title (for some unknown reason) which could also probably be solved with a warning of some kind, or an error but the error would break execution of the rest of downloads & merges so not sure which would be better.

Currently merging also only creates a new file on disk - ideally I would like there to be an option to create the merged version in place by merging N-1 parts into the first part and getting rid of already merged parts, so the disk usage is reduced to minimum, but I plan for that to be a future improvement.

I think you should be able to do this by opening the first file, resizing it with set_len to the final size, and then using seek to move to the end of the first file's bytes and start writing the contents of the others, in order.

Not sure how to present it in GUI though (in CLI probably with another flag) - it probably would have to be left as an option to the user, since some might not want to merge the files and some couldn't care less. Most users probably would prefer to have as default merging of files in place and removal of unneeded parts, since most users probably don't care that there are parts involved. That also would suggest that hash verification on merged file should be implemented beforehand since otherwise the application would remove files and with each download would waste network resources when not really necessary, since the merged file will be there but unchecked for presence before download. I will probably attempt that after adding merging support to CLI.