search

cheahjs / palworld-save-tools

Tools for converting Palworld .sav files to JSON and back
MIT License
795 stars 71 forks source link

[Suggestion] Refactor in low-level language #83

Closed nickheyer closed 8 months ago

nickheyer commented 9 months ago

I appreciate the clear amount of hard work you've put into this.

As worlds/files grow and your tool is required to parse more data, I think you will (or already do) find the overhead from Python's ability to parse binary to be overwhelming.

I suggest reading this little article on Python's performance when "mucking about in a binary file".

You may even find that you are writing less code as Python's abstractions are no longer as helpful when working with raw data types like uint32, double, etc.

Another plus is that you could deploy actual executables in your releases instead of python scripts, which ultimately would be less user-error-prone - ie: expecting a user to have Python installed and available in their $PATH.

This is just my opinion from the outside looking in (after briefly looking at your parsing script), and I could be totally incorrect in my assumption that refactoring would be a good solution for you. Take it with a grain of salt.

Thanks again!

cheahjs commented 9 months ago

I'm happy for other people to port over changes to something else, but it isn't something I am going to do myself. See existing projects:

There's always going to be some overhead compared to Unreal, as Unreal gets to decode into game objects directly, but external tools rely on the self-describing nature of Unreal's FArchive UObject serialization (which we have to track into order to perform inverse operations). There are several things that could be done to try and shave down memory use (eg stop using dicts and use classes with slots), but my main priority right now is correctness of parsing, and completely parsing all RawData fields.

Python as raw scripts was a specific choice for a few reasons.

  1. I'm not going to get tools and people screaming I'm distributing malware (see what has happened with https://github.com/EternalWraith/PalEdit) when it's all readable text files.
  2. I believe it's easier for people to hack save editing on top of the Python scripts, rather than have people faff about with build tools for a given language.
nickheyer commented 9 months ago

For the record, I think this is a very large undertaking - especially staying current with all incoming changes, and you seem to be managing it quite well with Python.

Thanks for the tool!

Dreded commented 9 months ago

also for the record there is nothing stopping them from packaging it as a exe using pyinstaller... takes 5 seconds

to the point of refacturing it is there any reading on how these sav files are structured? How did you learn to convert them to josn etc.. wouldn't mind looking into if this could be written in C++ as a learning tool

nickheyer commented 9 months ago

also for the record there is nothing stopping them from packaging it as a exe using pyinstaller... takes 5 seconds

to the point of refacturing it is there any reading on how these sav files are structured? How did you learn to convert them to josn etc.. wouldn't mind looking into if this could be written in C++ as a learning tool

Pyinstaller is not really a great way to deploy open source applications, or really any applications. It is quick and easy, but running an exe made from pyinstaller on any windows system with defender enabled (very difficult to actually disable) will often get immediately flagged as a PuP or Trojan Malware. This is due to multiple factors, but mostly because of it's inherent lack of code signing/cert + usage of reserved system resources, as well as the unfortunate reality that malware is really easy to write in python. I personally have tried and learned not to deploy via pyToEXE/Pyinstaller/etc.

For the documentation of the sav struct, im sure some googling would lead you to the write spot. It is apparently non-proprietary and freely available info - which is awesome!

Dreded commented 9 months ago

Yea, unfortunately damned if you do, damned if you don't... Windows users are generally bad at using python... i prefer github but one thing I like about gitlab is that you can have gitlab compile your releases so people can confirm the code in them is the code in the repository.

I personally have never had a compiled exe from pyinstaller flag a AV, but I know it happens. Probably depends on what packages it uses.

nickheyer commented 8 months ago

Yea, unfortunately damned if you do, damned if you don't... Windows users are generally bad at using python... i prefer github but one thing I like about gitlab is that you can have gitlab compile your releases so people can confirm the code in them is the code in the repository.

I personally have never had a compiled exe from pyinstaller flag a AV, but I know it happens. Probably depends on what packages it uses.

Github offers a similar service called Github actions, though it may differ as I haven't used gitlab automation workflows.