search

FanTranslatorsInternational / Kuriimu2

Kuriimu is a general purpose game translation project manager and toolkit for authors of fan translations and game mods.
GNU General Public License v3.0
311 stars 56 forks source link

Saving on a per file basis - SOR4 #126

Closed AyanamiRei1 closed 3 years ago

AyanamiRei1 commented 3 years ago

Well currently modding the Switch version of SOR4 is working fine however testing mods and other things like that takes far too long due the plugin saving the whole file instead of just replacing the modified files and currently the way it is isn't practical in my eyes.

I believe SOR4Explorer does instead of saving the whole file each time however even when editing the code that just spits out a compression error so that won't work.

So basically what I'm requesting is that the plugin saves on a per file basis instead of whole file every time if that's possible.

Thanks.

onepiecefreak3 commented 3 years ago

I don't understand what you mean. The plugin recreates the file based on only the changed information. If one file changes, I have to write back all other files according to the changed file (offset and size). Therefore we can't be more efficient than what we already do. Do you have a specific sample case we can test things on?

onepiecefreak3 commented 3 years ago

We only do things as necessary. We only recompress files that need it, and we write back data unchanged to the archive as much as it's possible.

AyanamiRei1 commented 3 years ago

It takes over 10 mins to save in my case I assume it's because it's repacking the whole textures file instead of just replacing the files you've modified.

onepiecefreak3 commented 3 years ago

I wouldn't know how one can only replace the files that were modified, seeing as how the files are compressed and can be anywhere in the file. "Replacing" just the modified files, will not work, since it could overwrite other files following it. So at least those have to be rewritten as well. Which takes away the benefit of "just replacing the modified files".

onepiecefreak3 commented 3 years ago

I also tried many things on the plugin now, I cannot reproduce any sort of slow behaviour. You're right that saving may take more time, but that was only in range of multiple seconds. This was due to the resaved file being moved from the temporary location to your chosen save location. Depending on your hard drive, moving a 1.7GB file may take its time then. And for me, that time was ~10-12 seconds. Otherwise I could not measure any long running or blocking tasks that would take up to minutes or even tens of minutes.

Therefore, I'm unsure what exactly it is that I should do here.

IcySon55 commented 3 years ago

@AyanamiRei1 The save mechanism for basically all archive plugins in Kuriimu 1 and 2 is to only re-compress (if the archive uses compression) files that have been replaced by the user. All other files are simply copied from the source file to the destination file.

In some cases the re-compressed file will be larger than the original that it is replacing. For this reason we cannot simply overwrite the original in-place in the existing archive file and instead create a new one. This gives our plugins maximum compatibility.

The only ways that an archive would take 10 minutes to save is if the source or destination is a very slow storage medium, like an SD card (in a LayeredFS directory ;P ), or if the file is HUGE, upwards of 15GB on an HDD and 50GB on a slow SSD.

I'm interested to know what your setup is so we can give you some pointers to speed things up. However if your target is indeed an SD card for testing on hardware, there isn't much we can do, other than risking broken archives by creating the files incorrectly.

Cheers~

AyanamiRei1 commented 3 years ago

Hello there first of all I'd like to apologize for my stupidity it was my old system I've got a new much better one I'll close this now.