AnimeWwise v2 leaks memory (RAM) until the total RAM in the computer reaches capacity, then the program bottoms out in performance after Windows forcibly corrects the RAM usage

[Boyishdude]

I installed this new version of AnimeWwise, v2 (after having installed and used the older version the other day), by following the instructions, but once I hit "Extract All" in the last step of using the application, for reasons unbeknownst to me, the program just started leaking memory, and it kept doing so until the RAM usage hit 100% (according to Task Manager). Once that happened, Windows corrected the RAM usage back down to something reasonable (~38-40% or so), and at the same time AnimeWwise's Disk usage went down from ~124+ MBs/s to 0 MBs/s (according to Task Manager) while it continued to extract the files .pck files from Genshin Impact (that I put in a specific folder just outside of the app's main directory on my external HDD), but at a considerably slower pace. This isn't just a one-off either, because it's happened both times I've used the program today.

Unfortunately, I don't know the exact reason for this issue, since nothing seemed out of the ordinary in Task Manager in terms of how much memory was being used by the system aside from my actual RAM Usage percentage at the very top skyrocketing out of nowhere, but I can say with confidence that while Python itself isn't using much memory according to TM, your program is definitely not optimized from what I can tell, which Resource Monitor makes abundantly clear.

To make troubleshooting this issue a bit easier, here's a rough idea of the hardware specs of my laptop, as well as the operating system:

Model: AsusTek ROG Zephyrus G14 GA401QM (I bought it at Best Buy back in 2021) OS: Windows 11 CPU: AMD Ryzen 5900 HS GPU: Nvidia GeForce RTX 3060 (Laptop GPU) RAM: 16 GBs VRAM (GPU): 6 GBs (Dedicated), 13.7 GBs (Total) Storage: 1 TB SSD (adding to this, as mentioned above, I was extracting the files to an external HDD, which I have plugged into the laptop directly via USB. It has 5 TBs of storage, and the directories for AnimeWwise v2, as well as the abandoned version of it, are also on this external HDD)

Until such a time as this issue is fixed, I would definitely recommend updating the installation/usage instructions to mention in no uncertain terms that this is a problem so that people are aware of it. Of course, I suppose it's technically possible that there's something on my end that's somehow causing the issue, but I didn't get any errors when I installed the application's requirements with pip, and I doubt that the other stuff I have installed on my system via Python (RVC WebUIs and TTS WebUIs) would be causing conflicts with an extraction program such as this. However, if there is something specific on my system that's causing the problem, there's no way for me to tell. All I know is that Python was using entirely too much Physical Memory, which I could see plain as day in Resource Monitor, which I always have open (in addition to always having Task Manager open) because I don't trust Windows, and I never will after all of the stupid nonsense Windows 10 put me through ~4-5 years ago that made me abandon it on my tower in favor of using Linux.

I like the fact that this app will let me extract audio from the game going forward (I happen to need a lot of the game's audio files for a project I'm working on, preferably in WAVE format specifically), but I don't like how it just.. eats memory. That's not fun, nor is it okay. The extraction still works properly (albeit at a much slower pace) even after the memory leak gets corrected by Windows from what I've been able to tell, but that's not an excuse. RAM usage hitting 100% at all can often have significant negative consequences for the end user, so it's better to be safe than sorry. Whatever your otherwise very useful program is doing that's causing it to eat memory like this, it needs to be fixed as soon as possible. I don't know what hardware you have that made it run so well for you, but you should definitely try to optimize the application for hardware that's on the lower-end compared to yours, laptops included.

[Escartem]

Hi, thanks for this very specific report, needless to say my intention was not to make a program causing memory leaks 😵 and since I have 50gb of ram I probably didn't noticed the ram issues anyway

Now based on how the tools works; the only two moments it uses ram is when loading a mapping, and extracting the files (at which point it loads every single .pck to extract audio from them) but if the users decides to load the entire game, I don't want the app to loads 16gb of audio into memory so I've made a specific library for this so it buffers the files instead of loading everything into ram, so my guess is that it's not working properly and still tries to load everything into ram. I did some testing and indeed while in task manager the app used no more than 400mb in resource monitor it was ~15gb (and since I still had many gb left I couldn't notice anything) As for disk usage, apart from extracting it can also use a lot of it if you use .hdiff files, since it stores a copy of every patched pck in a temporary folder until the app is closed

So anyway I'll try to work on a fix and keep your updated 👍

[Escartem]

After some fixes, i think i have solved the issue in https://github.com/Escartem/AnimeWwise/commit/85e7430f1076f2fc96629b6f9880606b50266d4b, loaded the entire game, extracted everything, tool did not exceeded 600mb of ram for both task manager and resource monitor, I'll let you test on your side and tell me if it works too

[Boyishdude]

Do I have to re-download the entire archive and paste it over the existing directory to get the update? I'm not really familiar with the knowhow for Github and related things. In any case, trying it again with the updated version will have to wait until later. I'm in the middle of extracting the rest of the .pck files from my copy of the game, so I want that to finish first (speaking of, a nice QoL feature would probably be the option to stop/cancel extraction, as well as loading the files in the earlier steps, within the app itself. Closing the app while it's extracting makes Python burp and give me a weird list of errors in Command Prompt that don't make sense to me).

[Escartem]

If you got the app using git clone, you can do git pull origin v2 (i think) to get changes, if you downloaded as zip you need to redownload the zip again, I'll work on a releases system later. As for your requests, I am aware closing the the app while extracting makes it panic a lot, I'll try to work on all these little details later

[Boyishdude]

I'm going to try it again on Star Rail using the latest version of your program once it's finished being installed to my external HDD (figure I might as well strike while the iron's hot in terms of getting audio files I'll "need"), but I was wondering if you have plans to update the mappings for Star Rail, as well as add Honkai Impact 3rd to your program? For my use case, though I've never played the game, HI3 having been around for so long surely means that it's got tons of audio files that are ripe for my use case, but not being able to extract them into WAVE format personally is a bit of a bother.

I also wanted to know if there's a separate tool on Github somewhere that's basically the same as your program, but it can be used to extract the cutscenes from both permanent content and limited-time events specifically. I'd like to be able to get Genshin Impact's FMVs myself, if I can.

Edit: The tool is working as it should now! I didn't have any memory problems extracting the Star Rail audio files, though I do wish that Star Rail was like Genshin Impact in that you can extract ALL of the audio files rather than just some of the dialogue.

[Escartem]

Glad to know it works, for star rail I haven't made the mapping data myself and haven't got the time to dig into it, It'll be updated at some point, and honkai 3rd it's the same story, I need to dig through the dumps and figure out the keys and names which is planned but not my priority. As for the cutscenes question you asked, there are programs for genshin and dumps I've uploaded myself here but for star rail I do not know, I assume they use the same system but this system requires keys to decode the cutscenes which are annoying to get. It probably needs a bit of search but I'm sure people have already uploaded those keys and I do plan on uploading hsr cutscenes myself at some point too

[Boyishdude]

I assume that the "keys" for Genshin Impact in your program are incomplete, too? I noticed while organizing all of the Paimon dialogue over the past day that while I managed to extract and convert all the dialogue from multiple limited time events in 3.x and 4.x, I wasn't able to extract and convert all of them, and none of the dialogue from limited-time events is from versions of the game prior to 3.x, which is a little annoying to me. I would've liked to have the dialogue from stuff like Irreconciled Stars and other events I never got to play.

[Escartem]

Absolutely, one key is one file, there are about 500k+ keys inside the Genshin mapping which is very good but far for an entire coverage that would be around 800 if not a million. Sadly the keys are only obtainable from what we can extract from dumps, it's better than nothing at least and depends on the games (zzz is around 100%). As for time limited events the keys were there but get deleted over time once the events are done and the mapping are for the game at that version. It would be possible to dump older version of the games but because of many dmca takedowns only recents version are available.

However it's not because the file is not in the list that's you can't get it, it just doesn't have a name and you'll need to dig through the unmapped folder