Poor performance on 7z with many files

[VibeNL]

When loading a cached 7z file with many small files (i.e. a Wii U game), the plugin takes a lot of time (and CPU resources) before launching the emulator. The taskmanager shows the CPU spiking for quite some time.

[fraganator]

Hi @VibeNL, thanks for raising the issue.

Could you please try starting a Wii U game, then exit it, then open the config window (under Tools -> Archive Cache Manager) and take a screenshot and post it here? I'm interested to see how big the extracted Wii U game is, and the configured cache size.

If you check the Keep checkbox in the config window next to the Wii U game you just launched, then close the config window and launch the game again, is it any quicker?

I'm not too familiar with Wii U emulation - how many files are in a typical 7z archive?

[VibeNL]

I suspect the problem lies in the discovery what file to launch :)

Setting the keep flag doesn't seem to make a difference. I have attached screens for the size and type of content

[fraganator]

Great, thanks for that. I think you're right in the file type check when launching - I hadn't anticipated archives with more than a few hundred files, certainly not 22 thousand! I'll try replicate the issue here and track down the bottleneck.

[VibeNL]

@fraganator If you need some help, let me know.

[fraganator]

@VibeNL Could you please try the attached build and let me know if it's any faster?

ArchiveCacheManager.v2.0.8.debug.zip

[VibeNL]

Ran a test with the new version:

2022-01-06 21:26:28 - -------- XENOBLADE CHRONICLES X -------- 2022-01-06 21:26:28 - Preparing cache for Xenoblade Chronicles X (Nintendo Wii U) running with Cemu. 2022-01-06 21:26:31 - ======== 2022-01-06 21:26:31 - Archive Cache Manager started with arguments: x XXX\Xenoblade Chronicles X [00050000101C4D00].7z -oG:\LaunchBox\ThirdParty\7-Zip\Temp -y -aoa -bsp1 -spe 2022-01-06 21:48:48 - Archive path set to "XXX\Xenoblade Chronicles X [00050000101C4D00].7z". 2022-01-06 21:48:48 - Archive cache path set to "G:\LaunchBox\ArchiveCache\Xenoblade Chronicles X [00050000101C4D00].7z - 0CED0AD11088586510290A1857056409". 2022-01-06 21:48:48 - Decompressed archive size is 20641126082 bytes. 2022-01-06 21:48:48 - Running in extraction mode. 2022-01-06 21:48:48 - Archive found in cache, bypassing extraction. 2022-01-06 21:48:50 - ======== 2022-01-06 21:48:50 - Archive Cache Manager started with arguments: l XXX\Xenoblade Chronicles X [00050000101C4D00].7z -slt 2022-01-06 22:10:58 - Archive path set to "XXX\Xenoblade Chronicles X [00050000101C4D00].7z". 2022-01-06 22:10:58 - Archive cache path set to "G:\LaunchBox\ArchiveCache\Xenoblade Chronicles X [00050000101C4D00].7z - 0CED0AD11088586510290A1857056409". 2022-01-06 22:10:58 - Decompressed archive size is 20641126082 bytes. 2022-01-06 22:10:58 - Running in list mode. 2022-01-06 22:10:58 - Game started, cleaning up 7-Zip folder.

Not sure it is was faster, but slow anyways :)

[fraganator]

Thanks for the log snippet, I can see a very long wait between two entries which I believe is related to calculating the decompressed archive size. I'll see if it can't be done in a more efficient way, and bypassed if it isn't needed (already cached).

[fraganator]

Hi @VibeNL. I did a bit of profiling with a dummy 7z with 22000 files, and identified a number of bottlenecks. So this version should be much faster. The decompressed size calc has been improved, and is bypassed completely if the archive is already cached. When the dummy file is already cached it launches almost immediately. The extract time is what it is, but the size calc is much faster, so overall should be quicker. Please test it out and let me know if it helps.

ArchiveCacheManager.v2.0.8.debug1.zip

[fraganator]

Another small update - this should boost speed when there are many files that match the file extension priority. Please test this one instead :smiley:

ArchiveCacheManager.v2.0.8.debug2.zip

(edit: uploaded correct build)

[VibeNL]

@fraganator , will check it asap. Probably will take some time before I'll sit behind the pc again 😅 (sometime next week)

[VibeNL]

@fraganator, seems to completely fix the performance. Starts up extremely fast after it was cached :)

2022-01-08 10:45:27 - -------- XENOBLADE CHRONICLES X -------- 2022-01-08 10:45:27 - Preparing cache for Xenoblade Chronicles X (Nintendo Wii U) running with Cemu. 2022-01-08 10:45:27 - ======== 2022-01-08 10:45:27 - Archive Cache Manager started with arguments: x XXX\Xenoblade Chronicles X [00050000101C4D00].7z -oG:\LaunchBox\ThirdParty\7-Zip\Temp -y -aoa -bsp1 -spe 2022-01-08 10:45:27 - Archive path set to "XXX\Xenoblade Chronicles X [00050000101C4D00].7z". 2022-01-08 10:45:27 - Archive cache path set to "G:\LaunchBox\ArchiveCache\Xenoblade Chronicles X [00050000101C4D00].7z - 0CED0AD11088586510290A1857056409". 2022-01-08 10:45:27 - Running in extraction mode. 2022-01-08 10:45:27 - Archive found in cache, bypassing extraction. 2022-01-08 10:45:27 - ======== 2022-01-08 10:45:27 - Archive Cache Manager started with arguments: l XXX\Xenoblade Chronicles X [00050000101C4D00].7z -slt 2022-01-08 10:45:27 - Archive path set to "XXX\Xenoblade Chronicles X [00050000101C4D00].7z". 2022-01-08 10:45:27 - Archive cache path set to "G:\LaunchBox\ArchiveCache\Xenoblade Chronicles X [00050000101C4D00].7z - 0CED0AD11088586510290A1857056409". 2022-01-08 10:45:27 - Running in list mode. 2022-01-08 10:45:27 - Game started, cleaning up 7-Zip folder.

[fraganator]

Glad to hear it's much faster now. Thanks for raising the issue, and helping test the changes. I'll close the issue now, but please raise a new issue if you run into any other problems. I'll include this change in the next official release, hopefully in the next few days pending some other changes. Thanks again.