installing apps to wii u seems very slow compared to switch counterpart

fortheusers / hb-appstore

Homebrew App Store - GUI for downloading/managing homebrew apps

https://hb-app.store

GNU General Public License v3.0

1.02k stars 60 forks source link

installing apps to wii u seems very slow compared to switch counterpart #78

Closed thedig123 closed 1 year ago

thedig123 commented 2 years ago

I am on hb appstore 2.2 and I am installing retroarch. I know it mentions it is a large download I did manage to download it pretty quickly but the install seems to just be crawling, Is this normal for the Wii U? I mean retroarch is only 27 MB surely it shouldnt take that long (currently not even half way after 30 minutes) granted I am extremely new to Wii U modding but it seems like it is chugging slowing, I love the app but would it be easier to FTP into the wii U and copy the files fore the other parts. My SD card is a samsung evo plus U3, maybe that is the issue. sorry for making an issue about this

Masamune3210 commented 2 years ago

Its zipped and has a lot of small files, it might just be easier to download and extract it on a computer and FTP the files over

vgmoose commented 2 years ago

As you noted, the download is faster, but then the install is where it actually goes and pulls each file out of the downloaded zip. Retroarch has 740 total files, and each file seems like it has a lot of overhead. It's like this on Switch as well (1655 files) but not nearly as slow as described here.

There is a print for every extracted file, would you be able to try this fix that silences that print: artifacts-7.zip (built from gitlab)

If it's not the print slowing it down, then likely there's some other wiiu-specific slow down with the zip extraction logic that becomes an issue when it's called repeatedly. (Kind of like an n+1 query, the overhead for each file within the zip being extracted being too great compared to just copying in the file like in FTP)

thedig123 commented 2 years ago

Its zipped and has a lot of small files, it might just be easier to download and extract it on a computer and FTP the files over

yeah i did that with more ra cores and the arcade cores

vgmoose commented 2 years ago

This may be a result of reading-from-SD (from the newly downloaded zip) and then writing-to-SD (for the extracted file) too rapidly without a large enough buffer in between.

hb-appstore uses libget's ZipUtil for extraction: https://github.com/fortheusers/libget/blob/master/src/ZipUtil.cpp#L297-L298

With a relatively small fixed buffer size for going in between extract and write. The problem is then exaggerated by doing this for each file in the zip as well.

Some options: load entire / larger chunks of the zip into memory as a whole, or increase buffer size within the Extract function in ZipUtil, and see how it affects performance

thedig123 commented 2 years ago

This may be a result of reading-from-SD (from the newly downloaded zip) and then writing-to-SD (for the extracted file) too rapidly without a large enough buffer in between.

hb-appstore uses libget's ZipUtil for extraction: https://github.com/fortheusers/libget/blob/master/src/ZipUtil.cpp#L297-L298

With a relatively small fixed buffer size for going in between extract and write. The problem is then exaggerated by doing this for each file in the zip as well.

Some options: load entire / larger chunks of the zip into memory as a whole, or increase buffer size within the Extract function in ZipUtil, and see how it affects performance

apologies for the late response I didn't see the message and truthfully forgot about this issue, I will check out those options how can I do those options sorry I am a bit new to this thank you so much for the response

vgmoose commented 1 year ago

I've been experimenting with a larger buffer size, and still the extracting is painfully slow. Since we have a fair amount of memory on Wii U available, going to try a change that loads the entire zip into ram first before extracting.

However, if the issue is related to just the cost of file creation on wiiu, this may not be fixable with the current state of the toolchain. For reference, downloading Opensupaplex (~100 files) has the following extraction times on different platforms:

PC: immediately extracts
Switch: around 20 seconds
Wii U: around 5 minutes

vgmoose commented 1 year ago

Should be resolved in 2.3 with unzip changes, see https://github.com/fortheusers/hb-appstore/issues/89#issuecomment-1314091767 for an explanation.

vgmoose commented 1 year ago

Fixed in 2.3.1