Option to use a temporary folder to download stuff

[nedex]

Hello, This is a feature more than a bug, I hope it may benefit everyone.

Each time the utility downloads a file, it is stored on the microsd (whose speed is limited). Then an extraction is performed at the "same time" as writing, virtually making it even slower.

It may be useful to have a setting to choose where temporarily download files should go (or just use the temporary folder of the OS) in order to only write to the microsd the useful stuff.

[lunchpacked130]

Firstly, you don't mention what card you're using and there are massively different speeds memory cards can have, some are sirupy slow, typically 10mb/s absolute max, (unless your using 2 or 4 gb cards or older cards under 32gb which can be even slower) while others cards can reach upto around 10x that and above, and some cards are made for different purposes, uhs, rating etc being better suited for file transfers etc (for instance using a sd card as a extra drive on a pc)

Secondly, you don't mention what card reader you're using or if you're connecting directly through the Analogue Pocket, this also matters because the speed of the reader you use for transferring files/running pupdate, can vary alot.

For instance, using the pocket itself as a card reader is slower in my experience (peaks of 10-12mb/s) vs using the built in usb2 card reader on my laptop (usb2 tops out at about 30mb/s, but more typical is 15-20-ish mb/s), and my usb-C 3.1 gives me peaks between 90-120mb/s

And lastly, download speeds in pupdate are very slow, typically way under 1mb/s.. consider this, 2 days ago, I noticed i hadn't disabled the "Skip alternative ROMs" option, but downloading all those roms took basically a whole day to finish, (atleast 8+ hours)

I guess as an alternative, you could run pupdate from a local drive in windows, it should recreate the card setup, from the folder its in, then you can just copy it or the parts you need over to the card, that would remove double duty during extraction for the sd card, but considering all other factors, i think that what time you would gain from that is more or less negligible.

Also, i might be totally wrong about this, and do correct me if that's the case; but i thought extractions programs typically use the OS system temp folder or similar for extractions 🤷🏻‍♂️

Tl:dr: If download speeds are under 1mb/s anyway, any extraction time saved is negligible..

I.e. if it takes 150 seconds to to download compressed file, and it takes 15 seconds to extract directly on the card, or 10 seconds on a harddrive, would you even notice the difference of saving 1/3rd(5s/15s) of 1/10th (15s /150s) of the total time each file uses? Or if 200 files takes 8hrs to download, you would save about 15-16 minutes in total.

[lunchpacked130]

One thing that could become an issue for some, is the extra space used by the conpressed file during compression, if the compressed file size is over 50% of the cards free size, you would most likely get out of space errors.

However, since we're not downloading compressed PCE CD disc images, and the largest compressed files for the AP are maybe 75-100mb, heck, let's say 150mb for good measure. Those with less than 300mb free space, could experience not enough space errors, though it's a kind of fringe case scenario.

However, adding i.e. a setting where you can set a download location for installation files would help with this, but you'd probably have to manually type the exact path to your dl location folder, and manually extract and copy over the files to the sd card afterwards.

Implementing a way to browse for download location isn't feasible i think, and having the pupdate script update its path for every compressed file it needs to extract, is not something i think is tricial, and probably need extensive rewrites, or atleast extra work for a select few instead of prioritising thay work to more needed and pressing issues

[nedex]

Hi @lunchpacked130, I assumed you were one of the contributors of this project or at least a contributor to any GitHub repository, but nonetheless I will try to answer your concerns.

Both in the worst case scenarios (connecting the pocket with the slowest sd card and the worst internet connection) and the best case (having a usb reader with the fastest sd card and the best internet connection), using a temporary folder would be beneficial, there's no need to bring up numbers.

Surely the "slow users" would have more benefits, but even "fast users" with less write operations on the sd card, that for example could shorten their lifetime.

Also, this would be an additional option, not something you would be forced to use, so not having enough space to use on the OS is not a concern for everybody. Neither having to "browse for download location", since there are many manual settings in the config file and this might be one of them (for example backup_saves_location).

having the pupdate script update its path for every compressed file it needs to extract, is not something i think is tricial, and probably need extensive rewrites

There are 8 matches where the temp folder is used and it's also redundant code: creating a new function to handle the extractions may also lead to better code.

[mattpannella]

I recommend never using the usb connection on the pocket itself. it's painfully slow (and also prone to errors in my experience) for anything larger than a couple mb

having a configurable temp location could be useful, I'll look into implementing it I've been pretty busy lately so I haven't had much time to work on this, but I'll keep this in mind

[hallem]

And lastly, download speeds in pupdate are very slow, typically way under 1mb/s.. @lunchpacked130 Pupdate isn't restricting the speed at which things get downloaded. Primarily that is due to the download site, archive.org, being extremely slow at times and just generally slow at others. There is a feature built into pupdate to use a custom archive to download the files from. If you're having that much trouble with speed, it should improve some using that feature.

@nedex As for the recommendation to use a temporary folder on the computer to download and decompress files to rather than the card, I'm not 100% sure that it will make any real difference to be honest. I've done a lot of testing running pupdate locally on my computer and haven't seen any major speed improvements when it comes to downloading and decompressing files. What I have seen happen faster is when assets are already downloaded, checking for their existence and verifying the checksum happens a lot faster locally than on the SD card.

I'll work with @mattpannella to see if this is something we feel is something to add and if so, see about getting it in. Both of us have been pretty slammed with life stuff so I can't promise it will happen quickly.

[nedex]

If you already have an idea on how to implement this, I could try to help you out. As I said in the previous message, there are 8 places where the temp folder is created. My idea was to create two functions, one to extract to a temporary folder and another one deleting it (with a variable referencing it): these functions would make use of the user's preference in the settings (whose name / value should also be discussed).

[mattpannella]

check v3.10.0

[nedex]

That's insane, I will try it ASAP. Thanks for the feature!