Endless "Arcade ROMs Database" loop when archive.org is down or not accesible

[javier-rodas]

After running just the "Arcade ROMs Database" sequence, it correctly downloads the games database and then outputs a lot of "No changes: games/mame/xxxxx.zip" lines, next it says "Downloading 76 files:" (in my particular setup) + shows 20 ZIP files + starts a sequence of 130 "*" , next it shows another 20 ZIP files and a sequence of 40 "\~" + 129 "*", next it shows another 20 ZIP files and a sequence of 40 "\~" + 129 "*", next another 16 ZIP files and a sequence of 40 "\~" + 129 "*" + 32 "\~"... and the problem happens here... It starts again from the beginning saying "Downloading 76 files:" + shows the first 20 ZIP files again, etc. in an ENDLESS loop (no matter how many hours you let it running, it won't stop).

I have realised that I cannot access "archive.org" (in particular) from my local network, and that maybe this is causing this endless loop. I really can access archive.org from my cellular network (from my smartphone) and from the Tor network too, so it's not that archive.org is down, it's simply inaccessible from my current ISP by cable.

So, maybe you could check if the archive.org is responding to ping or if the first ZIP file is not downloaded (with a timeout) to cancel the whole "Arcade ROMs Database" sequence to avoid letting users running update_all forever in an endless loop (at this particular section of the update_all)... and specially avoid the users to cold reset their mister in the middle of this endless loop (quite dangerous for the MicroSD card) :/

With check-host.net I can verify the ping to archive.org is working. Btw, the IP is being resolved as 207.241.224.2 from my local network (ie from the mister itself), so the DNS resolve is working but the ping itself is not working:

/root# ping archive.org
ping: socket: Address family not supported by protocol
PING archive.org (207.241.224.2) 56(84) bytes of data.
^C
--- archive.org ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5119ms

/root# 

Hope this gives you a better idea on how to reproduce this issue at your network and avoid this endless loop when this happens.

Good luck.

[theypsilon]

Hey @javier-rodas, thanks for your detailed report. I took strong measures to avoid any kind of "endless loop", and attempts are currently limited to 4 for each target file. This is a parameter users can tweak by editing downloader.ini, as it's shown in the README.

For your particular case, it's likely that retries took a very long time, and you got tired of it and thought it would be endless, but it wasn't. I can recommend you tweak the downloader.ini to better fit your expectations as previously described. And disabling that DB, and the BIOS one if that page is not working for you.

I don't think canceling a DB after one single attempt is a great idea, since network connections are often flaky for many users with subpar network setups and retries help with that.

[javier-rodas]

Ok Jose, thanks for your answer!. I didn't know you have added the downloader_retries parameter... but by default, the first time you launch the Update_All and the downloader.ini file is created it doesn't add such downloader_retries=3 (as shown in your README, or any other number) to the new created downloader.ini file. In my case, after the sequence explained in my first post here, the following file was created (without any extra parameters).

/root# cat /media/fat/downloader.ini 
[arcade_roms_db]
db_url = https://raw.githubusercontent.com/theypsilon/ArcadeROMsDB_MiSTer/db/arcade_roms_db.json.zip

/root# 

Wondering if in this case of not having an specific downloader_retries set to a specific value, if it would be limited to 4 retries or any other value... The other day I think I left it for several hours and that's why I thought it was in an endless loop (each loop trying to download 76x ZIP files as explained above, took maybe 30min? to complete, but maybe I'm wrong :/. Next time it happens I will try to count the number of retries.

Thanks again for this great tool!. Have a nice day.

[theypsilon]

That behavior is normal. The default value is used when the parameter is not in the INI, so there is no need to clutter the INI file with default parameters. If the numbers don't add up the next time you run it, get the log at Scripts/.config/downloader/downloader.log and share it here, please. We'll see if anything is going wrong then. But so far, it doesn't sound like that.

[javier-rodas]

Ok Jose!. Thanks for your tips.