Download improvements

[Th3Kacper]

It would be great if the download status could be monitored somewhere. It would be useful to see how much data has already been downloaded and the current transfer. In addition, mods could be downloaded in sequence and not all at once. This would allow for better control of what is happening. I had problems downloading when I ran all at once, and when I ran one at a time I observed that the problem was only with some mods, not all.

[fugasjunior]

Hi, thank you for the great suggestion.

I have implemented better download monitoring for servers. Now you will be able to see in which stage the installation or update is (i.e. download, validation, dry run, ...) and how much data has been downloaded or validated, as seen in the screenshot.

Unfortunately, downloading workshop mods doesn't allow the same level of granularity. SteamCMD only tells you when it starts downloading a mod and when it's done downloading the mod. As there is no download progress, I have at least implemented more states. Now you'll be able to differentiate between the following states of mod installation: 'In queue' -> 'Downloading' -> 'Pending installation' -> 'Installed'. You'll get the feedback almost immediately when the mod has finished downloading. The installation still happens after all the mods from a batch have been downloaded.

I have concerns regarding downloading mods sequentially instead of downloading in a batch. 1) Performance - starting the SteamCMD process and logging in to the user account would be required for each mod. This could drastically slow down the installation of a large number of small mods. 2) Rate limiting - I'm worried that, let's say, 100 SteamCMD logins would not be appreciated and the user would get rate limited by Steam, making the downloads impossible for a certain amount of time. However, I'm not 100% sure what are the exact conditions for the user to be rate limited. If anyone points me to the documentation, I'll be more than happy to read through it.

Given these concerns, I'm not going to make mod downloads sequential for now. However, a future enhancement could add an option to the application settings where the user could enable sequential downloads mainly for debugging purposes.

I had problems downloading when I ran all at once, and when I ran one at a time I observed that the problem was only with some mods, not all.

@Th3Kacper Could you by any chance remember what was the problem, please? Maybe the concrete problem could be taken into consideration so it wouldn't mark the whole installation batch as failed.

The changes are mention are currently WIP in the branch 122-steamcmd_download_status_improvement and will be a part of release v1.4.0.

[Th3Kacper]

Hi, Thank you for your feedback. I am glad that you liked my suggestion and that it will be implemented. I completely understand the problems with sequential mod downloads. You've done an amazing job and I hope to be able to do a donation in the near future.

I will try to describe as accurately as I am able the problems with downloading the mods. I'll start by saying that I'm not a native speaker, if anything is unclear, please let me know.

My setup: Debian 12 (Bookworm) shell + Docker engine.

I correctly configured and started Docker container by doing docker compose up. I then installed the server and uploaded the mod preset. The server was running everything was ok. I suspect the problem occurred when I started uploading the second preset of mods to the manager, then decided I didn't want the mods after all and deleted them while they were still downloading. I noticed that I had deleted more mods than I wanted, so I figured I would delete all the mods and upload the correct preset again. The preset added itself, the mods started downloading, the server space used started increasing. The download lasted a few tens of minutes and then was interrupted with an error. I do not remember the exact message. I think it said there was an unexpected or unknown error, or something like that. I then went into ssh to see what had happened and there were messages such as: Download of mod 'CBA_A3' (id 450814997) failed, reason: GENERIC.

As I tried to troubleshoot I discovered that some single mods were installing correctly. For example: ace installed correctly, CBA no longer does. If I selected both mods for installation I would get an error and neither mod would upload, even though individually ace installed correctly. I suspect it would only take one 'faulty' mod to break the installation of all the mods marked.

I did a little reasearch and regarding some other game I found a hint that something might be wrong in the files or cache or something. I decided in that case to clean all application files and docker prune. I then configured everything again and so far the problem has not recurred, and I have already uploaded a couple of presets to the server.

[Th3Kacper]

@fugasjunior It seems that my problem with downloading mods has returned. This time I managed to solve it more easily. We were changing our main modpack, so I wanted to clean it up by deleting all the mods and presets. The mods were successfully deleted, freeing up space on the server. Then I applied a new preset, mods started downloading and after about an hour it failed. There was this GENERIC error in the container. To fix it, I completely removed the "storage/mods" folder, then restarted the download and it went fine.

[fugasjunior]

I think deleting the temp and downloads directories can generally be a good idea to troubleshoot failing mod downloads. I will try to incorporate it into the GUI or at least the docs.

Some of the improvements to displaying download status were released in version v1.4.0. Feel free to open a follow-up tasks if you have any suggestions on how to improve this further. Thank you!