SteamCMD workshop timeout

SteelPhase commented 5 years ago

Your system information

Steam client version (build number or date): latest version (no idea how to get steamcmd to tell me)
Distribution (e.g. Ubuntu): Ubuntu 19.04
Opted into Steam client beta?: No
Have you checked for system updates?: Yes

Please describe your issue in as much detail as possible:

I've seen it reported in a bunch of places, but I don't see an issue here. It appears that steamcmd struggles with downloading larger workshop items, with no specific solution.

steam@test:~$ /usr/games/steamcmd +login user +workshop_download_item 107410 583496184 validate +quit
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 176.7ms!
OK.
Logging in user 'user' to Steam Public ...
Logged in OK
Waiting for user info...OK
Downloading item 583496184 ...ERROR! Timeout downloading item 583496184../common/pipes.cpp (774) : Assertion Failed: stalled cross-thread pipe
steamclient.cpp (779) : Assertion Failed: bufRet.TellPut() == sizeof(uint8)
CWorkThreadPool::~CWorkThreadPool: work processing queue not empty: 2 items discarded.

Steps for reproducing this issue:

execute /usr/games/steamcmd +login user +workshop_download_item 107410 583496184 validate +quit
wait
should receive an error.

SteelPhase commented 5 years ago

Not sure if these will help but these are some of the links I came across.

h1z1 commented 5 years ago

What kernel / distro ?

SteelPhase commented 5 years ago

Ubuntu 19.04, was whatever the latest version was available at the time of creation.

h1z1 commented 5 years ago

What is the output of "uname -a" ?

SteelPhase commented 5 years ago

root@ubuntu-test:~# uname -a
Linux ubuntu-test 5.0.0-17-generic #18-Ubuntu SMP Tue Jun 4 15:34:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

h1z1 commented 5 years ago

Likely related to #6326

michaelsstuff commented 5 years ago

Is there any solution or workaround? I don't think #6326 is really related to it.

./steamcmd.sh +login $user "$pass" +workshop_download_item 107410 450814997 +quit
Redirecting stderr to '/home/steam/Steam/logs/stderr.txt'
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 150.4ms!
OK.
Logging in user 'myuser' to Steam Public ...
Logged in OK
Waiting for user info...FAILED. Timed out.
OK

This is on a centos7 - 3.10.0-957.21.3.el7.x86_64

This kind of breaks my update script for arma server :-(

wakeuphate commented 5 years ago

@michaelsstuff If you monitor your download folder, are you getting some of the item downloaded each time? Often SteamCMD times out on large downloads as you're aware, but if you keep retrying, it'll finally get to the end as long as you're using +validate.

It does however from your most recent log, look like it's timing out even with login details? Have you tried just typing your 2fa code into the blank space and hitting return when it's doing its "time out" part? I've had that be successful at times.

I run an ARMA server too and honestly, SteamCMD is borderline awful for us. We have to devise workarounds, scripts and more to keep re-trying SteamCMD for larger downloads. We've even considered having to remove security features on our own accounts because each time SteamCMD has to re-run, we're forced to put in our 2fa credentials each time. Plus, when you've done this 8 or 9 times, you'll get rate-restricted and have to wait another 30+ minutes until you can try the rest of your mods. I'm surprised this day and age that Valve have no way of setting up an access key for SteamCMD or something that allow us to restrict what it can do.

The fact that SteamCMD still cant take a list of mods and simply stay logged in and download them from the steam workshop in one go is very frustrating and hasn't changed in years.

michaelsstuff commented 5 years ago

@wakeuphate I can just wholeheartedly agree with everything. The mod i tried to download was CBA3 i think, so very small. And it is a timeout for the userinfo it seems. This is happens randomly throughout my testing. Sometimes is happens, sometimes not. Not sure why, or whats causing it.

But I will always get Loading Steam API...../steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (117) : Assertion Failed: SteamDatagramTransportLock held for 190.1ms

There is just no usable API for steam :-/ As a workaround, I am creating a scrpipt, that can manually be triggered to download workshop stuff and put it on a ftp server, for my actual arma server to download from there....

In the end, it looks if retry the same error happens, but it will download the stuff. Kind of pray and repeat?

SteelPhase commented 5 years ago

I can confirm that this does not appear to be related #6326 and that running the same command over and over did resolve the issue. I was using the below script workshop.sh that would loop through each workshop item, and attempt to download it. if one failed I could run workshop.sh 615007497 --only. I could also start form any point in the list by providing the last workshop item to fail workshop.sh 615007497. This worked pretty well, but was slow. In my case I only needed to authenticate once, and use SteamGuard once. For all subsequent executions it appeared to use cached authentication based on the provided user name.

/home/steam/scripts/workshop.sh

#!/usr/bin/env bash
set -e
set -o pipefail
WORKSHOP_ITEMS=(
    787892271
    463939057
    773131200
    773125288
    884966711
    708250744
    1104460924
    713709341
    615007497
    639837898
    730310357
    1537973181
    642457233
    849435425
    886857451
    820924072
    450814997
    837729515
    724064220
    871504836
    583496184
    583544987
    882231372
    686802825
    333310405
    743099837
    366425329
    433566590
    648172507
    694603075
    1224892496
    1494127420
    873999690
    520618345
    909547724
    736829758
    860158903
    823636749
    1208517358
    1400574293
    1400566170
    950966660
    735566597
    843425103
    843593391
    843632231
    843577117
    1368857262
    498740884
    699630614
    1493291920
    946763963
    894678801
    1206332670
    784218341
    949701797
    949696541
    940841212
    1103511215
)

START_FROM=0
ONLY=0

if [[ -n "$1" ]]; then
    found=0
    for workshop_item in ${WORKSHOP_ITEMS[@]}; do
        if [[ $1 -eq $workshop_item ]]; then
            found=1
            break
        fi
    done
    if [[ $found -eq 1 ]]; then
        echo "Starting from $1"
        START_FROM=$1
        if [[ $2 -eq "--only" ]]; then
            ONLY=1
        fi
    else
        echo "Provided workshop id $1 is not in list"
    fi
fi

found_start=0

for workshop_item in ${WORKSHOP_ITEMS[@]}; do
    if [[ $START_FROM -ne 0 ]]; then
        if [[ $found_start -eq 0 && $workshop_item -ne $START_FROM ]]; then
            continue
        elif [[ $workshop_item -eq $START_FROM ]]; then
            found_start=1
        fi
    fi

    /usr/games/steamcmd +login user +workshop_download_item 107410 ${workshop_item} validate +quit | egrep "^(Downloading item ${workshop_item} ...|Success. Downloaded item ${workshop_item})"
    if [[ $ONLY -eq 1 ]]; then
        break
    fi
done

michaelsstuff commented 5 years ago

Thanks to @SteelPhase I made a little something to download workshop stuff.

short version:

for workshop_item in "${WS_IDS[@]}"; do
    modname="$(curl -s https://steamcommunity.com/sharedfiles/filedetails/?id="${workshop_item}" | grep "<title>" | sed -e 's/<[^>]*>//g' | cut -d ' ' -f 4-)"
    modname_clean=$(echo "$modname" | dos2unix)
    counter=1
    printf "Downloading %s \n" "$modname_clean"
    until steamcmd +login "${STEAMUSER}" "${STEAMPASS_decrypted}" +workshop_download_item 107410 "${workshop_item}" validate +quit; do
        printf "Error Downloading %s. Will try again \n" "$modname_clean"
        counter++
        if ((counter > 4)); then
            exit 1
        fi
    done
    if [ ! -L "${home}/mods/@${modname_clean}" ]; then
        ln -s "${home}/.steam/steamapps/workshop/content/107410/${workshop_item}/" "${home}/mods/@${modname_clean}"
    fi
done

It basically tries again and again... not nice but eh

A bit more and explanation here:

https://github.com/michaelsstuff/Arma3-stuff/tree/22fde3f14590d0d52a2fb198334d250af20ebd34/mod-sync

Next step will be to upload it via ftp for other reasons... Thanks everybody for the ideas and feedback.

SteelPhase commented 5 years ago

You could probably make this script a little smarter by monitoring the locations where the files are downloaded, and the location where the completed files are stored. Monitor the file size, and retry until it completes without error, or stop when the file size decreases. I've yet to run in to an instance where just running it over and over doesn't work.

FireCulex commented 5 years ago

This has been an issue with SteamCMD across platforms for atleast 5 years.

iaidan commented 5 years ago

Any fix for this?

SteelPhase commented 5 years ago

There has been no movement on this at all.

iaidan commented 5 years ago

Only thing we've found is to do delete the .steam and Steam folders from the users directory. It has to be done for every single download, and doesn't always work.

h1z1 commented 5 years ago

@SteelPhase That kernel version would have been affected by the bug. Anyone else still getting bit by this should update their kernel, the fixes have been in most distros for a while now.

SteelPhase commented 5 years ago

I confirmed earlier that my issues weren't related to that bug. i can try and reproduce again if need be. but it can't happen tonight.

iaidan commented 5 years ago

@h1z1 I can tell you, I'm having this bug on about 30 machines, none of them are running the kernel you said and even after updating kernel it doesn't work.

Only fix that has been found is deleting .steam and Steam/ before running steamcmd every single time it needs to run. Which would indicate it's broke in a steamcmd update.

h1z1 commented 5 years ago

Not denying the message is valid rather there was a serious bug in the kernel version reported. I've run the script above for about a week out of curiosity and not hit that but then again I'm likely in a different region (NA). Can you throw up your logs anywhere?

SteelPhase commented 5 years ago

I will try and set some time aside to retest this, and get the logs. It may take me a few days.

viktor-ferenczi commented 3 years ago

It happens with Space Engineers blueprints as well.

It also happens with the Windows version of steamcmd, so this is not Linux specific.

Always works: 2306920249

Microsoft Windows [Version 10.0.19041.685]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\SEServer\steamcmd>steamcmd +"login anonymous" +"workshop_download_item 244850 2306920249" +quit
Redirecting stderr to 'C:\SEServer\steamcmd\logs\stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for user info...OK
Downloading item 2306920249 ...
Success. Downloaded item 2306920249 to "C:\SEServer\steamcmd\steamapps\workshop\content\244850\2306920249" (1056527 bytes)

Never works, even if retried many times: 358485226

C:\SEServer\steamcmd>steamcmd +"login anonymous" +"workshop_download_item 244850 358485226" +quit
Redirecting stderr to 'C:\SEServer\steamcmd\logs\stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for user info...OK
Downloading item 358485226 ...
ERROR! Download item 358485226 failed (Timeout).

Same error on Debian 10.5 Linux server. I don't repeat the output here. All are the same, only the path differs.

Nothing is logged into C:\SEServer\steamcmd\logs\stderr.txt, the file is empty!

A detailed error message why the download does not work would be highly useful.

Space Engineers (the game itself) can happily load both blueprints.

The main difference between them is that the non-working one is from 2014, while the working one is from 2019.

You got two actual examples now, this issue is quite easy to reproduce.

Come on Steam, you can fix it!

Or just give us plain HTTP(S) download URLs for the workshop files without the extra layer of difficulty.

You can cache those as well on the edge or just use a CDN. It won't be that expensive that way.

Anybody else: Could you please test the above two commands? Same behavior for you or both/none works? Thanks!

Raznic commented 3 years ago

@viktor-ferenczi I was running into the same issue setting up a Space Engineers server, but I believe I've found a solution. It seems that some mods require SteamCMD to authenticate with a Steam account. Anonymous login just doesn't work with some mods, though I'm not sure why. I setup a separate Steam account just for my dedicated servers and that seems to have worked. The timeout error is very misleading here and a better error message could probably save a lot of folks some debugging.

I don't think this solves everyone's problems though. Looking at some of the other error message, I see assert statements regarding data packets. The errors that you and I were seeing don't include any of those assert statements, so I think they're separate issues masked by the same generic error message. Hopefully this helps at least a few folks get their mods downloaded.

SteelPhase commented 3 years ago

That's definitely a good option to try, but I can say that I still ran into issues even with authentication. Not sure why it only helps sometimes either, but you may be on to something.

Raznic commented 3 years ago

One thing that I've noticed (and this aligns with Viktor's post) is that the mods that require a Steam account were last updated before or during 2018. Any mods that were updated as of 2019 or later work with anonymous login. I'm wondering if something changed in the workshop around mid to late 2018 that caused this change in behavior. Just a hunch.

viktor-ferenczi commented 3 years ago

Thank you for the hint @Raznic, I will give the secondary account a try as soon as I get back the Torch based servers running after today's game update...

xdudi commented 3 years ago

This bug occurs only with slow internet speed. I figured out that after this timeout occured, it still download this item in background. User has to wait before call next command. If it will not wait then next command will break background download and remove that downloading data from disk and start the execution of new command.

Its easy to reproduce: download item 583544987 by steamcmd (using interactive mode, not by script) with speed reduction to 1Mbits/sec. After about 10 minutes you will see something like that:

Steam>workshop_download_item 107410 583544987 validate
Downloading item 583544987 ...ERROR! Timeout downloading item 583544987
## HERE I'am waiting to finish steamcmd by looking on `sudo nethogs eth0` ##
Steam>workshop_download_item 107410 549676314 validate
Downloading item 549676314 ...

I have been struggling with this problem for 4 years.

viktor-ferenczi commented 3 years ago

It also happens on hosted dedicated server on a 1Gbps direct connection to the backbone. With plenty of CPU and RAM available.

I run a background process downloading Space Engineers blueprints on demand for pasting into a running game and that has this exact same issue on such a server. It depends only on the blueprint downloaded (workshop item) and seem to happen only to the older ones (sometime before 2018).

If this is a timeout due to slow download speed, then it may be that Steam is hosting the older data on dead slow or crowded edge servers or otherwise throttle their download. It would be nice to figure whether the slow speed is consistent, which would mean throttling.

I guess it would be possible to "patch" steamcmd and increase this timeout somehow, if this is not configurable.

xdudi commented 3 years ago

The timeout period is counted from the start of the download. This should be counted from the moment of network inactivity. I think that the time limit is too short for certain items to download.

viktor-ferenczi commented 3 years ago

Maybe there is a way to disable the timeout entirely. It could be wrapped over it from the outside (equally badly) if really needed for hosted/backend scenarios. If not configurable, then may be there is a way to patch the binary to disable it.

ArwynFr commented 2 years ago

I just found out that, if you run workshop_download_item in console, the client continues to download the file in the background after the timeout yields, up until completion.

ArwynFr commented 2 years ago

The timeout occurs after 5 minutes of download. The steamcmd is limited to 100 Mbps download so this makes the problem systematic for all content above 3,75 GB, but YMMV if your download speed is slower.

I'd say the command spawns a download thread, then wait to join it for 5 minutes and then times out. But it does not stop the thread nicely, it only move to the next command which fails since a download is pending. I don't understand why this was not implemented like app_update which works fine for larger content.

minuq commented 2 years ago

Edit: I'll keep what i wrote earliner, but apparently that only works sometimes and fails on slower networks.

The last message made me check what happens to files when workshop_download_item is called. I found that it deletes files from ../steamapps/workshop/downloads/<ID>/ and restarts the whole download.

If there are already files in ../steamapps/workshop/content/<ID>/ it'll validate those and only download missing files.

The solution to download mods with a filesize that causes steamcmd to timeout was to copy files from downloads to content after each timeout but before restarting workshop_download_item. Using mv instead of cp gave errors about busy resources, so i went with the slower but more reliable cp.

Working example in python:

def mod(ids):
    steamcmd = [os.environ["STEAMCMDDIR"] + "/steamcmd.sh"]
    steamcmd.extend(["+force_install_dir", "/arma3"])
    steamcmd.extend(["+login", os.environ["STEAM_USER"]])
    for id in ids:
        steamcmd.extend(["+workshop_download_item", "107410", id])
        steamcmd.extend(["validate"])
    steamcmd.extend(["+quit"])
    res = ""
    # steamcmd returns 10 for errors like timeouts
    while res != 0:
        res = subprocess.call(steamcmd)
        subprocess.call(["/bin/cp","-a","/arma3/steamapps/workshop/downloads/107410/.","/arma3/steamapps/workshop/content/107410/"])

https://github.com/mylesagray/Arma3Server/blob/master/app/workshop.py#L12-L24

Securitybits-io commented 2 years ago

Running in the same issue with Timeout when downloading mods through the workshop, especially for Arma3-Server. am running @minuq docker image, but it seems that the validate flag redownloads all the mod-files and then tries to validate to the files on disk. This results in the downloader getting stuck on a specific mod, in my case 843577117. Is there a way to adjust the Timeout Period?

ArwynFr commented 2 years ago

Is there a way to adjust the Timeout Period?

Unfortunately, no

magnusjjj commented 2 years ago

Running into this, while writing my own tool. Reproducible for me, multiple friends on windows:

workshop_download_item 4000 260275546 validate

That times out. It does not even start to download the file, as it looks like, it just creates an empty folder and never writes anything into it, and then after it times out it deletes the empty folder.

We are all on windows. I would raise the issue elsewhere, but I can't find anything more relevant and it shows that it's an issue that is far broader than a single kernel version :).

ArwynFr commented 2 years ago

That times out. It does not even start to download the file, as it looks like, it just creates an empty folder and never writes anything into it, and then after it times out it deletes the empty folder.

Try running steamcmd in an elevated prompt. It looks like a permission issue rather than a network timeout.

magnusjjj commented 2 years ago

It was very, very much not. It times out if you try to download a pre-workshoprework addon.

Btw, how could it have been permissions? Does anyone actually run steamcmd in an administrator controlled directory? O_o

ArwynFr commented 2 years ago

I installed steamcmd in Program Files so it requires administrator privilege to auto-upgrade. When steamcmd is run without elevation, it works fine unless there is a platform upgrade. In such case, auto-upgrade fails silently, and then the application can hang pending with no actual download.

DanielCeregatti commented 2 years ago

ERROR! Timeout downloading item 1602372402src/clientdll/contentupdatecontext.cpp (743) : m_uIOCurrentCacheSize == 0
src/clientdll/contentupdatecontext.cpp (743) : m_uIOCurrentCacheSize == 0

This is the Deerisle workshop mod of DayZ. It's very large. If one doesn't have a fast connection to the internet, this always fails with the error above.

I'm left to wonder:

1) WHY? What if steamcmd just didn't time out? Leave it up to the user to terminate the program manually if they wish! For some people, it'd surely take over an hour to download large workshop content over slow links. 2) Why download anew for every attempt? The download for this mod gets to about 3.3G in size before it times out. Not only that, the files are left in place when it fails! When re-attempting the download, they're deleted! Why? The ability to resume an HTTP download has existed now for decades. Just resume the download instead? Better yet, use an rsync/bittorrent based protocol instead? All the other modern game launchers do.

I'm trying to build tooling to facilitate playing DayZ on Linux using Proton. These bugs, which are mostly Linux-only bugs (i.e. have no analog in the windows client), are making that really hard.

ArwynFr commented 2 years ago

Please read above, the download process does not time out actually. You can keep the client open (with steam> prompt) and the download will continue in background until completion, even after the client yielded the error. Monitor your network bandwidth for actual download completion.

Yes, this is a bad behavior for server tooling development, but it doesn't look like Valve cares.

DanielCeregatti commented 2 years ago

Except it does timeout. Running it via a prompt is silly when my goal is automation. Is steamcmd not supposed to be an automation tool?

I'm hoping an LD_PRELOAD wrapper for setsockopt's SO_RCVTIMEO will address this more elegantly.

Well, other than an actual fix from upstream, that is, but I'm 0 and 2 here already, so not holding my breath.

minuq commented 2 years ago

Except it does timeout. Running it via a prompt is silly when my goal is automation. Is steamcmd not supposed to be an automation tool?

Completely agree with this. Which is why we spent another couple hours on coming up with a solution that works reliably, even if it's slow.

Depending on your download speed and/or maximum mod size you'll need to change the TIMEOUT variable. What it does is download files using steamcmd but as soon as it runs into a timeout, it runs steamcmd +$parameters again without quitting it, so the download continues in the background and ends the spawned process after a predefined TIMEOUT.

def mod(ids):
    steamcmd = [os.environ["STEAMCMDDIR"] + "/steamcmd.sh"]
    steamcmd.extend(["+force_install_dir", "/arma3"])
    steamcmd.extend(["+login", os.environ["STEAM_USER"]])
    for id in ids:
        steamcmd.extend(["+workshop_download_item", "107410", id])
        steamcmd.extend(["validate"])
    steamcmd.extend(["+quit"])
    try:
        subprocess.run(steamcmd, check=True)
    except subprocess.CalledProcessError:
        # If this is triggered, steamcmd ran into an issue, most likely a server side timeout
        # Retrying the download with the timeout set in .env, without +quit
        steamcmd.pop(-1)
        if "WORKSHOP_TIMEOUT" in os.environ and len(os.environ["WORKSHOP_TIMEOUT"]) > 0:
            timeout = int(os.environ["WORKSHOP_TIMEOUT"])*60
        else:
            timeout = 600
        subprocess.run(steamcmd, timeout=timeout, check=True)

https://github.com/mylesagray/Arma3Server/blob/master/app/workshop.py#L11-L29

jsirianni commented 5 months ago

This is till an issue in 2024. Having issues download Dayz Deerisl (workshop id 1602372402).

ValveSoftware / steam-for-linux