[SteamCMD] Non TTY output truncated

[remygrandin]

Some variant of this problem have been reported but I wanted to give this topic a fresh start and to help provide solid and real exemples to this problem to try and find a solution or at least help the devs to find one.

Syptom

As some of you may know, SteamCMD, as great a tool as it is, have some major output issues for quite some time now when used outsite a TTY (like writing to a file or in some environement, to a variable)

For exemple, execution this simple command :

C:\steamcmd>steamcmd.exe +login Anonymous +app_info_print 298740 +quit > out.log

Will always return this output :

Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for license info...OK

Whereas we are expecting a big json-like outupt with all the app info in it. Note that the behavior seem to depend on the size of the supposed output, and by so, on the app id used

More troubleing, is if we try to call the app_info_print twice in a row :

C:\steamcmd>steamcmd.exe +login Anonymous +app_info_print 298740 +app_info_print 298740 +quit > out.log

We have as output :

Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for license info...OK
AppID : 298740, change number : 1399450/0, token 0, last change : Fri Oct 30 14:44:59 2015 
"298740"
{
    "common"
    {
        "name"      "Space Engineers Dedicated Server"
        "type"      "Tool"
        "ReleaseState"      "released"
        "parent"        "244850"
        "oslist"        "windows"
        "gameid"        "298740"
    }
    "config"
    {
        "contenttype"       "3"
        "installdir"        "SpaceEngineersDedicatedServer"
        "launch"
        {
            "0"
            {
                "executable"        "DedicatedServer/SpaceEngineersDedicated.exe"
                "description"       "Launch"
                "workingdir"        "DedicatedServer"
                "config"
                {
                    "oslist"        "windows"
                    "osarch"        "32"
                }
            }
            "1"
            {
                "executable"        "DedicatedServer64/SpaceEngineersDedicated.exe"
                "description"       "Launch"
                "workingdir"        "DedicatedServer64"
                "config"
                {
                    "oslist"        "windows"
                    "osarch"        "64"
                }
            }
        }
    }
    "depots"
    {
        "1004"
        {
            "name"      "Steamworks SDK Redist (WIN32)"
            "config"
            {
                "oslist"        "windows"
            }
            "manifests"
            {
                "public"        "1559674099137950788"
            }
            "maxsize"       "23831088"
            "depotfromapp"      "1007"
        }
        "1005"
        {
            "name"      "Steamworks SDK Redist (OSX32)"
            "config"
            {
                "oslist"        "macos"
            }
            "manifests"
            {
                "public"        "9125417765790372519"
            }
            "maxsize"       "59600296"
            "depotfromapp"      "1007"
        }
        "1006"
        {
            "name"      "Steamworks SDK Redist (LINUX32)"
            "config"
            {
                "oslist"        "linux"
            }
            "manifests"
            {
                "public"        "8597155966586059395"
            }
            "maxsize"       "37526082"
            "depotfromapp"      "1007"
        }
        "229002"
        {
            "name"      ".NET 4.0 Redist"
            "config"
            {
                "oslist"        "windows"
            }
            "manifests"
            {
                "public"        "7260605429366465749"
            }
            "maxsize"       "50450161"
            "depotfromapp"      "228980"
        }
        "298741"
        {
            "name"      "Space Engineers Dedicated Server Content"
            "manifests"
            {
                "public"        "5934293394975131601"
                "old_multiplayer"       "8181467767056662663"
            }
            "maxsize"       "444174393"
            "encryptedmanifests"
            {
                "planets"
                {
                    "encrypted_gid"     "3B0F80378215865928E5CD2FAB7E05A94E62672A25B65C6FEC95FF7E52D813FD83969B38B40015C69C2E28F2C4C653C95FE89EE5"
                    "encrypted_size"        "3CFD7A77099E8BAEECCEAD04C5C057574E62672A25B65C6FEC95FF7E52D813FD9844ABD166C5DDDD93CBEA454C624C348B2A02E8"
                    "encrypted_gid_2"       "3B9874911C186B2AD2F0CA6800CF9F4F"
                    "encrypted_size_2"      "DAD630EF3FCC084FCFB8E2C6B301FCC2"
                }
                "automatic"
                {
                    "encrypted_gid"     "14296C5578EF9639F832E53F6044A5911AE748E2FD7B416769518F40ACD2EF0DAF57BBD6036F5FCB61A0A38EF9F888A2C68BC0FB"
                    "encrypted_size"        "A3A352490B6295357C4E5F6E9544937A1AE748E2FD7B416769518F40ACD2EF0D86690E47C514EBB1FF6DD53F6301602E43C4BCD1"
                    "encrypted_gid_2"       "5415C6176DB8F402C99A51237777043F"
                    "encrypted_size_2"      "C422A6891F9678E19C5B8667DD2F118A"
                }
            }
        }
        "branches"
        {
            "public"
            {
                "buildid"       "837156"
            }
            "dev"
            {
                "buildid"       "837156"
                "description"       "Closed testing branch"
                "pwdrequired"       "1"
                "pwdtestgid"        "7C1964220ECF36920FEC45753C664A7B94FDEDD7159E9050EB6C67453F4EC0D1844FC64A6693D46621B727809208F9C09FB6236C"
            }
            "old_multiplayer"
            {
                "buildid"       "818682"
                "description"       ""
            }
            "planets"
            {
                "buildid"       "837507"
                "description"       "Planets testing"
                "pwdrequired"       "1"
                "pwdtestgid"        "768E74817439A72249EB3FE20B00B7174E62672A25B65C6FEC95FF7E52D813FD7A859707531EFF60DB5CBF5DE37BC896DD747300"
            }
            "automatic"
            {
                "buildid"       "832673"
                "description"       ""
                "pwdrequired"       "1"
                "pwdtestgid"        "5FC9EB7019A488014FA5883DB3342ED81AE748E2FD7B416769518F40ACD2EF0D17EED8E44FBD9B5E6F20CA83296E18491D01D641"
            }
        }
    }
}
AppID : 298740, change number : 1399450/0, token 0, last change : Fri Oct 30 14:44:59 2015 
"298740"
{
    "common"
    {
        "name"      "Space Engineers Dedicated Server"
        "type"      "Tool"
        "ReleaseState"      "released"
        "parent"        "244850"
        "oslist"        "windows"
        "gameid"        "298740"
    }
    "config"
    {
        "contenttype"       "3"
        "installdir"        "SpaceEngineersDedicatedServer"
        "launch"
        {
            "0"
            {
                "executable"        "DedicatedServer/SpaceEngineersDedicated.exe"
                "description"       "Launch"
                "workingdir"        "DedicatedServer"
                "config"

Which seem to be 1 and a half times the info where we expected 2 time the same thing

Another exemple with app_status this time. with the app installed (to get the installed buildid) :

C:\steamcmd>steamcmd.exe +force_install_dir C:\gamedir\SyncData\ +login Anonymous +app_status 298740 +quit > out.log

Here again, we have no usable output :

Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for license info...OK

Even if we try to call it multiple times in a row :

C:\steamcmd>steamcmd.exe +force_install_dir C:\gamedir\SyncData\ +login Anonymous +app_status 298740 +app_status 298740 +app_status 298740 +app_status 298740 +app_status 298740 +app_status 298740 +app_status 298740 +quit > out.log

No real output :

Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...OK.

Connecting anonymously to Steam Public...Logged in OK
Waiting for license info...OK

Analysis

The key problem here is outsite a TTY as all those command executed without redirection stdout to a file works perfectly.

Possible explanaition :

• SteamCMD stdout buffer filling too fast
  • Possible but unlikely as numerous other cmd apps output much larger text data without a hiccup
  • Also, in the app_status exemple, we are only speaking about 8-10 short lines of output, probably not enought to fill it up.
• SteamCMD emtying it's stdout buffer instead of flushing it in some case (condition unknow)
  • My best guess so far, If more C/C++ guys have any ideas, post them below

    Severity

This issue is I think a major one as it prevent the use of SteamCMD in one of it's main uses : as part of automation scripts and tools. The unreliable output is really a big setback for anyone who is trying to use SteamCMD to get dynamicly info on apps version status and other, as we don't have (yet, I'm always hoping) a proper http API to get those infos

I Invite anyone who is having the same problem to post here some real, repeatable exemple of the issue and disscuss the probable causes.

[Tele42]

Hello @remygrandin, first of all, this is not the correct issue tracker for steamcmd, I'm not sure what would be.

My dirt simple updater solved this issue by forcing an appinfo update by removing appcache/appinfo.vdf immediately before running a non-interactive steamcmd (found at https://github.com/SteamLUG/steamlug-gaming-servers/wiki/GS5#update-script). linuxgsm also does something simular (https://github.com/dgibbs64/linuxgsm/blob/master/functions/fn_update_check#L124).

The issue with app_info_print that in non-interactive mode, steamcmd is querying the local cache of appinfo.vdf, the update is deferred or run in parallel, and steamcmd processes the query and closes before new information is gathered.

I have no info regarding app_status.

[remygrandin]

As it is the only "official" issue tracker that I found speaking of SteamCMD, I posted it here, but if anyone know where it would fit better, I would be happy to move it there ^^.

Anyway, i think you misunderstood the issue here, i'm not talking about not up to date version info (altrought i also have encountered the issue and have come with a solution similar to the one you linked). The problem I'm writing about have nothing to do with the rightness of the data in the output but with the data output not being outputed at all !

It seem more of a bug rather than a conception issue.

I only have reproduced the issue on windows, it's possible that this output problem don't occur on linux with your scripts

[Tele42]

@remygrandin I read your issue report carefully before adding my peace. In my humble opinion, the root cause is the same. While troubleshooting this quirk for myself, I found that steamcmd is looking in appinfo.vdf, if there is an existing entry that matches, it returns the potentially out of date information, if there is no match, then it returns nothing at all, which is the same as asking for an invalid appid.

Double app_info_print appears to only work because the first query stalls long enough for steamcmd to be ready to give a proper result on the second pass. Deleting appinfo.vdf potentially works by the same principle -- stalling steamcmd long enough for it to be ready to use the remote dataset.

[remygrandin]

For the sake of the argumentation, I have teste your theory by deleting the appinfo.vdf and re-runed each of my test cases, but the output (and more precisely the output problem) is still the same.I've rune each test case with the deletion 5 time to be sure, exactly the same, up to the byte.

I'm convinced this has nothing to do with a time or even a network syncronisation issue. I'll present 2 simple exemple bellow :

1 - If you run any of my first post command without the +exit, the output is still the same

2 - If you run the following command :

C:\SteamCMD>steamcmd.exe +find a +exit > out.log

The output have the same constant problem, it's missing about 15 lines (and is ironicly stopping on the middle of the description of the "exit" command ...). I think we can agree that the find command is a totally offline one and shouldn't event depend on anything.

All of that to say that, in the facts, this problem is a systemic one, reproducible on every windows machine and stateless.

[remygrandin]

Sorry, missclicked

[Visagalis]

Wow this issue is so old, but still here...

[Pingger]

Yep still existing the issue ... just ran into it ... again.

[Pingger]

I can't even access the data using a java console implementation. The output of the steamcmd just ends after the login

[Foxlider]

Still an issue

[duytnguyendtn]

Not a great fix, but on Windows I've been able to circumvent this using powershell's transcribe feature:

Start-Transcript -Path <path_to_file.log>
& .\steamcmd.exe app_info_update 1 +app_info_print <your_id_here>
Stop-Transcript

[ztgasdf]

Bump!

For Linux users, you can use the unbuffer command found in the expect package to get an unbuffered output, which can then be piped anywhere.

[characharm]

After nearly 10 years, I still have to resort to workarounds that unfortunately require a lot of unnecessary "hacks". Why is that? Please, Valve.

[Visagalis]

After nearly 10 years, I still have to resort to workarounds that unfortunately require a lot of unnecessary "hacks". Why is that? Please, Valve.

I have a feeling that will be fixed with half life 3 release 😂