Encountered 401 for depot manifest

[imkzh]

I'm getting Encountered 401 for depot manifest ... error and failed to download. i'm using the latest release (depotdownloader-2.3.6, from releases page) and (.NET core runtime 3.1.7)

.\depotdownloader.bat -app 578080 -depot 578081 -manifest 1809847898669233696 -username [] -dir C:\Users\root\Desktop\projects\PUBG_RESOURCE\depot -debug

Enter account password for "[]":
Connecting to Steam3...[ServerList] Resolving server list
[ServerList] Server list provider had no entries, will query SteamDirectory
[ServerList] Resolved 200 servers
[ServerList] Next server candidiate: 203.80.149.68:27018 (Tcp)
[d010c4b267864bad851e33f1ced2e0d2/TcpConnection] Connecting to 203.80.149.68:27018...
[d010c4b267864bad851e33f1ced2e0d2/TcpConnection] Connected to 203.80.149.68:27018
[d010c4b267864bad851e33f1ced2e0d2/EnvelopeEncryptedConnection] Got encryption request. Universe: Public Protocol ver: 1
[d010c4b267864bad851e33f1ced2e0d2/EnvelopeEncryptedConnection] Encryption result: OK
 Done!
Logging '[]' into Steam3...Disconnected from Steam
This account is protected by Steam Guard.
Please enter your 2 factor auth code from your authenticator app: [d010c4b267864bad851e33f1ced2e0d2/TcpConnection] Socket exception occurred while reading packet: System.IO.IOException: Connection lost while reading packet header.
 ---> System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at System.IO.BinaryReader.ReadUInt32()
   at SteamKit2.TcpConnection.ReadPacket() in C:\Projects\SteamKit\SteamKit2\SteamKit2\Networking\Steam3\TcpConnection.cs:line 284
   --- End of inner exception stack trace ---
   at SteamKit2.TcpConnection.ReadPacket() in C:\Projects\SteamKit\SteamKit2\SteamKit2\Networking\Steam3\TcpConnection.cs:line 289
   at SteamKit2.TcpConnection.NetLoop() in C:\Projects\SteamKit\SteamKit2\SteamKit2\Networking\Steam3\TcpConnection.cs:line 247
KD33X
Retrying Steam3 connection...[ServerList] Next server candidiate: 203.80.149.68:27018 (Tcp)
[d010c4b267864bad851e33f1ced2e0d2/TcpConnection] Connecting to 203.80.149.68:27018...
[d010c4b267864bad851e33f1ced2e0d2/TcpConnection] Connected to 203.80.149.68:27018
[d010c4b267864bad851e33f1ced2e0d2/EnvelopeEncryptedConnection] Got encryption request. Universe: Public Protocol ver: 1
[d010c4b267864bad851e33f1ced2e0d2/EnvelopeEncryptedConnection] Encryption result: OK
 Done!
Logging '[]' into Steam3... Done!
Using Steam3 suggested CellID: 198
Got 79 licenses for account!
Got session token!
Got AppInfo for 578080
Using app branch: 'Public'.
Got depot key for 578081 result: OK
Downloading depot 578081 - PLAYERUNKNOWN'S BATTLEGROUNDS  Content
Downloading depot manifest...Accepted new login key for account []
Got CDN auth token for update4.dota2.wmsj.cn result: OK (expires 2020/9/11 13:07:18)
[CDNClient] Failed to complete web request to http://update4.dota2.wmsj.cn/depot/578081/manifest/1809847898669233696/5: Response status code does not indicate success: 403 (Forbidden).
Encountered 401 for depot manifest 578081 1809847898669233696. Aborting.

Unable to download manifest 1809847898669233696 for depot 578081
Disconnected from Steam

I have tried several ManifestIDs (from very old one to latest one) for Depot 578081 App 578080 which contain PUBG resources, but all of them failed with same error. (I have PUBG in my library)

Any hint?

[PaulNevada]

First of all - great to see (fan?) of PUBG... Second - will try to do the same a bit later and let you know here, for example...

[PaulNevada]

It's working well... Maybe, you try to restart your networking devices and your PC? It helps in some cases on my side... 3.1.401 [C:\Program Files\dotnet\sdk]...

[azuisleet]

This is difficult to diagnose because I'm not certain how Steam works behind the firewall. There's a missing piece somewhere because there wasn't an auth string provided for the CDN

[imkzh]

Yes i do have really bad connection here (especially for oversea connections), i'll try again elsewhere and come back really soon.

By the way, does anybody know if an old entry (manifest) in depot will become unavailable after some time, saying 1 or 2 years? (since PUBG updates really quickly, and old entries are quite big and useless (for normal users)) I would like to download a really old version because there is a time when resources(like mesh and texture) in .pak files from PUBG could be exported (sadly i lost that copy).

@paulnevada1983 Haha, yes I love this game, but i'm always too busy to play.

[PaulNevada]

As far as I know, all the content versions are there...

[PaulNevada]

What exactly Manifest ID are you searching for? Just wandering...

[imkzh]

after doing some research on this, i found that multiple CDN is configured for my place,

in steam console:

download_sources ]

- host update6.dota2.wmsj.cn, vhost update6.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update5.dota2.wmsj.cn, vhost update5.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update1.csgo.wmsj.cn, vhost update1.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update2.dota2.wmsj.cn, vhost update2.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update5.csgo.wmsj.cn, vhost update5.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update4.dota2.wmsj.cn, vhost update4.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update1.dota2.wmsj.cn, vhost update1.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update2.csgo.wmsj.cn, vhost update2.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update3.csgo.wmsj.cn, vhost update3.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update6.csgo.wmsj.cn, vhost update6.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update4.csgo.wmsj.cn, vhost update4.csgo.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host update3.dota2.wmsj.cn, vhost update3.dota2.wmsj.cn, cell 0, CDN (#20), 0 connections of 1 max
- host dl.steam.ksyna.com, vhost dl.steam.ksyna.com, cell 0, CDN (#30), 3 connections of 3 max
- host st.dl.pinyuncloud.com, vhost st.dl.pinyuncloud.com, cell 0, CDN (#29), 0 connections of 3 max
- host cdn.mileweb.cs.steampowered.com.8686c.com, vhost cdn.mileweb.cs.steampowered.com.8686c.com, cell 0, CDN (#19), 0 connections of 3 max

Download jobs:
- type CDN, name dl.steam.ksyna.com, chunks: download = 174, outstanding = 7, deferred = 19, succeeded = 11, failed = 0 
- type CDN, name dl.steam.ksyna.com, chunks: download = 180, outstanding = 5, deferred = 28, succeeded = 9, failed = 0 
- type CDN, name dl.steam.ksyna.com, chunks: download = 138, outstanding = 17, deferred = 50, succeeded = 17, failed = 0 

and from the DepotDownloader log we can see we are downloading from update4.dota2.wmsj.cn: http://update4.dota2.wmsj.cn/depot/578081/manifest/1809847898669233696/5

After doing some search on internet, I was told that *.dota2.wmsj.cn only provide updates for Dota2 and *.csgo.wmsj.cn only for CS:GO.

Based on above findings, it seems that wrong CDN server is used, so the problem become is there some option that i can use for manually specify the CDN (or my only choice is to compile the downloader from source)?

@paulnevada1983 I'm not sure which ManifestID will work, i will just go through them and find one suitable. But I am pretty sure the .pak file change happend in the past 12 months.

[imkzh]

I patched the source code adding a new option -blacklist-cdn which allows me to blacklist any CDN host matching a given Regex, and now download started and progress advances gradually, but i am getting Failed to complete error.

DepotDownloader log (with -debug)

Logging '[]' into Steam3... Done!
Using Steam3 suggested CellID: 198
Got 79 licenses for account!
Got session token!
Got Machine Auth: ssfn725811375........03 0 2048 2048
Accepted new login key for account []
Got AppInfo for 578080
Using app branch: 'Public'.
Got depot key for 578081 result: OK
Downloading depot 578081 - PLAYERUNKNOWN'S BATTLEGROUNDS  Content
Downloading depot manifest...Got CDN auth token for cdn.mileweb.cs.steampowered.com.8686c.com result: OK (expires 2020/9/11 18:49:46)
 Done!
Manifest 8772400417083205880 (2020/2/19 13:49:05)
 00.00% C:\Users\ken\Desktop\projects\PUBG_RESOURCE\depot\Engine\Config\controller.vdf
 00.00% C:\Users\ken\Desktop\projects\PUBG_RESOURCE\depot\Engine\Build\Build.version
 00.00% C:\Users\ken\Desktop\projects\PUBG_RESOURCE\depot\TslGame\Content\Movies\LoadingScreen_Xbox.mp4
Got CDN auth token for st.dl.pinyuncloud.com result: OK (expires 2020/9/11 18:49:49)
 00.01% C:\Users\ken\Desktop\projects\PUBG_RESOURCE\depot\TslGame\Content\Movies\LoadingScreen.mp4
[CDNClient] Failed to complete web request to http://cdn.mileweb.cs.steampowered.com.8686c.com/depot/578081/chunk/047815ec2509b1f78ea740e0df120af01bb81327: The operation was canceled.
[CDNClient] Failed to complete web request to http://cdn.mileweb.cs.steampowered.com.8686c.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: The operation was canceled.
Connection timeout downloading chunk 11d5f2632501d01f4660f31a70d11fe73c36bf4a
Connection timeout downloading chunk 047815ec2509b1f78ea740e0df120af01bb81327
[CDNClient] Failed to complete web request to http://st.dl.pinyuncloud.com/depot/578081/chunk/04831d4fd683daddf435b605c4f2668399051bb9: The operation was canceled.
Connection timeout downloading chunk 04831d4fd683daddf435b605c4f2668399051bb9
Got CDN auth token for dl.steam.ksyna.com result: OK (expires 2020/9/11 18:49:59)
[CDNClient] Failed to complete web request to http://st.dl.pinyuncloud.com/depot/578081/chunk/047815ec2509b1f78ea740e0df120af01bb81327: The operation was canceled.
Connection timeout downloading chunk 047815ec2509b1f78ea740e0df120af01bb81327
[CDNClient] Failed to complete web request to http://st.dl.pinyuncloud.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: The operation was canceled.
Connection timeout downloading chunk 11d5f2632501d01f4660f31a70d11fe73c36bf4a
[CDNClient] Failed to complete web request to http://dl.steam.ksyna.com/depot/578081/chunk/205975a99e82219dc8134ccbb6267d2c034c2664: The operation was canceled.
Connection timeout downloading chunk 205975a99e82219dc8134ccbb6267d2c034c2664
[CDNClient] Failed to complete web request to http://dl.steam.ksyna.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: The operation was canceled.
Connection timeout downloading chunk 11d5f2632501d01f4660f31a70d11fe73c36bf4a
[CDNClient] Failed to complete web request to http://dl.steam.ksyna.com/depot/578081/chunk/047815ec2509b1f78ea740e0df120af01bb81327: The operation was canceled.
Connection timeout downloading chunk 047815ec2509b1f78ea740e0df120af01bb81327
[CDNClient] Failed to complete web request to http://cdn.mileweb.cs.steampowered.com.8686c.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: The operation was canceled.
Connection timeout downloading chunk 11d5f2632501d01f4660f31a70d11fe73c36bf4a
[CDNClient] Failed to complete web request to http://cdn.mileweb.cs.steampowered.com.8686c.com/depot/578081/chunk/047815ec2509b1f78ea740e0df120af01bb81327: The operation was canceled.
Connection timeout downloading chunk 047815ec2509b1f78ea740e0df120af01bb81327
[CDNClient] Failed to complete web request to http://cdn.mileweb.cs.steampowered.com.8686c.com/depot/578081/chunk/8401e1a961416cfcc38232921f48af919cee2709: The operation was canceled.
Connection timeout downloading chunk 8401e1a961416cfcc38232921f48af919cee2709
[CDNClient] Failed to complete web request to http://dl.steam.ksyna.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: The operation was canceled.
Connection timeout downloading chunk 11d5f2632501d01f4660f31a70d11fe73c36bf4a
[CDNClient] Failed to complete web request to http://dl.steam.ksyna.com/depot/578081/chunk/047815ec2509b1f78ea740e0df120af01bb81327: The operation was canceled.
Connection timeout downloading chunk 047815ec2509b1f78ea740e0df120af01bb81327
[CDNClient] Failed to complete web request to http://st.dl.pinyuncloud.com/depot/578081/chunk/11d5f2632501d01f4660f31a70d11fe73c36bf4a: A task was canceled.
...

Is this normal behavior, or indicates a incomplete download?

[PaulNevada]

Take another depot - 578082 (Binaries) - it is about 200 MBs and you'll see all the tool's messages including the finishing ones in case your download started to work all in all...

[azuisleet]

Connection timeouts are OK, but it does indicate your download is slow. We probably need to see what https://api.steampowered.com/IContentServerDirectoryService/GetServersForSteamPipe/v1/?cell_id=198 looks like from your network.

There are probably additional rules for how to handle these CDN servers.

[imkzh]

An extra entry named allowed_app_ids will be returned for servers that only provide updates for certain games.

https://api.steampowered.com/IContentServerDirectoryService/GetServersForSteamPipe/v1/?cell_id=198 :

{
    "response":{
        "servers":[
            {
                "type":"CDN",
                "source_id":20,
                "load":0,
                "weighted_load":0,
                "num_entries_in_client_list":1,
                "host":"update6.csgo.wmsj.cn",
                "vhost":"update6.csgo.wmsj.cn",
                "https_support":"unavailable",
                "allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],
                "preferred_server":true
            },
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update1.dota2.wmsj.cn","vhost":"update1.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":true},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update2.dota2.wmsj.cn","vhost":"update2.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update3.csgo.wmsj.cn","vhost":"update3.csgo.wmsj.cn","https_support":"unavailable","allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update1.csgo.wmsj.cn","vhost":"update1.csgo.wmsj.cn","https_support":"unavailable","allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update4.dota2.wmsj.cn","vhost":"update4.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update5.dota2.wmsj.cn","vhost":"update5.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update6.dota2.wmsj.cn","vhost":"update6.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update2.csgo.wmsj.cn","vhost":"update2.csgo.wmsj.cn","https_support":"unavailable","allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update4.csgo.wmsj.cn","vhost":"update4.csgo.wmsj.cn","https_support":"unavailable","allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update3.dota2.wmsj.cn","vhost":"update3.dota2.wmsj.cn","https_support":"unavailable","allowed_app_ids":[570,580,741,816,817,205790,205800,228983,247040,247060,247200,247220,373301,373310,381450,381451,381452],"preferred_server":false},
            {"type":"CDN","source_id":20,"load":0,"weighted_load":0,"num_entries_in_client_list":1,"host":"update5.csgo.wmsj.cn","vhost":"update5.csgo.wmsj.cn","https_support":"unavailable","allowed_app_ids":[711,712,730,731,732,733,734,735,736,741,181452,268441,268442,268443,268444],"preferred_server":false},
            {"type":"CDN","source_id":29,"load":0,"weighted_load":50,"num_entries_in_client_list":3,"host":"st.dl.pinyuncloud.com","vhost":"st.dl.pinyuncloud.com","https_support":"unavailable","preferred_server":true},
            {
                "type":"CDN",
                "source_id":30,
                "load":0,
                "weighted_load":50,
                "num_entries_in_client_list":3,
                "host":"dl.steam.ksyna.com",
                "vhost":"dl.steam.ksyna.com",
                "https_support":"unavailable",
                "preferred_server":true
            },{
                "type":"CDN",
                "source_id":19,
                "load":0,
                "weighted_load":50,
                "num_entries_in_client_list":3,
                "host":"cdn.mileweb.cs.steampowered.com.8686c.com",
                "vhost":"cdn.mileweb.cs.steampowered.com.8686c.com",
                "https_support":"unavailable",
                "preferred_server":false
            }
        ]
    }
}

[azuisleet]

This should be addressed by the new logic in ac97c011077f8476beee540f04c8a3e220744233. A new release should be out eventually.