Steam appears to getting captured by SNI Proxy

[NullOrNaN]

It looks like SNIProxy is hitting a domain owned by Comcast for steam downloads. I suspect that Steam/Comcast have an agreement that allows Comcast to host a CDN for their clients for Steam downloads.

If I simply disable SNIProxy, however, the downloads get cached with Steamcache Generic as expected. Not really sure what to make of this or if this is requiring a pull request sent to UKLans to have this resolved.

Brief logs

[slazz]

I'm seeing some chunks downloaded via SNIProxy also, however this does not appear to be limited to Comcast URLs, e.g:

2019-01-02 13:35:46 172.30.0.1:37850 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 4331/4331 bytes tx 994/994 bytes rx 0.296 seconds
2019-01-02 13:35:46 172.30.0.1:37848 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 7692/7692 bytes tx 994/994 bytes rx 0.398 seconds
2019-01-02 13:35:46 172.30.0.1:37852 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 4330/4330 bytes tx 994/994 bytes rx 0.347 seconds
2019-01-02 13:35:47 172.30.0.1:37872 -> 0.0.0.0:443 -> 152.195.38.181:443 [edgecast.cdn.steampipe.steamcontent.com] 60237/60237 bytes tx 1555/1555 bytes rx 0.523 seconds
2019-01-02 13:35:47 172.30.0.1:37892 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 1005657/1005657 bytes tx 1283/1283 bytes rx 0.649 seconds
2019-01-02 13:35:47 172.30.0.1:37842 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 3484265/3484265 bytes tx 4195/4195 bytes rx 1.647 seconds
2019-01-02 13:35:47 172.30.0.1:37866 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 1093795/1093795 bytes tx 3178/3178 bytes rx 1.249 seconds

I also note the hostnames seen above apparently have optional https_support:

$ curl -s http://valve2004.steamcontent.com/serverlist/53/20/
"serverlist"
{
        "0"
        {
                "type"          "CDN"
                "vhost"         "steam.cdn.on.net"
                "Host"          "steam.cdn.on.net"
                "load"          "0"
                "weightedload"          "15.00"
                "sourceid"              "6"
                "NumEntriesInClientList"                "3"
                "https_support"         "unavailable"
        }
        "1"
        {
                "type"          "CS"
                "sourceid"              "2004"
                "cell"          "52"
                "load"          "93"
                "weightedload"          "122.52"
                "NumEntriesInClientList"                "1"
                "Host"          "valve2004.steamcontent.com"
                "vhost"         "valve2004.steamcontent.com"
                "usetokenauth"          "1"
                "https_support"         "optional"
        }
        "2"
        {
                "type"          "CDN"
                "vhost"         "edgecast.cdn.steampipe.steamcontent.com"
                "Host"          "edgecast.cdn.steampipe.steamcontent.com"
                "load"          "0"
                "weightedload"          "130.00"
                "sourceid"              "32"
                "NumEntriesInClientList"                "1"
                "https_support"         "optional"
        }
        "3"
        {
                "type"          "CDN"
                "vhost"         "steampipe.akamaized.net"
                "Host"          "steampipe.akamaized.net"
                "load"          "0"
                "weightedload"          "130.00"
                "sourceid"              "11"
                "NumEntriesInClientList"                "1"
                "https_support"         "optional"
        }
        "4"
        {
                "type"          "CDN"
                "vhost"         "steam.ix.asn.au"
                "Host"          "steam.ix.asn.au"
                "load"          "0"
                "weightedload"          "180.00"
                "sourceid"              "10"
                "NumEntriesInClientList"                "2"
                "https_support"         "unavailable"
        }
}
$

Interestingly enough, the steam client doesn't seem to switch completely over to https, and will happily download some chunks via http (and be cached), while simultaneously downloading other chunks over https.

As a quick hack I just redirected the hostnames I saw in my logs to a closed port on localhost, and once the steam client tries and fails once via https it does seem to happily failover to cacheable http.

$ docker exec -ti 23984493152a tail -n5 /etc/sniproxy.conf
table {
        .*\.steamcontent\.com 127.0.0.1:1
        steampipe.akamaized.net 127.0.0.1:1
        .* *:443
}
$

$ docker logs 23984493152a 2>&1 | tail
2019-01-02 13:36:18 172.30.0.1:37950 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 5195/5195 bytes tx 994/994 bytes rx 30.340 seconds
2019-01-02 13:36:18 172.30.0.1:37952 -> 0.0.0.0:443 -> 103.10.125.3:443 [valve2004.steamcontent.com] 9995/9995 bytes tx 994/994 bytes rx 30.383 seconds
2019-01-02 13:36:55 recv(): Connection refused, closing connection
2019-01-02 13:36:55 172.30.0.1:37998 -> 0.0.0.0:443 -> 127.0.0.1:1 [valve2004.steamcontent.com] 0/0 bytes tx 220/440 bytes rx 0.045 seconds
2019-01-02 13:37:14 recv(): Connection refused, closing connection
2019-01-02 13:37:14 172.30.0.1:38474 -> 0.0.0.0:443 -> 127.0.0.1:1 [edgecast.cdn.steampipe.steamcontent.com] 0/0 bytes tx 517/1034 bytes rx 0.096 seconds
2019-01-02 13:37:14 172.30.0.1:38480 -> 0.0.0.0:443 -> 127.0.0.1:1 [steampipe.akamaized.net] 0/0 bytes tx 517/1034 bytes rx 0.094 seconds
2019-01-02 13:37:14 recv(): Connection refused, closing connection
2019-01-02 13:49:08 recv(): Connection refused, closing connection
2019-01-02 13:49:08 172.30.0.1:56186 -> 0.0.0.0:443 -> 127.0.0.1:1 [valve2004.steamcontent.com] 0/0 bytes tx 220/440 bytes rx 0.046 seconds
$

[unspec]

See https://github.com/lancachenet/monolithic/issues/85 for updates on the new steam supported http failback system - this should resolve outstanding steam/https/comcast issues. Feel free to reopen if still having issues with the new one.