lancachenet / monolithic

A monolithic lancache service capable of caching all CDNs in a single instance
https://hub.docker.com/r/lancachenet/monolithic
Other
725 stars 73 forks source link

Lancache Epic and RIOT (probably some others too, steam is fine) downloading keep doing start and stop, start and pause etc. #192

Open coddec opened 3 weeks ago

coddec commented 3 weeks ago

Describe the issue you are having

Case 1 Without lancache, Epic downloads GTA fine with full WAN speed

Case 2 With lancache, Epic prefill (https://github.com/tpill90/steam-lancache-prefill) for GTA will download at full WAN speed no stop or pause.

Case 3 With lancache, Epic and RIOT games downloading via Epic client or RIOT client will download at full speed at start then drop to 0, stay at 0 for sometime then back to full speed, then repeat. While this is happening lancache will have error log entries like shown below (I can see that the container is not downloading too, certainly not that container is downloading the client is waiting case. In other words not this https://lancache.net/docs/common-issues/#slow-download-speeds) The Epic client also show error message "We're sorry. It looks like we're having trouble connecting. We'll continue to retry" Click on it lead to https://www.epicgames.com/help/en-US/c-Category_EpicGamesStore/c-EpicGamesStore_ErrorCodes/is-0005-unable-to-connect-a000084850 stating "IS-0005: Unable to connect, This error is caused when your computer is not connected to the internet." image

My internet is certainly working and there's no fluctuation, otherwise Epic prefill will experience the same issue. Client computer to LAN and internet connection is also working fine. Content are cached though. Enough disk space, enough CPU and RAM etc. Client PC is fine too. 12MB/s should not upset NIC since tested on 1g NIC and 10g sfp+ dac. This issue makes the download taking way longer than it should be.

Case 4 Steam download works fine. Content are cached.

error.log
2024/06/09 12:09:40 [error] 1842#1842: *481 upstream prematurely closed connection while reading upstream, client: 192.168.0.11, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk HTTP/1.1", upstream: "http://127.0.0.1:3128/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:44 [error] 1834#1834: *1128 upstream prematurely closed connection while reading upstream, client: 192.168.0.11, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk HTTP/1.1", upstream: "http://127.0.0.1:3128/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk", host: "cloudflare.epicgamescdn.com"
upstream-error.log
2024/06/09 12:09:37 [error] 1869#1869: *829 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/84/D93D5307BC2E3B0A_E1FA523C40934DA164061687A71F786F.chunk HTTP/1.0", upstream: "http://104.18.50.190:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/84/D93D5307BC2E3B0A_E1FA523C40934DA164061687A71F786F.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:40 [error] 1899#1899: *889 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk HTTP/1.0", upstream: "http://104.18.50.190:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:44 [error] 1831#1831: *1241 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk HTTP/1.0", upstream: "http://104.18.50.184:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk", host: "cloudflare.epicgamescdn.com"
stream-error.log
2024/06/09 12:49:00 [error] 1831#1831: *11952 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 192.168.0.11, server: 0.0.0.0:443, upstream: "52.169.159.67:443", bytes from/to client:1076/3101, bytes from/to upstream:3101/1296
2024/06/09 12:49:03 [error] 1831#1831: *11951 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 192.168.0.11, server: 0.0.0.0:443, upstream: "13.74.139.29:443", bytes from/to client:1077/3101, bytes from/to upstream:3101/1297
C:\Users\[username]\AppData\Local\EpicGamesLauncher\Saved\Logs\EpicGamesLauncher.log
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/21/6319E1D547314493_723E74A54B7801F60D864397C8A55F75.chunk
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: request failed, libcurl error: 0 (No error)
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x261a5f33c50 [can pipeline])
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 1 (Re-using existing connection! (#639) with host cloudflare.epicgamescdn.com)
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#639))
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 3 (Closing connection 639)
[2024.06.10-04.24.51:488][354]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/21/6319E1D547314493_723E74A54B7801F60D864397C8A55F75.chunk
[2024.06.10-04.27.35:943][418]LogEOSMessageService: Error: Unable to find port.
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/27/F9BB3A429600D379_D84F1160496C7259D4D650BC32B1B74F.chunk
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: request failed, libcurl error: 0 (No error)
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x261a5f33c50 [can pipeline])
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 1 (Re-using existing connection! (#644) with host cloudflare.epicgamescdn.com)
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#644))
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 3 (Closing connection 644)
[2024.06.10-04.28.13:637][849]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/27/F9BB3A429600D379_D84F1160496C7259D4D650BC32B1B74F.chunk
C:\Users\[username]\AppData\Local\EpicGamesLauncher\Saved\Logs\EpicGamesLauncher.log
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/36/E07192FAF958A2D6_F6023FE84A6C3AAB76E8818FA9596FD3.chunk
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: request failed, libcurl error: 0 (No error)
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x239b9683e50 [can pipeline])
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 1 (Re-using existing connection! (#37) with host cloudflare.epicgamescdn.com)
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#37))
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 3 (Closing connection 37)
[2024.06.10-06.35.55:052][659]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/36/E07192FAF958A2D6_F6023FE84A6C3AAB76E8818FA9596FD3.chunk
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/58/F6295F141ABBF1C9_CC10965F49FE379F76A96094FF9791B3.chunk
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: request failed, libcurl error: 0 (No error)
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x239b9683e50 [can pipeline])
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 1 (Re-using existing connection! (#23) with host cloudflare.epicgamescdn.com)
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#23))
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 3 (Closing connection 23)
[2024.06.10-06.35.59:273][913]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/58/F6295F141ABBF1C9_CC10965F49FE379F76A96094FF9791B3.chunk
[2024.06.10-06.35.59:273][913]LogPortal: Error: FAlertReporter: Reporting Alert code: IS-0005
[2024.06.10-06.35.59:273][913]LogAlertListener: No user messaging in AlertMessageRepository for: IS-0005
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: HTTP request timed out after 30.00 seconds URL=http://epicgames-download1.akamaized.net/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/08/96490E5B268F9C52_2C9FCF2C4948373FD3010E8E1B9AF54A.chunk
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: request failed, libcurl error: 0 (No error)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 0 (  Trying 192.168.0.11...)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 1 (TCP_NODELAY set)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 2 (Connected to epicgames-download1.akamaized.net (192.168.0.11) port 80 (#39))
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 3 (Closing connection 39)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: HTTP request timed out after 30.00 seconds URL=http://epicgames-download1.akamaized.net/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/04/2E4D5B054046D34A_B4913E714BD8A998C6763C8BB0E8F513.chunk
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: request failed, libcurl error: 0 (No error)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 0 (Found bundle for host epicgames-download1.akamaized.net: 0x239b9680ed0 [serially])
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 1 (Hostname epicgames-download1.akamaized.net was found in DNS cache)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 2 (  Trying 192.168.0.11...)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 3 (TCP_NODELAY set)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 4 (Connected to epicgames-download1.akamaized.net (192.168.0.11) port 80 (#41))
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 5 (Closing connection 41)

Tried to clear all cache Tried to change slice size between 1M - 8M Tried to disable slice, by setting to 0 Add multiple IP to lancache host tested, no help No client RAM, CPU, Disk issue No lancache host RAM, CPU, Disk issue ...

How are you running the container(s)

Case 1
Container in OMV VM, OMV VM in PVE
monolithic container using host network

Case 2
Container in LXC container in PVE

Case 3
Container in Ubuntu on another laptop bare metal
monolithic container using docker0 network

Same results as described.

DNS Configuration

Unbound on pfsense as main DNS with custom DNS settings like below covering all uklancache defined domains
local-zone: "cloudflare.epicgamescdn.com" redirect
local-data: "cloudflare.epicgamescdn.com 3600 IN A 192.168.0.2"

Output of container(s)

Refer to error log entries in the beginning

Any help appreciated. TIA

coddec commented 3 weeks ago

For record. Disabling nginx buffer makes the 0KB time form 10 seconds to none, however nothing is cached.

/etc/nginx/sites-enabled/20_upstream.conf

 Upstream server to proxy and handle inconsistent 302 redirects
# All cache traffic is passed through this proxy to allow rewriting of redirects without caching

# This is particularly important for sony / ps5 as upstreams redirect between them which confuses slice map on caching

server {

  # Internal bind on 3128, this should not be externally mapped
  listen localhost:3128 reuseport;
proxy_request_buffering off;
proxy_buffering off;

  # No access_log tracking as all requests to this instance are already logged through monolithic

  # access_log /data/logs/upstream-access.log cachelog;
  error_log /data/logs/upstream-error.log;

  include /etc/nginx/sites-available/upstream.conf.d/*.conf;
}
/etc/nginx/sites-enabled/10_generic.conf

# Primary monolithic cache engine

server {
  listen 80 reuseport;
proxy_request_buffering off;
proxy_buffering off;

  access_log /data/logs/access.log cachelog;
  error_log /data/logs/error.log;

  include /etc/nginx/sites-available/cache.conf.d/*.conf;
}
Redrrx commented 3 weeks ago

I second this issue, same situation right now.

coddec commented 3 weeks ago

Tried every fix I can think of. None worked. Squid forward proxy can easily cache epic downloads, though default config will have cache miss problem if the CDN changes without storid program. Squid can't cache Steam by default, as steam uses https by default. Can't point lancache.steamcontent.com to squid as it is in forward proxy mode. Hope lancache can get it fixed, so that we can still use it for game caching. Thanks,

AlexanderSalge commented 1 week ago

also having this issue

Redrrx commented 1 week ago

also having this issue

Tried every fix I can think of. None worked. Squid forward proxy can easily cache epic downloads, though default config will have cache miss problem if the CDN changes without storid program. Squid can't cache Steam by default, as steam uses https by default. Can't point lancache.steamcontent.com to squid as it is in forward proxy mode. Hope lancache can get it fixed, so that we can still use it for game caching. Thanks,

Disabled nginx slicing on my end i had little improvements but its still not reliable as it used to be unfortunately.

AlexanderSalge commented 1 week ago

Do they have any version tags to use other than latest?

Edit1: Switching to a "latest" release from couple of ~weeks~ months ago doesn't fix it either. Since lancache doesn't do version tags I used the only other sha256 I could find in a hurry from the truecharts release, where they switched to upstream images instead of their own: https://github.com/truecharts/charts/commit/e6a70eef32f584e2ca86c54ae5ba6fb0a7067c39

here the tag i used: lancachenet/monolithic:latest@sha256:b72d6b909b9e3fb7b521e90aab97479f7977bf6bee97e89a095e1afdbd6d3b85

Edit2: same with 7077061f078ebf2208a0906153cf13f9a979f47cab19619b28c47b35b3d8eaf5 and with tpill90's version with updated base images. He's got a PR open for those changes (https://github.com/lancachenet/monolithic/pull/191)