search

lancachenet / monolithic

A monolithic lancache service capable of caching all CDNs in a single instance
https://hub.docker.com/r/lancachenet/monolithic
Other
755 stars 77 forks source link

Error upon running lancachenet/monolithic #36

Closed Nuc1eoN closed 4 years ago

Nuc1eoN commented 5 years ago

I have just followed the first step described on lancache.net docker run --restart unless-stopped --name lancache -v /cache/data:/data/cache -v /cache/logs:/data/logs -p 80:80 lancachenet/monolithic:latest and I am getting the following error messages at the bottom:

Checking nginx config
nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 32
nginx: configuration file /etc/nginx/nginx.conf test failed
ERROR: hook /hooks/supervisord-pre.d/99_config_check.sh} returned a non-zero exit status '0'

Am I doing something wrong?

VibroAxe commented 5 years ago

Can we have the full log please?

markusressel commented 4 years ago

I have the same issue. I want to setup lancache using lancache-dns as well as the sniproxy in a Rancher managed Kubernetes cluster. While the dns and sniproxy containers start up fine, the main lancache container does not.

I saw #38 but I use the latest version of docker (see below) so I doubt that this is the reason.

The volume mounted to /data/cache is a NFS-Share, but chown should work over NFS right?

Docker Version: Docker version 19.03.5-ce, build 633a0ea838 Rancher Version: v2.3.3

    *.hsar.steampowered.com.edgesuite.net steam; 
    *.akamai.steamstatic.com steam; 
    content-origin.steampowered.com steam; 
    clientconfig.akamai.steamtransparent.com steam; 
    steampipe.akamaized.net steam; 
    edgecast.steamstatic.com steam; 
    steam.apac.qtlglb.com.mwcloudcdn.com steam; 
    *.cs.steampowered.com steam; 
    *.cm.steampowered.com steam; 
    *.edgecast.steamstatic.com steam; 
    *.steamcontent.com steam; 
    cdn1-sea1.valve.net steam; 
    cdn2-sea1.valve.net steam; 
    *.steam-content-dnld-1.apac-1-cdn.cqloud.com steam; 
    *.steam-content-dnld-1.eu-c1-cdn.cqloud.com steam; 
    steam.apac.qtlglb.com steam; 
    edge.steam-dns.top.comcast.net steam; 
    edge.steam-dns-2.top.comcast.net steam; 
    steam.naeu.qtlglb.com steam; 
    steampipe-kr.akamaized.net steam; 
    steam.ix.asn.au steam; 
    steam.eca.qtlglb.com steam; 
    steam.cdn.on.net steam; 
    update5.dota2.wmsj.cn steam; 
    update2.dota2.wmsj.cn steam; 
    update6.dota2.wmsj.cn steam; 
    update3.dota2.wmsj.cn steam; 
    update1.dota2.wmsj.cn steam; 
    update4.dota2.wmsj.cn steam; 
    update5.csgo.wmsj.cn steam; 
    update2.csgo.wmsj.cn steam; 
    update4.csgo.wmsj.cn steam; 
    update3.csgo.wmsj.cn steam; 
    update6.csgo.wmsj.cn steam; 
    update1.csgo.wmsj.cn steam; 
    st.dl.bscstorage.net steam; 
    cdn.mileweb.cs.steampowered.com.8686c.com steam; 
    *.cdn.ubi.com uplay; 
    live.patcher.elderscrollsonline.com teso; 
    d3rmjivj4k4f0t.cloudfront.net twitch; 
    addons.forgesvc.net twitch; 
    media.forgecdn.net twitch; 
    files.forgecdn.net twitch; 
    content.warframe.com warframe; 
    dl1.wargaming.net wargaming; 
    dl2.wargaming.net wargaming; 
    wg.gcdn.co wargaming; 
    wgusst-na.wargaming.net wargaming; 
    wgusst-eu.wargaming.net wargaming; 
    update-v4r4h10x.worldofwarships.com wargaming; 
    wgus-wotasia.wargaming.net wargaming; 
    dl-wot-ak.wargaming.net wargaming; 
    dl-wot-gc.wargaming.net wargaming; 
    dl-wot-se.wargaming.net wargaming; 
    dl-wot-cdx.wargaming.net wargaming; 
    dl-wows-ak.wargaming.net wargaming; 
    dl-wows-gc.wargaming.net wargaming; 
    dl-wows-se.wargaming.net wargaming; 
    dl-wows-cdx.wargaming.net wargaming; 
    dl-wowp-ak.wargaming.net wargaming; 
    dl-wowp-gc.wargaming.net wargaming; 
    dl-wowp-se.wargaming.net wargaming; 
    dl-wowp-cdx.wargaming.net wargaming; 
    officecdn.microsoft.com wsus; 
    *.windowsupdate.com wsus; 
    windowsupdate.com wsus; 
    *.dl.delivery.mp.microsoft.com wsus; 
    dl.delivery.mp.microsoft.com wsus; 
    *.update.microsoft.com wsus; 
    *.do.dsp.mp.microsoft.com wsus; 
    *.microsoft.com.edgesuite.net wsus; 
    amupdatedl.microsoft.com wsus; 
    amupdatedl2.microsoft.com wsus; 
    amupdatedl3.microsoft.com wsus; 
    amupdatedl4.microsoft.com wsus; 
    amupdatedl5.microsoft.com wsus; 
    assets1.xboxlive.com xboxlive; 
    assets2.xboxlive.com xboxlive; 
    dlassets.xboxlive.com xboxlive; 
    xboxone.loris.llnwd.net xboxlive; 
    *.xboxone.loris.llnwd.net xboxlive; 
    xboxone.vo.llnwd.net xboxlive; 
    xbox-mbr.xboxlive.com xboxlive; 
    assets1.xboxlive.com.nsatc.net xboxlive; 
} 
Executing hook /hooks/entrypoint-pre.d/19_doc_root_setup 
/var/www/html already exists. 
Setting document root to /var/www/html 
Executing hook /hooks/entrypoint-pre.d/20_perms_check.sh 
Running fast permissions check 
Fast permissions check successful, if you have any permissions error try running with -e FORCE_PERMS_CHECK = true 
Executing hook /hooks/entrypoint-pre.d/20_ssl_setup 
Not enabling SSL as neither key nor cert provided. 
Executing hook /hooks/supervisord-pre.d/20_test_files_setup 
Checking if /var/www/html is empty - Directory not empty.. don't touch content 
Executing hook /hooks/supervisord-pre.d/21_cleanup_log_files 
Cleaning up log files older than 3560 days 
Executing hook /hooks/supervisord-pre.d/99_config_check.sh 
Currently configured config: 
NGINX CONFIG DUMP FOR /etc/nginx/nginx.conf 
# Including /etc/nginx/nginx.conf 
user www-data; 
# Including /etc/nginx/workers.conf 
worker_processes auto; 
# Finished including /etc/nginx/workers.conf 
pid /run/nginx.pid; 
events { 
worker_connections 4096; 
multi_accept on; 
use epoll; 
} 

http { 
aio threads; 
sendfile on; 
tcp_nopush on; 
tcp_nodelay on; 
keepalive_timeout 65; 
types_hash_max_size 2048; 

# Including /etc/nginx/mime.types 

types { 
text/html html htm shtml; 
text/css css; 
text/xml xml; 
image/gif gif; 
image/jpeg jpeg jpg; 
application/javascript js; 
application/atom+xml atom; 
application/rss+xml rss; 

text/mathml mml; 
text/plain txt; 
text/vnd.sun.j2me.app-descriptor jad; 
text/vnd.wap.wml wml; 
text/x-component htc; 

image/png png; 
image/tiff tif tiff; 
image/vnd.wap.wbmp wbmp; 
image/x-icon ico; 
image/x-jng jng; 
image/x-ms-bmp bmp; 
image/svg+xml svg svgz; 
image/webp webp; 

application/font-woff woff; 
application/java-archive jar war ear; 
application/json json; 
application/mac-binhex40 hqx; 
application/msword doc; 
application/pdf pdf; 
application/postscript ps eps ai; 
application/rtf rtf; 
application/vnd.apple.mpegurl m3u8; 
application/vnd.ms-excel xls; 
application/vnd.ms-fontobject eot; 
application/vnd.ms-powerpoint ppt; 
application/vnd.wap.wmlc wmlc; 
application/vnd.google-earth.kml+xml kml; 
application/vnd.google-earth.kmz kmz; 
application/x-7z-compressed 7z; 
application/x-cocoa cco; 
application/x-java-archive-diff jardiff; 
application/x-java-jnlp-file jnlp; 
application/x-makeself run; 
application/x-perl pl pm; 
application/x-pilot prc pdb; 
application/x-rar-compressed rar; 
application/x-redhat-package-manager rpm; 
application/x-sea sea; 
application/x-shockwave-flash swf; 
application/x-stuffit sit; 
application/x-tcl tcl tk; 
application/x-x509-ca-cert der pem crt; 
application/x-xpinstall xpi; 
application/xhtml+xml xhtml; 
application/xspf+xml xspf; 
application/zip zip; 

application/octet-stream bin exe dll; 
application/octet-stream deb; 
application/octet-stream dmg; 
application/octet-stream iso img; 
application/octet-stream msi msp msm; 

application/vnd.openxmlformats-officedocument.wordprocessingml.document docx; 
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx; 
application/vnd.openxmlformats-officedocument.presentationml.presentation pptx; 

audio/midi mid midi kar; 
audio/mpeg mp3; 
audio/ogg ogg; 
audio/x-m4a m4a; 
audio/x-realaudio ra; 

video/3gpp 3gpp 3gp; 
video/mp2t ts; 
video/mp4 mp4; 
video/mpeg mpeg mpg; 
video/quicktime mov; 
video/webm webm; 
video/x-flv flv; 
video/x-m4v m4v; 
video/x-mng mng; 
video/x-ms-asf asx asf; 
video/x-ms-wmv wmv; 
video/x-msvideo avi; 
} 
# Finished including /etc/nginx/mime.types 
default_type application/octet-stream; 

access_log /var/log/nginx/access.log; 
error_log /var/log/nginx/error.log; 

gzip on; 

# Including /etc/nginx/conf.d/10_log_format.conf 
log_format cachelog '[$cacheidentifier] $remote_addr / $http_x_forwarded_for - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_cache_status" "$host" "$http_range"'; 

# Finished including /etc/nginx/conf.d/10_log_format.conf 
# Including /etc/nginx/conf.d/20_proxy_cache_path.conf 
proxy_cache_path /data/cache/cache levels=2:2 keys_zone=generic:500m inactive=200d max_size=500000m loader_files=1000 loader_sleep=50ms loader_threshold=300ms use_temp_path=off; 
# Finished including /etc/nginx/conf.d/20_proxy_cache_path.conf 
# Including /etc/nginx/conf.d/30_maps.conf 
map $http_host $cacheidentifier { 
hostnames; 
default $http_host; 
assetcdn.101.arenanetworks.com arenanet; 
assetcdn.102.arenanetworks.com arenanet; 
assetcdn.103.arenanetworks.com arenanet; 
live.patcher.bladeandsoul.com arenanet; 
dist.blizzard.com blizzard; 
dist.blizzard.com.edgesuite.net blizzard; 
llnw.blizzard.com blizzard; 
edgecast.blizzard.com blizzard; 
blizzard.vo.llnwd.net blizzard; 
blzddist1-a.akamaihd.net blizzard; 
blzddist2-a.akamaihd.net blizzard; 
blzddist3-a.akamaihd.net blizzard; 
blzddist4-a.akamaihd.net blizzard; 
level3.blizzard.com blizzard; 
nydus.battle.net blizzard; 
edge.blizzard.top.comcast.net blizzard; 
cdn.blizzard.com blizzard; 
*.cdn.blizzard.com blizzard; 
cdn.homecomingservers.com cityofheroes; 
#Daybreakgames daybreak; 
# daybreak; 
#DoNOTcachemanifest.patch.daybreakgames.com daybreak; 
# daybreak; 
#PS2 daybreak; 
pls.patch.daybreakgames.com daybreak; 
epicgames-download1.akamaized.net epicgames; 
download.epicgames.com epicgames; 
download2.epicgames.com epicgames; 
download3.epicgames.com epicgames; 
download4.epicgames.com epicgames; 
cdn.zaonce.net frontier; 
hirez.http.internapcdn.net hirez; 
filedelivery.nexusmods.com nexusmods; 
level3.nwhttppatch.crypticstudios.com neverwinter; 
ccs.cdn.wup.shop.nintendo.com nintendo; 
ccs.cdn.wup.shop.nintendo.net nintendo; 
ccs.cdn.wup.shop.nintendo.net.edgesuite.net nintendo; 
geisha-wup.cdn.nintendo.net nintendo; 
geisha-wup.cdn.nintendo.net.edgekey.net nintendo; 
idbe-wup.cdn.nintendo.net nintendo; 
idbe-wup.cdn.nintendo.net.edgekey.net nintendo; 
ecs-lp1.hac.shop.nintendo.net nintendo; 
receive-lp1.dg.srv.nintendo.net nintendo; 
*.wup.shop.nintendo.net nintendo; 
*.wup.eshop.nintendo.net nintendo; 
*.hac.lp1.d4c.nintendo.net nintendo; 
*.hac.lp1.eshop.nintendo.net nintendo; 
rxp-fl.cncirc.net renegadex; 
rxp-chi.cncirc.net renegadex; 
rxp-nz.cncirc.net renegadex; 
rxp-bgr.cncirc.net renegadex; 
rxp-fr.cncirc.net renegadex; 
rxp-nyc.cncirc.net renegadex; 
rxp-uk.cncirc.net renegadex; 
rxp-sg.cncirc.net renegadex; 
rxp-la.cncirc.net renegadex; 
rxp-fin.cncirc.net renegadex; 
denver1.renegade-x.com renegadex; 
seattle1.renegade-x.com renegadex; 
l3cdn.riotgames.com riot; 
worldwide.l3cdn.riotgames.com riot; 
riotgamespatcher-a.akamaihd.net riot; 
riotgamespatcher-a.akamaihd.net.edgesuite.net riot; 
*.dyn.riotcdn.net riot; 
patches.rockstargames.com rockstar; 
pls.patch.station.sony.com sony; 
gs2.ww.prod.dl.playstation.net sony; 
gs2.sonycoment.loris-e.llnwd.net sony; 
*.content.steampowered.com steam; 
content1.steampowered.com steam; 
content2.steampowered.com steam; 
content3.steampowered.com steam; 
content4.steampowered.com steam; 
content5.steampowered.com steam; 
content6.steampowered.com steam; 
content7.steampowered.com steam; 
content8.steampowered.com steam; 
cs.steampowered.com steam; 
steamcontent.com steam; 
client-download.steampowered.com steam; 
*.hsar.steampowered.com.edgesuite.net steam; 
*.akamai.steamstatic.com steam; 
content-origin.steampowered.com steam; 
clientconfig.akamai.steamtransparent.com steam; 
steampipe.akamaized.net steam; 
edgecast.steamstatic.com steam; 
steam.apac.qtlglb.com.mwcloudcdn.com steam; 
*.cs.steampowered.com steam; 
*.cm.steampowered.com steam; 
*.edgecast.steamstatic.com steam; 
*.steamcontent.com steam; 
cdn1-sea1.valve.net steam; 
cdn2-sea1.valve.net steam; 
*.steam-content-dnld-1.apac-1-cdn.cqloud.com steam; 
*.steam-content-dnld-1.eu-c1-cdn.cqloud.com steam; 
steam.apac.qtlglb.com steam; 
edge.steam-dns.top.comcast.net steam; 
edge.steam-dns-2.top.comcast.net steam; 
steam.naeu.qtlglb.com steam; 
steampipe-kr.akamaized.net steam; 
steam.ix.asn.au steam; 
steam.eca.qtlglb.com steam; 
steam.cdn.on.net steam; 
update5.dota2.wmsj.cn steam; 
update2.dota2.wmsj.cn steam; 
update6.dota2.wmsj.cn steam; 
update3.dota2.wmsj.cn steam; 
update1.dota2.wmsj.cn steam; 
update4.dota2.wmsj.cn steam; 
update5.csgo.wmsj.cn steam; 
update2.csgo.wmsj.cn steam; 
update4.csgo.wmsj.cn steam; 
update3.csgo.wmsj.cn steam; 
update6.csgo.wmsj.cn steam; 
update1.csgo.wmsj.cn steam; 
st.dl.bscstorage.net steam; 
cdn.mileweb.cs.steampowered.com.8686c.com steam; 
*.cdn.ubi.com uplay; 
live.patcher.elderscrollsonline.com teso; 
d3rmjivj4k4f0t.cloudfront.net twitch; 
addons.forgesvc.net twitch; 
media.forgecdn.net twitch; 
files.forgecdn.net twitch; 
content.warframe.com warframe; 
dl1.wargaming.net wargaming; 
dl2.wargaming.net wargaming; 
wg.gcdn.co wargaming; 
wgusst-na.wargaming.net wargaming; 
wgusst-eu.wargaming.net wargaming; 
update-v4r4h10x.worldofwarships.com wargaming; 
wgus-wotasia.wargaming.net wargaming; 
dl-wot-ak.wargaming.net wargaming; 
dl-wot-gc.wargaming.net wargaming; 
dl-wot-se.wargaming.net wargaming; 
dl-wot-cdx.wargaming.net wargaming; 
dl-wows-ak.wargaming.net wargaming; 
dl-wows-gc.wargaming.net wargaming; 
dl-wows-se.wargaming.net wargaming; 
dl-wows-cdx.wargaming.net wargaming; 
dl-wowp-ak.wargaming.net wargaming; 
dl-wowp-gc.wargaming.net wargaming; 
dl-wowp-se.wargaming.net wargaming; 
dl-wowp-cdx.wargaming.net wargaming; 
officecdn.microsoft.com wsus; 
*.windowsupdate.com wsus; 
windowsupdate.com wsus; 
*.dl.delivery.mp.microsoft.com wsus; 
dl.delivery.mp.microsoft.com wsus; 
*.update.microsoft.com wsus; 
*.do.dsp.mp.microsoft.com wsus; 
*.microsoft.com.edgesuite.net wsus; 
amupdatedl.microsoft.com wsus; 
amupdatedl2.microsoft.com wsus; 
amupdatedl3.microsoft.com wsus; 
amupdatedl4.microsoft.com wsus; 
amupdatedl5.microsoft.com wsus; 
assets1.xboxlive.com xboxlive; 
assets2.xboxlive.com xboxlive; 
dlassets.xboxlive.com xboxlive; 
xboxone.loris.llnwd.net xboxlive; 
*.xboxone.loris.llnwd.net xboxlive; 
xboxone.vo.llnwd.net xboxlive; 
xbox-mbr.xboxlive.com xboxlive; 
assets1.xboxlive.com.nsatc.net xboxlive; 
} 
# Finished including /etc/nginx/conf.d/30_maps.conf 

# Including /etc/nginx/sites-enabled/10_generic.conf 

server { 
listen 80 reuseport; 

access_log /data/logs/access.log cachelog; 
error_log /data/logs/error.log; 

# Including /etc/nginx/sites-available/generic.conf.d/10_generic.conf 
resolver 8.8.8.8 8.8.4.4 ipv6=off; 

location / { 

# Including /etc/nginx/sites-available/generic.conf.d/root/10_loop_detection.conf 
# Abort any circular requests 
if ($http_X_LanCache_Processed_By = $hostname) { 
return 508; 
} 

proxy_set_header X-LanCache-Processed-By $hostname; 
add_header X-LanCache-Processed-By $hostname,$http_X_LanCache_Processed_By; 
# Finished including /etc/nginx/sites-available/generic.conf.d/root/10_loop_detection.conf 
# Including /etc/nginx/sites-available/generic.conf.d/root/20_cache.conf 
# Cache Location 
slice 1m; 
proxy_cache generic; 

proxy_ignore_headers Expires Cache-Control; 
proxy_cache_valid 200 206 3560d; 
proxy_set_header Range $slice_range; 

# Only download one copy at a time and use a large timeout so 
# this really happens, otherwise we end up wasting bandwith 
# getting the file multiple times. 
proxy_cache_lock on; 
proxy_cache_lock_timeout 1h; 

# Allow the use of state entries 
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; 

# Allow caching of 200 but not 301 or 302 as our cache key may not include query params 
# hence may not be valid for all users 
proxy_cache_valid 301 302 0; 

# Enable cache revalidation 
proxy_cache_revalidate on; 

# Don't cache requests marked as nocache=1 
proxy_cache_bypass $arg_nocache; 

# 40G max file 
proxy_max_temp_file_size 40960m; 

# Finished including /etc/nginx/sites-available/generic.conf.d/root/20_cache.conf 
# Including /etc/nginx/sites-available/generic.conf.d/root/30_cache_key.conf 
proxy_cache_key $cacheidentifier$uri$slice_range; 
# Finished including /etc/nginx/sites-available/generic.conf.d/root/30_cache_key.conf 
# Including /etc/nginx/sites-available/generic.conf.d/root/40_etags.conf 
# Battle.net Fix 
proxy_hide_header ETag; 

# Finished including /etc/nginx/sites-available/generic.conf.d/root/40_etags.conf 
# Including /etc/nginx/sites-available/generic.conf.d/root/90_upstream.conf 
# Upstream Configuration 
proxy_next_upstream error timeout http_404; 
proxy_pass http://$host$request_uri; 
proxy_redirect off; 
proxy_ignore_client_abort on; 

# Upstream request headers 
proxy_set_header Host $host; 
proxy_set_header X-Real-IP $remote_addr; 
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
# Finished including /etc/nginx/sites-available/generic.conf.d/root/90_upstream.conf 
# Including /etc/nginx/sites-available/generic.conf.d/root/99_debug_header.conf 
# Debug Headers 
add_header X-Upstream-Status $upstream_status; 
add_header X-Upstream-Response-Time $upstream_response_time; 
add_header X-Upstream-Cache-Status $upstream_cache_status; 
# Finished including /etc/nginx/sites-available/generic.conf.d/root/99_debug_header.conf 

} 

# Fix for League of Legends Updater 
location ~ ^.+(releaselisting_.*|.version$) { 
proxy_pass http://$host; 
} 

location = /steamcache-heartbeat { 
add_header X-LanCache-Processed-By $hostname; 
return 204; 
} 
# Finished including /etc/nginx/sites-available/generic.conf.d/10_generic.conf 
} 
# Finished including /etc/nginx/sites-enabled/10_generic.conf 
} 
# Finished including /etc/nginx/nginx.conf 
Checking nginx config 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok 
nginx: [emerg] chown("/data/cache/cache", 33) failed (22: Invalid argument) 
nginx: configuration file /etc/nginx/nginx.conf test failed 
ERROR: hook /hooks/supervisord-pre.d/99_config_check.sh} returned a non-zero exit status '0'
markusressel commented 4 years ago

Because I found a couple of sites that mention the inability to chown via nfs shares I tried it using a locally mounted volume and with this configuration it does not give this error.

My question would be: is this chown necessary?

VibroAxe commented 4 years ago

@markusressel your issue is probably a root_squash issue with NFS. Please reexport with no root squash and you'll likely see this start working. And yes, the chown is nessecary because the system has detected invalid permissions

If it doesnt can you raise a new issue. This is a hashmap directive issue we need to look at

markusressel commented 4 years ago

I think the problem with my NFS share is that I used NFSv4 and not NFSv3.

The machine that is hosting the storage that is mounted via NFS is a FreeNAS system. Despite trying a number of combinations of possible configuration options I was not able to use a chown command from my laptop when mounting the share using NFSv4. For a quick test I disabled the Enable NFSv4 checkbox in FreeNas, remounted the share on my laptop and voila, I was able to chown. I'm not that well informed about the differences between NFSv3 and NFSv4 except the fact that there are lots of them. Since I can not easily switch my whole setup to NFSv3 I will resort to using the local drive for now.

Hope this helps and thx for the quick response.

unspec commented 4 years ago

Closing as the original issue is inactive and Markus is now working.

May require a new issue for NFSv4 matters, but that may be out of scope at this time.