search

lancachenet / monolithic

A monolithic lancache service capable of caching all CDNs in a single instance
https://hub.docker.com/r/lancachenet/monolithic
Other
737 stars 73 forks source link

Tried to change environment variables and now it won't start #8

Closed Deanosim closed 5 years ago

Deanosim commented 5 years ago

Describe the issue you are having

Tried to change environment variables and now it won't start

Output of container(s)


Executing hook /hooks/entrypoint-pre.d/19_doc_root_setup

/var/www/html already exists.

Setting document root to /var/www/html

Executing hook /hooks/entrypoint-pre.d/20_perms_check.sh

Checking permissions (This may take a long time if the permissions are incorrect on large caches)...

Permisions ok

Executing hook /hooks/entrypoint-pre.d/20_ssl_setup

Not enabling SSL as neither key nor cert provided.

/init/entrypoint: line 66: /scripts/'/init/supervisord': No such file or directory

Executing hook /hooks/entrypoint-pre.d/10_setup.sh

Executing hook /hooks/entrypoint-pre.d/15_generate_maps.sh

HEAD is now at db4f1f8 Remove images-eds from xboxlive (#50)

Reading cache apple from apple.txt

Reading cache arenanet from arenanet.txt

Reading cache blizzard from blizzard.txt

Reading cache daybreak from daybreak.txt

Reading cache frontier from frontier.txt

Reading cache hirez from hirez.txt

Reading cache nexusmods from nexusmods.txt

Reading cache nintendo from nintendo.txt

Reading cache origin from origin.txt

Reading cache renegadex from renegadex.txt

Reading cache riot from riot.txt

Reading cache rockstar from rockstar.txt

Reading cache sony from sony.txt

Reading cache steam from steam.txt

Reading cache uplay from uplay.txt

Reading cache twitch from twitchapp.txt

Reading cache wargaming from wargaming.net.txt

Reading cache wsus from windowsupdates.txt

Reading cache xboxlive from xboxlive.txt

map $http_host $cacheidentifier {

    hostnames;

    default $http_host;

    assetcdn.101.arenanetworks.com arenanet;

    assetcdn.102.arenanetworks.com arenanet;

    assetcdn.103.arenanetworks.com arenanet;

    dist.blizzard.com blizzard;

    dist.blizzard.com.edgesuite.net blizzard;

    llnw.blizzard.com blizzard;

    edgecast.blizzard.com blizzard;

    blizzard.vo.llnwd.net blizzard;

    blzddist1-a.akamaihd.net blizzard;

    blzddist2-a.akamaihd.net blizzard;

    blzddist3-a.akamaihd.net blizzard;

    blzddist4-a.akamaihd.net blizzard;

    level3.blizzard.com blizzard;

    nydus.battle.net blizzard;

    edge.blizzard.top.comcast.net blizzard;

    cdn.blizzard.com blizzard;

    *.cdn.blizzard.com blizzard;

    # Daybreak games daybreak;

    # daybreak;

    # Do NOT cache manifest.patch.daybreakgames.com daybreak;

    # daybreak;

    #PS2 daybreak;

    pls.patch.daybreakgames.com daybreak;

    ccs.cdn.wup.shop.nintendo.com nintendo;

    pushmo.hac.lp1.eshop.nintendo.net nintendo;

    ecs-lp1.hac.shop.nintendo.net nintendo;

    receive-lp1.dg.srv.nintendo.net nintendo;

    aqua.hac.lp1.d4c.nintendo.net nintendo;

    atum.hac.lp1.d4c.nintendo.net nintendo;

    bugyo.hac.lp1.eshop.nintendo.net nintendo;

    tagaya.hac.lp1.eshop.nintendo.net nintendo;

    # WARNING:  Origin has been seen downloading https client downloads on origin-a.akamaihd.net.  A solution should be in place to forward https to the origin server (eg sniproxy) origin;

    origin-a.akamaihd.net origin;

    akamai.cdn.ea.com origin;

    lvlt.cdn.ea.com origin;

    river.data.ea.com origin;

    origin-a.akamaihd.net.edgesuite.net origin;

    rxp-fl.cncirc.net renegadex;

    rxp-chi.cncirc.net renegadex;

    rxp-nz.cncirc.net renegadex;

    rxp-bgr.cncirc.net renegadex;

    rxp-fr.cncirc.net renegadex;

    rxp-nyc.cncirc.net renegadex;

    rxp-uk.cncirc.net renegadex;

    rxp-sg.cncirc.net renegadex;

    rxp-la.cncirc.net renegadex;

    rxp-fin.cncirc.net renegadex;

    denver1.renegade-x.com renegadex;

    l3cdn.riotgames.com riot;

    worldwide.l3cdn.riotgames.com riot;

    riotgamespatcher-a.akamaihd.net riot;

    riotgamespatcher-a.akamaihd.net.edgesuite.net riot;

    pls.patch.station.sony.com sony;

    gs2.ww.prod.dl.playstation.net sony;

    *.content.steampowered.com steam;

    content1.steampowered.com steam;

    content2.steampowered.com steam;

    content3.steampowered.com steam;

    content4.steampowered.com steam;

    content5.steampowered.com steam;

    content6.steampowered.com steam;

    content7.steampowered.com steam;

    content8.steampowered.com steam;

    cs.steampowered.com steam;

    steamcontent.com steam;

    client-download.steampowered.com steam;

    *.hsar.steampowered.com.edgesuite.net steam;

    *.akamai.steamstatic.com steam;

    content-origin.steampowered.com steam;

    clientconfig.akamai.steamtransparent.com steam;

    steampipe.akamaized.net steam;

    edgecast.steamstatic.com steam;

    steam.apac.qtlglb.com.mwcloudcdn.com steam;

    *.cs.steampowered.com steam;

    *.edgecast.steamstatic.com steam;

    *.steamcontent.com steam;

    cdn1-sea1.valve.net steam;

    cdn2-sea1.valve.net steam;

    *.steam-content-dnld-1.apac-1-cdn.cqloud.com steam;

    steam.apac.qtlglb.com steam;

    edge.steam-dns.top.comcast.net steam;

    edge.steam-dns-2.top.comcast.net steam;

    steamcdn-a.akamaihd.net steam;

    steam.naeu.qtlglb.com steam;

    steampipe-kr.akamaized.net steam;

    steam.ix.asn.au steam;

    steam.eca.qtlglb.com steam;

    steam.cdn.on.net steam;

    *.cdn.ubi.com uplay;

    d3rmjivj4k4f0t.cloudfront.net twitch;

    addons.forgesvc.net twitch;

    media.forgecdn.net twitch;

    files.forgecdn.net twitch;

    dl1.wargaming.net wargaming;

    dl2.wargaming.net wargaming;

    wg.gcdn.co wargaming;

    wgusst-na.wargaming.net wargaming;

    wgusst-eu.wargaming.net wargaming;

    update-v4r4h10x.worldofwarships.com wargaming;

    dl-wows-ak.wargaming.net wargaming;

    wgus-wotasia.wargaming.net wargaming;

    dl-wot-ak.wargaming.net wargaming;

    dl-wot-gc.wargaming.net wargaming;

    dl-wot-cdx.wargaming.net wargaming;

    officecdn.microsoft.com wsus;

    *.windowsupdate.com wsus;

    windowsupdate.com wsus;

    *.dl.delivery.mp.microsoft.com wsus;

    dl.delivery.mp.microsoft.com wsus;

    *.update.microsoft.com wsus;

    *.do.dsp.mp.microsoft.com wsus;

    *.microsoft.com.edgesuite.net wsus;

    assets1.xboxlive.com xboxlive;

    assets2.xboxlive.com xboxlive;

    dlassets.xboxlive.com xboxlive;

    xboxone.loris.llnwd.net xboxlive;

    *.xboxone.loris.llnwd.net xboxlive;

    xboxone.vo.llnwd.net xboxlive;

    xbox-mbr.xboxlive.com xboxlive;

    assets1.xboxlive.com.nsatc.net xboxlive;

}

Executing hook /hooks/entrypoint-pre.d/19_doc_root_setup

/var/www/html already exists.

Setting document root to /var/www/html

Executing hook /hooks/entrypoint-pre.d/20_perms_check.sh

Checking permissions (This may take a long time if the permissions are incorrect on large caches)...
VibroAxe commented 5 years ago

What's your run command?

How did you "change environment variables"?

Deanosim commented 5 years ago

Well I was using portainer to do this but something keeps breaking it for some reason, I use the run command from the readme file to start it the first time and then after that I used portainer to edit it, but the command would have been 

sudo docker run -d --name cache-steam  -e UPSTREAM_DNS="192.168.1.201 1.1.1.1" -e CACHE_DISK_SIZE=2000g -e LOGFILE_RETENTION=90 -v /mnt/steamcache/cache:/data/cache -v /mnt/steamcache/logs:/data/logs -v steamcache_nginx:/etc/nginx -p 80:80 steamcache/monolithic:latest

After removing the container and making a new one using the above command it now works. I'm not sure if I actually need the nginx volume, but I feel like it would loose the information about what it had cached and where and break all the files I had already cached, Am I wrong?

VibroAxe commented 5 years ago

You are indeed wrong, so long as you have the -v /mnt/steamcache/cache:/data/cache retained monolithic will be able to reload all of your cached files. It looks like whatever you did with portainer you broke the init command. Generally for docker if you want to edit the env vars/run command, docker rm then docker run.

Deanosim commented 5 years ago

Okay one more question, would it be possible to run two instances of the steamcache/monolithic and have a load balancer in front of them?

VibroAxe commented 5 years ago

Yes technically.

The full answer is slightly more complicated. If you have sufficient disk space and bandwidth to manage EVERYTHING in duplicate (2 cache stores and twice the download) then yes, you can have two caches running on two different hosts and an haproxy (or similar) load balancer.

HOWEVER, there are a few caveats with this: 1) They CANNOT share the same cache filestore (nginx is very particular about this) 2) You effectively double your bandwidth requirements (if person a requests file from ha, via cache 1 it downloads once, when another person requests the same file from ha, via cache 2, the second cache will then redownload the same file.

There are a few work arounds to the above depending on what you are trying to achieve:

If you are trying to increase your cache performance/throughput you can split the dns such that one "half" of the cache domains (steam) is served from one cache and the other "half" (everything else, inc origin, battle.net) is served from the other cache. This reduces bandwidth duplication and increases overall performance throughput to "the cache"

If you are genuinely trying to have a HA system, then having two full caches is the correct answer, if you still need to reduce your input bandwidth there is an additional (albeit hardware expensive) option: Add a third cache (not on the ha lb) which both primary caches use as an upstream cache. This means that the upstream cache only downloads a file once, and then serves it to the downstream client facing caches. This obviously uses more diskspace (you now need Cx3) but reduces input bandwidth back to I (rather than Ix2). The upstream cache doesn't need to be particularly "fast" so long as it can serve data at least 1:1 with your internet connection you won't see any change in performance over that.

Hope this makes sense, questions on a postcard ;)

VibroAxe commented 5 years ago

@lepidopterist might like to check the above statement and see if he has any additional thoughts or inputs from our last HA discussions i've forgotten