search

lancachenet / monolithic

A monolithic lancache service capable of caching all CDNs in a single instance
https://hub.docker.com/r/lancachenet/monolithic
Other
726 stars 73 forks source link

Not working on Unraid #104

Closed sonuyos closed 4 years ago

sonuyos commented 4 years ago

Describe the issue you are having

I have downloaded lancache using community apps on my unraid server, however, i saw 2 problems.

1 - Origin does not work, it doesn't go through lancache, no matter what, i tried steamcache, and it worked there, however it doesn't work on lancache, when i ping to origin-a.akamaihd.net, it gives me the real server ip, however with steamcache, it works and gives me ip of steamcache, which is local.

2 - Now this is where it completely breaks, both lancache and steamcache. It shows that my steam games are caching, and i can see some files are there too, but when i try to delete and re-dwonload it just doesn't work. I am not sure what i am doing wrong, but both same. It redownloads from the internet instead of unraid.

How are you running the container(s)

In Unraid

DNS Configuration

Set it to 192.168.1.202, and touched nothing, doesnt work.

Output of container(s)

Cannot find it in unraid, will gladly post if anyone can point it out to me.
unspec commented 4 years ago

Hi,

For question 1, yes, that is expected. Origin have enabled https downloads which breaks all attempts at caching. They have a potential http downgrade option being worked on at the moment (and it is in the origin beta client) but origin is disabled in lancache until they release this - monitor https://github.com/uklans/cache-domains/issues/98 for updates.

I'm afraid I'm not familar with unraid, so i can't comment on how it does / doesn't work. We do have a note on it here: http://lancache.net/docs/common-issues/#unraid-port-bindings because its container implementation tends to break the required docker networking.

From your pc, if you do an nslookup lancache.steamcontent.com do you correctly get the IP of your lancache?

When you say "It shows that my steam games are caching", what do you mean by "it"? We really need the logs from the container to know what it is doing.

sonuyos commented 4 years ago

Hello,

I think origin did it, as I can see the cache in steam cache docker.

There is a recent release on community docker for lancache.

The lookup works correct.

It shows me the local ip.

By the statement you quoted, I mean that I can see the files being downloaded by the cache but does not download from cache when I reinstall.,

unspec commented 4 years ago

Hopefully somebody with some unraid experience may be able to help you troubleshoot running the containers in unraid.

In the mean time i would suggest following the advise here: http://lancache.net/docs/common-issues/#unraid-port-bindings and setting up a new ubuntu vm and then using our quick start guide http://lancache.net/docs/ to easily setup the required containers within it via docker-compose.

That should allow you to validate that lancache works and meets your needs in the meantime.

sonuyos commented 4 years ago

I was able to find the logs - http://www.filedropper.com/lancache

Alright, i setup ubuntu in my unraid and then used the website script.

however it still doesnt work.

I mean it works better than unraid, like i downloaded superhot and it gave 79mb/s, but thats for 650mb game, but when i download boneworks it just doesnt go over 15mb/s, that game is of 5.5gb.

I have only downloaded these 2 games. i can see my lancache folder is of 4.6gb, however total size of game is of 6.1GB,

I am confused.

------EDIT----- Now i deleted Superhot and tried to redownload, not going over 7mb/s.

This is very flicky.

unspec commented 4 years ago

What does your cache log say? There will be a logs directory in your cache root in the VM (there was an option to set CACHE_ROOT in the .env file but it defaults to ./lancache/logs/ from wherever you ran the git command). The access.log and error.log will show what lancache is doing for each request its getting.

Does it show lots of HIT or lots of MISS entries when you try to download a game for the second time? (The first time will be all MISS as the game is not yet in cache).

If you are correctly getting HITs the second time you download a game and its still slow, then its likely down to the hardware you are using. What sort of disks are in your server? What speed is your network etc. These containers are capable of serving thousands of gamers at the same time on the right (very high powered) hardware: http://lancache.net/docs/hardware/

sonuyos commented 4 years ago

Actually i am not the only one.

Here - https://www.reddit.com/r/DataHoarder/comments/f8w6kd/steamcache_not_working_on_unraid/fiq1is1/?context=3

Same problem having to someone else.

As for logs, they are pretty much normal.

in access.log it shows HIT and MISS both the second time it downloads. And its very flicky.

I think when i start steam and download it afresh (as in the cache is already there) it downloads fast, but after sometime it doesnt, it downloads off internet. (Though i cannot verify).

I am really confused.

If you want i can show you on teamviewer.

MathewBurnett commented 4 years ago

The lancache-bundle app on unraid is not affiliated with this project. A third party has wrapped it up as an unraid app. Go see https://github.com/Josh5/lancache-bundle.

I use unraid at home and I am aware in its default storage configuration that disks are quite slow by caching standards as its setup for long term resilient storage. I would advise you to seek advice from the app maintainer as its very hard for us to know what unraid specific things they are doing.

sonuyos commented 4 years ago

The lancache-bundle app on unraid is not affiliated with this project. A third party has wrapped it up as an unraid app. Go see https://github.com/Josh5/lancache-bundle.

I use unraid at home and I am aware in its default storage configuration that disks are quite slow by caching standards as its setup for long term resilient storage. I would advise you to seek advice from the app maintainer as its very hard for us to know what unraid specific things they are doing.

I have tried that, as well as ubuntu, both are same.

MathewBurnett commented 4 years ago

how does dns work in your network?

unspec commented 4 years ago

And do you have ipv6 enabled on your network?

sonuyos commented 4 years ago

And do you have ipv6 enabled on your network?

Disabled it too, still same :(

sonuyos commented 4 years ago

how does dns work in your network?

I do not understand your question.

MathewBurnett commented 4 years ago

Your client PC gets its dns server to use from something in your network. Have you set that manually? is it handed out by DHCP? do you have muliple dns servers set?

sonuyos commented 4 years ago

Hello

I have my router set to 8.8.8.8 and 8.8.4.4. but when using cache I set the dns manually in networks setting. 192.168.1.xxx - ip of cache and 2nd box I put 8.8.4.4.

Am I doing anything wrong?

On Thu, 27 Feb 2020, 15:46 Proto, notifications@github.com wrote:

Your client PC gets its dns server to use from something in your network. Have you set that manually? is it handed out by DHCP? do you have muliple dns servers set?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lancachenet/monolithic/issues/104?email_source=notifications&email_token=AA7I2PUPCJH2N4SHYHQRVU3RE6HI7A5CNFSM4K2OAWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENDYZ5Y#issuecomment-591891703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7I2PSSLIWJ7RXBX4JFXVTRE6HI7ANCNFSM4K2OAWEQ .

MathewBurnett commented 4 years ago

and xxx is the ubuntu vm ip?

sonuyos commented 4 years ago

Yes, Ubuntu as well as unraid, i tried both, both same results.

sonuyos commented 4 years ago

Update - https://prnt.sc/r8m01d

Downloaded 350mb in steam, but on lancache only 45mb.

MathewBurnett commented 4 years ago

show me the dns servers in the output of ipconfig /all

sonuyos commented 4 years ago

https://prnt.sc/r8malg here

show me the dns servers in the output of ipconfig /all

MathewBurnett commented 4 years ago

you have two dns servers set so it will use both, hence some of your requests don't hit the cache. Stop using 8.8.8.8 on your client

sonuyos commented 4 years ago

I cannot enter same IP twice in the DNS. It doesnt allow.

sonuyos commented 4 years ago

Also, i cannot use internet without adding 8.8.8.8.

MathewBurnett commented 4 years ago

you should set your client to talk to the container on 192.168.1.206 and then have the upstream dns on the dns container set to 8.8.8.8 , if you don't do that then you will not be using the container to get your dns all of the time and it won't work.

sonuyos commented 4 years ago

Upstream dns is already set to 1.1.1.1 and it still doesn't work.

On Thu, 27 Feb 2020, 23:43 Proto, notifications@github.com wrote:

you should set your client to talk to the container on 192.168.1.206 and then have the upstream dns on the dns container set to 8.8.8.8 , if you don't do that then you will not be using the container to get your dns all of the time and it won't work.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lancachenet/monolithic/issues/104?email_source=notifications&email_token=AA7I2PSRGEJOHOC6AZLFO73RE77DJA5CNFSM4K2OAWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFL47Q#issuecomment-592100990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7I2PQPYMYY6G75MSDVHR3RE77DJANCNFSM4K2OAWEQ .

MathewBurnett commented 4 years ago

Lets have one last look at your setup ... 1) Are you are running the docker-compose on a vm now? 2) If so can you show me the env file? 3) can you also show me docker ps?

sonuyos commented 4 years ago

Nah i removed the vm as i gave up on that.

Right now its on Unraid only.

https://ibb.co/MVDhRwj

Config on unraid docker

On Thu, 27 Feb 2020, 23:52 Proto, notifications@github.com wrote:

Lets have one last look at your setup ...

  1. Are you are running the docker-compose on a vm now?
  2. If so can you show me the env file?
  3. can you also show me docker ps?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lancachenet/monolithic/issues/104?email_source=notifications&email_token=AA7I2PXXMSUPHIRPUNGHFIDRFAAFPA5CNFSM4K2OAWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFM5WI#issuecomment-592105177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7I2PV57AUT3OOGFAYJBM3RFAAFPANCNFSM4K2OAWEQ .

MathewBurnett commented 4 years ago

ok then we are back to pointing out the specifics of the lancache-bundle are nothing to do with us see (https://github.com/lancachenet/monolithic/issues/104#issuecomment-590918798) and we don't know what it may or may not be doing. You clearly have a dns issue in your network.

sonuyos commented 4 years ago

I can try to install ubuntu and try with it. But it gives same result.

If I have DNS issue then it should be visible too right, I just can't find any issue.

On Fri, 28 Feb 2020, 00:00 Proto, notifications@github.com wrote:

ok then we are back to pointing out the specifics of the lancache-bundle are nothing to do with us see (#104 (comment) https://github.com/lancachenet/monolithic/issues/104#issuecomment-590918798) and we don't know what it may or may not be doing. You clearly have a dns issue in your network.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lancachenet/monolithic/issues/104?email_source=notifications&email_token=AA7I2PTCSPY7VBEAGZE3GV3RFABEBA5CNFSM4K2OAWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFN2VQ#issuecomment-592108886, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7I2PWZMRHASJNJ3O2JLRTRFABEBANCNFSM4K2OAWEQ .

MathewBurnett commented 4 years ago

your issue is that you have multiple client dns servers defined and that you don't have a correctly configured dns container that you can talk to that can talk out to its upstream dns.

sonuyos commented 4 years ago

But I have even checked linus' setup and even there they used 1.1.1.1 for alternate dns.

As for dns container there is literally nothing I have edited, I have left everything as it is. Also I am not the only one who is facing this issue.. so many people are facing it too.

MathewBurnett commented 4 years ago

As for dns container there is literally nothing I have edited,

you mean you didn't edit someone else's unraid community app?

sonuyos commented 4 years ago

I meant the setting.

It isn't someone else's apps setting. The setting is done by the person who is running the container. And same issue is replicated even in Ubuntu VM.

unspec commented 4 years ago

On your pc:

Do you get any results? If so, what?

sonuyos commented 4 years ago

Here - https://prnt.sc/r8oc8e

unspec commented 4 years ago

Right, so it looks like the container can do upstream lookups ok, so you shouldnt need to add a second dns server to your pc. As mentioned doing that means your pc may end up caching the "none poisoned" dns results for some records.

That can also happen if you are changing your pc's dns servers to/from the container ip manually without flushing dns.

You should now see all of the game download going via the cache.

sonuyos commented 4 years ago

I tired, it still doesnt work.

https://prnt.sc/r8oeol

1.2GB downloaded of boneworks, but shows only 1.4gb on cache (787mb was there from Jurassic World download).

https://prnt.sc/r8ofbf

1.9GB downloaded, cache is 1.6g,

sonuyos commented 4 years ago

There is literally no other dns - https://prnt.sc/r8ogi7

i ipconfig /flushdns too

https://prnt.sc/r8ogxw

It just doesnt want to fuckin work :(

https://prnt.sc/r8oh4p

Not going past 1.6G now.

unspec commented 4 years ago

Did you restart steam and/or your pc after flushing dns? Steam will cache certain IPs internally.

sonuyos commented 4 years ago

Yup

On Fri, 28 Feb 2020, 03:09 unspec, notifications@github.com wrote:

Did you restart steam and/or your pc after flushing dns? Steam will cache certain IPs internally.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lancachenet/monolithic/issues/104?email_source=notifications&email_token=AA7I2PSZRPRMXROM6EYQN23RFAXHNA5CNFSM4K2OAWE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENGBRWQ#issuecomment-592189658, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7I2PUBLS2U4ACVHRB2SRDRFAXHNANCNFSM4K2OAWEQ .

sonuyos commented 4 years ago

Downloaded 950mb from previous.

1GB of Boneworks, yet it remained at 1.2GB

https://prnt.sc/r8tnys

It just doesnt cache.

VibroAxe commented 4 years ago

Can we see the access.log and error.log please?

sonuyos commented 4 years ago

bind() to 0.0.0.0:80 failed is only there cuz i was trying to bind to ip which was in use (Unraid Main IP, didnt work).

error.log

access.log

Lepidopterist commented 4 years ago

Having looked at the access log, it bears absolutely no resemblance to our access.log format. This would lead me to believe that whatever you are running is either (a) not based on any of our work or (b) at least three years old and entirely unsupported.

I would suggest you ask for vendor support from whoever made the Unraid package. We are unable to provide support on this.

sonuyos commented 4 years ago

Funny thing, it was updated yesterday itself. Want me to setup ubuntu and try on that?