search

steamcmd / api

Read-only API interface for steamcmd app_info
https://www.steamcmd.net
MIT License
56 stars 7 forks source link

[Suggestion] Varnish cache #14

Closed mvaessen closed 3 years ago

mvaessen commented 3 years ago

Hi Jona,

Thanks for your work on the Steam API. It came in very useful. I did notice that the API can become quite slow when it is hit with simultaneous traffic for multiple games. Because of that I placed Varnish in front of the steam api, to make sure only 1 request per game is allowed to go to the backend while the other duplicate requests are hold of by varnish.

I'm not sure if you are familiar with Varnish, so here is a page from the Varnish docs, explaining this particular feature. Perhaps you find this useful for the project (for example adding it by default to the docker file). https://varnish-cache.org/docs/4.0/users-guide/vcl-grace.html

Cheers & thanks! Max

jonakoudijs commented 3 years ago

Hi Max,

Sorry for my late reply. Great to hear you find the API useful!

Yes, that is a problem indeed. The API executes the actual steamcmd binary under the hood and that is not made to be run multiple times within the same instance. A while ago I was looking at splitting the API in separate components so it's easier to have multiple workers that can execute the steamcmd command. This would help with the scaling issues.

I like your solution though. Could you share your configuration perhaps? Perhaps it's a good idea to add this solution to the README or website for now if someone needs this.

mvaessen commented 3 years ago

Hi Jona,

Varnish works pretty much straight out of the box. There are a couple things I changed:

Docker In order to make sure the Docker container is back online after a reboot of crash, I changed the RUN command to auto restart the docker instance like so: docker run --restart unless-stopped --name=api -p 8080:8080 -d steamcmd-api:test. This is great for production setups.

Varnish Out of the box Varnish is pretty well setup. I've used it in the past for setups that endured very high peaks and it is pretty smart on its own. For the API I would generally recommend to leave the Varnish configs as is.

To setup Varnish with the Steam API I used this guide to setup Varnish (I skipped the Apache parts) https://linuxhint.com/varnish_cache_ubuntu_1804/. You are not limited by Ubuntu 18.04 of course.

I think it's a great work around to mitigate the steamcmd limitations. And it doesn't seem to be a dirty work around either. Works pretty well.

If you need something else, let me know. I'm happy to contribute :)

jonakoudijs commented 3 years ago

Thanks for the extra information. Although I still would like to implement a worker system to allow for proper scaling, this could solve some scaling issues for now. Do you got the Dockerfile and Varnish configuration in a repository by any chance? I could reference it in the README for the time being.

Sorry for the late reply (again) but I took an internet break last few days :) I hope you had a nice Christmas!

mvaessen commented 3 years ago

Hi Jona,

I had a great Christmas. Hope you did so too! Unfortunately I don't have a Varnish Docker file. I set it up manually (quick and dirty). Due to the end of year and the holidays I am a little busy, but perhaps I can set something up for you later.

Fijne jaarwisseling!

jonakoudijs commented 3 years ago

Good to hear. Yes, same here!

That's ok, I will keep this issue open so either of us (or someone else of course) can test and add a working Varnish configuration to this issue. Then this can be added to the README as an optional step in case for the people who are running into scaling issues like you did.

Dankjewel, jij ook :)

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.