search

LibreTexts / metalc

LibreTexts/UCDavis bare-metal Kubernetes cluster running JupyterHub and Binder
http://jupyter.libretexts.org
MIT License
14 stars 2 forks source link

Create manual and/or automated operating procedure for dealing with binder users that abuse the free resources #229

Open moorepants opened 3 years ago

moorepants commented 3 years ago

For example, we've had binder users launch crypto mining software a few times over the years we've run the service. The current process is to simply kill their pods when noticed. We should define a standard operating procedure for identifying this and manually dealing with it. We may also be able to deal with it automated ways too. The mybinder.org people deal with it much more and must have some solutions. At the minimum, let's setup timeouts on all binderpods. There should be no reason to run binder pods longer than a few hours. The intended use for our binder is to execute and work with a libretexts page. It may take person some time to work through a page, but we can set a max time limit on that. That would at least automatically cutoff miners.

moorepants commented 3 years ago

@sandertyu found this: https://github.com/jupyterhub/mybinder.org-deploy/pull/1778 Looks like there is a binder minesweeper setup.

sandertyu commented 3 years ago

It looks like mybinder is simply running a kubernetes daemonset and each spawned pod will run a script minesweeper.py which can identify and kill suspicious pods through the kubernetes API. There's some secrets that they have which we may have to ask about in order to implement their version of crypto-mining security, but that seems to be their general measure.

Furthermore, you want to add a timeout to all binderhub user spawned pods after an hour or two, regardless of activity?

sandertyu commented 3 years ago

We've got a serviceable script to kill crypto processes when they occur, and this has served to ward off cryptominers so far. I'll keep the issue open because it's a good idea, but will lower its priority.

sandertyu commented 3 years ago

We have an automated systemd script to kill processes according to known miner names, as well as a kubernetes program which detects when pods are running at high cpu usage for a given amount of time and automatically deletes them. We have documentation for everything except for the latter method, which Kevin made.