enable jobs to pass in resources for containers

[fmcwalters-edited]

in order to prevent jobs from hogging resources, we need to be able to add limits. These are optional and can be passed in as a dictionary adhering to: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Purpose of PR

Parts of the app this will impact

Todos

Additional information

Related links

[seangarner]

@fmcwalters-edited what's the philosophy here - should we be proactively setting limits in all projects, or only ones where we run into issues?

[fmcwalters-edited]

@seangarner this is a reactive change required for Skuscraper as we had a few crawlers that would end up hogging RAM. I made resources a non-required variable as I didn't want to have to retroactively enforce this for every single project.
Going forward it would be nice to have limits set on containers in projects, this enables us to do that; however it's beyond the scope of this PR to enforce this and I don't feel comfortable unilaterally forcing architectural changes on every single project that uses Kubetools and containers.

[seangarner]

Thanks for the clarification @fmcwalters-edited.

It's made me think about the perspective of smart-proxy, given it's a node project and handles memory differently to python. If you're curious, I've asked the question specific to smart-proxy over in slack.

[ryan109]

@seangarner in an ideal world every single project would have limits on it. Resource limits have a number of applications, but our two biggest benefits are:

• A hard cap, doesn't allow the container to go over the set amount of RAM or CPU usage (although from a chat with @gchazot this morning, apparently the CPU limits are a bit buggy in k8s). So if you set a limit of 2g and the container immediately comes up and try to init a package that requires 3, the container falls over and gets stuck in a crashloop.
• k8s can be more effective in balancing deployments across the cluster. If we have 10 deployments with proper CPU & RAM allocation, then it knows exactly what to expect and how its best to spread these across nodes. If we add an 11th deployment with no specification then k8s is essentially operating blind, as it could be super lightweight and not impact that much, or each pod might need 30g of RAM. It has no way of making those resource claims until the deployment comes up and it notices how much extra it needs, which we have to retroactively expand the cluster. If we had the limits in and the cluster was full, the deployment should just fail.

Point 1) is useful regardless of how many projects we apply limits to, point 2) is only useful if every deployment in the cluster has limits.

[ryan109]

@gchazot the use case isn't within another kubetools function, but when we import the package in other services such as here

[ryan109]

The idea being we can import kubetools inside another python service, this function will generate a kuberetes config using an input kubetools file and then call k8s directly to create the job with desired resources attached.

[ryan109]

@fmcwalters-edited tests are failing as we're passing 'resources': None} in (as no resources are specified in the test config file). We need to confirm k8s will accept 'resources': None, if so then we need to update the test file/test to handle it.

[gchazot]

From the point of view of kubetools, this is a half-baked functionality. Adding this as a possibility in the config generator means it also needs to be available from kubetools.yml. kubetools is first a CLI tool.

[ryan109]

@gchazot see 932b171 and ce94a27.

Right now we can only create job objects from the "upgrade" functionality in kubetools.yml. There's a bit of work to be done if we wanted to add the ability to parse job objects the same way we do cronjobs/deployments. But that's OOS for this PR.