search

galaxyproject / galaxy-helm

Minimal setup required to run Galaxy under Kubernetes
MIT License
41 stars 38 forks source link

Hardcoded job priorityClass prohibits scaling #450

Closed mapk-amazon closed 7 months ago

mapk-amazon commented 7 months ago

The job priorityClass value is set here to -1000: https://github.com/galaxyproject/galaxy-helm/blob/4e12bbb7caee93cd05d997ba93ff33e7a596c74e/galaxy/templates/priorityclass-job.yaml#L8

As the value is very low, k8s does not start new nodes to work these jobs.

Use Case: k8s uses two node groups: one "exclusively" for jobs and one for the web server and other non-performance intensive pods. The job node group consists of powerful machines, which should scale to 0 when no job is running. Currently, k8s won't spin those machines up, as the priority of the jobs is too low.

Suggestion Expose this as variable to allow customizations to allow the increase of the priority for such a use case.

Thank you!

nuwang commented 7 months ago

This was originally put in to support single node deployments. For your case, it sounds like disabling the priorityClass altogether using enabled=false, or pointing to a new priorityClass using existingClass with the desired values would work?

mapk-amazon commented 7 months ago

Hi, thank you, and yes, absolutely. I am hesitating to disable it completely as it would, in my understanding, default to "preempting" lower priority pods. I will create then a new class and assign the needed parameters. Just wanted to check whether it will be customizable out-of-the-box upstream. :)

nuwang commented 7 months ago

I see what you mean. No objections to parameterizing this, it's a simple enough change.

mapk-amazon commented 7 months ago

Thank you. I created a PR here: https://github.com/galaxyproject/galaxy-helm/pull/451

ksuderman commented 7 months ago

Thanks for the PR!

Closed by #451