Open UrkoAT opened 6 months ago
Thank you for creating this issue @UrkoAT. This is interesting, by default we use the following resources for Suggestion deployment: https://github.com/kubeflow/katib/blob/master/pkg/apis/config/v1beta1/defaults.go#L28-L34.
I was trying to run your example with those HPs and yes my Suggestion went OOM after 12 Trials: The latest log that I got:
INFO:pkg.suggestion.v1beta1.skopt.base_service:----------------------------------------------------------------------------------------------------
INFO:pkg.suggestion.v1beta1.skopt.base_service:New GetSuggestions call with current request number: 3
INFO:pkg.suggestion.v1beta1.skopt.base_service:Succeeded Trials changed: 12
INFO:pkg.suggestion.v1beta1.skopt.base_service:Running Optimizer tell to record observation
INFO:pkg.suggestion.v1beta1.skopt.base_service:Evaluated parameters: [['256', '2.8', '3.2', '50', 'test', '0.001', '0.1', 'dense_12;dense_11'], ['256', '2.8', '3.2', '30', 'test2', '0.0001', '0.01', 'dense_12']]
INFO:pkg.suggestion.v1beta1.skopt.base_service:Objective values: [-2.347, -3.06]
INFO:pkg.suggestion.v1beta1.skopt.base_service:Optimizer tell method takes 1 seconds
INFO:pkg.suggestion.v1beta1.skopt.base_service:List of recorded Trials names: ['test-memory-6kjct2bf', 'test-memory-nqvprjz6', 'test-memory-jr9cczjb', 'test-memory-cbpx68tw', 'test-memory-7hxnzbng', 'test-memory-fh2kzbf8', 'test-memory-t2mcsc7j', 'test-memory-lcs7phk4', 'test-memory-gc7mcfk7', 'test-memory-xw4klqgh', 'test-memory-697h6ccg', 'test-memory-dx9qt2p7']
We use Skopt for Bayesian Optimization algorithm: https://github.com/scikit-optimize/scikit-optimize. It seems that when number of HyperParameters grow Skopt requires more resources to store them in memory.
What do you think about it @johnugeorge @tenzen-y ?
Maybe by default we should not specify resources for Suggestion and Metrics Collector deployment (e.g. instead of asking user to set -1
in Katib Config, we just remove resources assignment) ?
If users want to limit resources that can be consumed by Suggestion, they can always do it via Katib Config: https://www.kubeflow.org/docs/components/katib/katib-config/
Thanks for answering this fast lol. Reading the documentation you just provided, I realize it isn't necessary to manually change the resources in each experiment.
While you guys discuss what should be done, if any other person has this problem, for now, you can kubectl edit cm -n kubeflow katib-config
and in suggestions:
you can specify the resources for any algorithm. For example, in my case:
- algorithmName: bayesianoptimization
image: docker.io/kubeflowkatib/suggestion-skopt:v0.16.0-rc.1
resources:
requests:
memory: 100Mi
limits:
memory: 1Gi
I hope it helps.
What do you think about it @johnugeorge @tenzen-y ? Maybe by default we should not specify resources for Suggestion and Metrics Collector deployment (e.g. instead of asking user to set -1 in Katib Config, we just remove resources assignment) ?
@andreyvelich, I think that we should keep setting default resources to the suggestion services regardless of whether we increase default resources. Because the Pods without resource requests/limits mean that the QoS class is BestEffort
, and as a result, we allow the k8s cluster to evict the suggestion pods despite being mandatory for the Experiments.
The eviction of the suggestion Pods often occurs, and it will bring users confusion since, in general, the ML cluster is busier than other types of clusters.
When you say Kubernetes can evict Suggestion pod can you explain it please @tenzen-y ? E.g. we don't set Pod Priority for our Suggestion pods or other parameters that identify that this pod can evicted: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction
Also, if you take a look on our Katib Controller Deployment, we also don't specify resources: https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/components/controller/controller.yaml
Should it be users or admins responsibility to specify appropriate Pod resources for Katib components ?
I guess, you mention about this BestEffort
QoS class: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#besteffort.
The question is for us what is better ?
What are your thoughts @johnugeorge ?
I guess, you mention about this
BestEffort
QoS class: https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#besteffort. The question is for us what is better ?
- Users to see OOM on their Suggestion pods
- Users to see pod being evicted by Kubernetes if Node doesn't have enough resources to run Suggestion pod.
What are your thoughts @johnugeorge ?
Yes, I meant what you said.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
/good-first-issue /help
@andreyvelich: This request has been marked as suitable for new contributors.
Please ensure the request meets the requirements listed here.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue
command.
/assign
/kind bug
What steps did you take and what happened:
Every time I try to run an experiment (in this case using Bayesian Optimization) after 18-25 trials, the pod that schedules the trials with suggested hyperparameters gets OOM Killed and the experiment does not continue.
If you manually increase the memory resources and limits of the deployment, it works like a charm.
What did you expect to happen:
It should either take less memory and work with this kind of numbers of trials without any problem, or more easily just increase the default limit of the suggestion pod memory to a more realistic number.
Anything else you would like to add: The experiment is the following:
Environment:
kubectl version
): Server Version: v1.25.9uname -a
): Linux 6.5.0-15-genericImpacted by this bug? Give it a 👍 We prioritize the issues with the most 👍