search

kubeflow-kale / kale

Kubeflow’s superfood for Data Scientists
http://kubeflow-kale.github.io
Apache License 2.0
628 stars 129 forks source link

Creating Katib experiment, not errors but the trial pods are not created #443

Closed vitobotta closed 1 year ago

vitobotta commented 1 year ago

Hi! I am new to Kubeflow, Katib and Kale so perhaps I'm missing something obvious. I have a notebook that works fine with Kale as far as creating and running pipelines is concerned. However when I create an experiment by enabling and configuring Katib, it creates the firs pod for the search (grid, random, etc) but then nothing else happens. The katib controller log shows that the trials are "running" and that's also what the UI says, but nothing else really happens. There are no new pods created and I don't get any errors in either the logs or the UI. What could be causing this? Thanks in advance!

vitobotta commented 1 year ago

It was an issue with the service account.