[SDK] Support Docker image as objective in the `tune` API

[andreyvelich]

Ref discussion: https://github.com/kubeflow/website/pull/3723#discussion_r1590261777.

Currently, user can only pass the training function as objective in the tune API in Katib Python SDK.

Similar to create_job API in Training Python SDK, we should give user an ability to set objective as Docker image.

/area sdk /good-first-issue

[google-oss-prow[bot]]

@andreyvelich: This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to [this](https://github.com/kubeflow/katib/issues/2326): >Ref discussion: https://github.com/kubeflow/website/pull/3723#discussion_r1590261777. > >Currently, user can only pass the training function as [objective in the `tune` API](https://github.com/kubeflow/katib/blob/af900202c67843650d80ff955e4e7fb7510cc68a/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L156) in Katib Python SDK. > >Similar to [`create_job` API in Training Python SDK](https://github.com/kubeflow/training-operator/blob/f23c5c8b0c87d93f61b539015883152a0872124f/sdk/python/kubeflow/training/api/training_client.py#L327), we should give user an ability to set objective as Docker image. > >/area sdk >/good-first-issue > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[akhilsaivenkata]

/assign

[akhilsaivenkata]

Hi @andreyvelich , I am new to kubeflow community and I have been going through the Ref discussion as well as code snippets of tune API and create job API. This is my understanding:

1. create_job API creates the job using one of the following options:

   • Define custom resource object in job parameter (e.g. TFJob or PyTorchJob).
   • Define training function in train_func parameter and number of workers.
   • Define Docker image in base_image parameter and number of workers.

2. So if train_func and custom resource are not provided then method takes base_image and tries to create the job and its template without train_func.

3. Now we are looking to have similar functionality in tune API by giving user ability to use Docker image instead of callable function to tune the hyperparameters.

4. After looking at the tune API method parameters, I can see base_image as one of the parameters which is already taking Docker image as input(constants.BASE_IMAGE_TENSORFLOW image as default). So I wonder If we could make the objective parameter as an optional parameter where it takes 'None' value if no callable function is passed and make the tune API execute the steps with provided Docker image as base_image .

I would like to know whether I am on the right page or not. Please correct me If I am wrong.

[andreyvelich]

HI @akhilsaivenkata, yes, you are absolutely right for the TrainingClient. Also, we are planning to add target_image to the create_job in the future to build training image before job creation: https://github.com/kubeflow/training-operator/issues/1878

So I wonder If we could make the objective parameter as an optional parameter where it takes 'None' value if no callable function is passed and make the tune API execute the steps with provided Docker image as base_image .

What do you think about re-using objective parameter to pass the Docker image ? In that case, we will just omit base_image and use image for Trial from objective parameter. In the future, we can give users ability to set more parameters in objective (e.g. Git repo, tarball file). What do you think @kubeflow/wg-training-leads @akhilsaivenkata @droctothorpe ?

[akhilsaivenkata]

HI @akhilsaivenkata, yes, you are absolutely right for the TrainingClient. Also, we are planning to add target_image to the create_job in the future to build training image before job creation: kubeflow/training-operator#1878

So I wonder If we could make the objective parameter as an optional parameter where it takes 'None' value if no callable function is passed and make the tune API execute the steps with provided Docker image as base_image .

What do you think about re-using objective parameter to pass the Docker image ? In that case, we will just omit base_image and use image for Trial from objective parameter. In the future, we can give users ability to set more parameters in objective (e.g. Git repo, tarball file). What do you think @kubeflow/wg-training-leads @akhilsaivenkata @droctothorpe ?

Thank you so much for your review @andreyvelich . If we have plans to give users the ability to set more parameters then I believe it would definitely be better option to go with your approach. If everyone is positive with this plan then I can proceed with implementation.