kubeflow / examples

A repository to host extended examples and tutorials
Apache License 2.0
1.39k stars 751 forks source link

bad generation of components path when buliding with target image #1007

Closed Sergiodiaz53 closed 1 year ago

Sergiodiaz53 commented 1 year ago

Hello there!

I'm having an issue regarding the building of python-based components in containers. First of all let me say that Im trying to pack a set of components in the same image, which I think is not a expected behaviour (correct me if im wrong). The idea is to have 1 docker image with a set of components shared between different pipelines. I dont want to maintain 8/9 images, so the idea is that this generic_components are not going to be actively changing all the time so we can have only 1 image with all of these components

So we have this src/generic_components folder with this kind of structure:

src/generic_components/
     |-- component_metadata/
       |--- download_from_azure.yaml
       |--- download_from_ftp.yaml
       |--- visualize_dataset.yaml
     |-- download/
       |-- download_from_azure/
          |--- download_from_azure_comp.py
          |--- test/                                            
       |--- download_from_ftp/
          |--- download_from_ftp_comp.py
          |--- test/
     |-- visualization/
       |--- visualize_dataset/
          |--- visualize_dataset_comp.py
          |--- test/ 
     |-- utils/
          |--- modules that are used in the _comp files   

All of them use the same target_image and the same base_image. So I'm using:

kfp components build src/generic_components/ --component-filepattern */*/*_comp.py --push-image

And everything runs fine. Files are copied, yamls are generated and the image is created and pushed but, when use in a pipeline, it fails in the execution because in the generated kfp_config.ini I have this:

[Components]
download_from_azure = download\download_from_azure_comp.py
....

but the real path (in the docker image) is:

[Components]
download_from_azure = download/download_from_azure_comp.py
....

This is probably because Im building the image in a windows 10 enviroment but my docker-image is based on linux so:

  1. Is this an error or am I doing something wrong?
  2. Is there anyway to modify the kfp_config.ini before the image is push to fix the problem?
  3. It would be nice to hear also some opinions about this folder structure and several-components-container idea and possible alternatives

Thank you so much!