StatCan / aaw

Documentation for the Advanced Analytics Workspace Platform
https://statcan.github.io/aaw/
Other
68 stars 12 forks source link

Create reusable kfp components MinIO #344

Closed ca-scribner closed 3 years ago

ca-scribner commented 3 years ago

To help users access files from MinIO when using kubeflow pipelines, create reusable components (eg: component yaml files that can be accessed via load_component_from_url()). These components should use the automatically mounted secrets for MinIO and avoid errors due to the race condition where the main pod may be running before vault has mounted the secrets.

ca-scribner commented 3 years ago

Components get_from_minio and put_to_minio are complete. They:

A simple example of use is also included, and serves as a rudimentary test case.

The files are temporarily located off StatCan repos because I haven't found a natural home for them yet.

Future work:

ca-scribner commented 3 years ago

cc @blairdrummond @brendangadd