Tracking Issue AWS integration

argoflow / argoflow-aws

Argoflow-AWS has been superseded by deployKF

GNU Affero General Public License v3.0

44 stars 31 forks source link

Tracking Issue AWS integration #21

Open davidspek opened 3 years ago

davidspek commented 3 years ago

Creating a quick list of things that still need to be done so we can track progress.

What still needs to be done:

[ ] Setup Cognito with OAuth2-Proxy
[ ] Documentation for Cognito integration and IAM setup
[x] Use RDS for pipelines and katib database
[x] Use S3 for pipelines instead of MinIO
[ ] AWS plugin for profile controller?

davidspek commented 3 years ago

@karlschriek @matty1979 I'm expecting you guys will run into a lot more things while working on this deployment so it would be good to document that here.

Ping @samdobson In case you are interesting in helping out.

Also ping @PatrickXYS in response to https://github.com/kubeflow/manifests/pull/1832#issuecomment-822859477 if anybody from AWS would like to help out with this effort.

PatrickXYS commented 3 years ago

Use RDS for pipelines database

This is part of KFP support, if we want to enable this, this should be added in kubeflow/pipelines instead.

Use S3 for pipelines instead of MinIO

Same for KFP

For other parts of Cognito stuff, we will explore those in AWS distribution, so I don't think we will be able to contribute to argoflow.

davidspek commented 3 years ago

@PatrickXYS The AWS integrations for things like Cognito in ArgoFlow would all be kustomize manifests, so they would likely be the same as for the AWS distribution.

PatrickXYS commented 3 years ago

@DavidSpek I agree, so you can simply check out AWS side best practice after we're done with those features.

davidspek commented 3 years ago

Somewhat related PR/issue about enabling AWS ALB authentication in KFP SDK Client: https://github.com/kubeflow/pipelines/pull/4182

soleares commented 3 years ago

@DavidSpek Are all of the AWS integrations relatively complete besides Cognito? We currently have the standard KF 1.2 distribution deployed on AWS and I’m wondering if this distribution is at a good place for us to try it.

AWS plugin for profile controller

Is this required if we’re not using Cognito?

davidspek commented 3 years ago

@soleares Most of the AWS integrations are complete. Cognito we just haven’t gotten around to yet, but from what I’ve read it shouldn’t be difficult to setup with Oauth2-Proxy.

The plug-in for the profile controller is just to delegate IAM roles to users and their workloads, it shouldn’t be necessary for anything else.

soleares commented 3 years ago

@DavidSpek What Kubeflow components use this plug-in? Is it required for basic pipeline functionality?

davidspek commented 3 years ago

@soleares The profile controller uses the plugin for service accounts. It is not necessary for functionality of the components.