surajkota
commented
1 year ago Hi @sagi-shimoni, thanks for creating the feature request. We are aware of this and also have this in our backlog #161. We do not have an ETA currently but will take this into consideration. In the meantime, I would recommend you to look at this blog post for a blue green upgrade process implemented by one of our users - https://aws.amazon.com/blogs/machine-learning/build-repeatable-secure-and-extensible-end-to-end-machine-learning-workflows-using-kubeflow-on-aws/
Note that 1.5 and 1.7 have different EKS version support and hence recommend deploying a new EKS cluster a latest version of Kubeflow and utilize the AWS service capabilities while doing a blue green deployment(some of these demonstrated in the blog above), e.g.
- You can use the same Cognito userpool since nothing changes on the auth side
- Backup/Snapshot storage layer using AWS backup or Snapshotting capabilities e.g. RDS, S3, EBS volumes, you can use access points for NFS like EFS and FSx
- For backing up and restoring Kubernetes resources(e.g. notebooks, PodDefaults, Inference services), you can look into using Velero. It also helps with PVs
- If you are using SageMaker resources, you can take adopt resources into new cluster by using the adopted resource feature
What is the URL of the document? https://awslabs.github.io/kubeflow-manifests/docs/deployment/cognito-rds-s3/guide/
Which section(s) is the issue in? N/A
What needs fixing and describe the solution you'd like? Recommended upgrade guide , such as this: https://karpenter.sh/v0.27.5/upgrade-guide/
Additional context We are planning to upgrade from v1.5.1-aws-b1.0.2 to v1.7.0-aws-b1.0.1 (latest) Currently deployed using Cognito-RDS-S3 on EKS version 1.21
It would be really helpful if there was upgrade guide that summarizes breaking changes or recommended actions to take prior to upgrading versions. (Is blue/green required to confirm compatibility?)
We are not concerned about short downtime, however I'm worried that we will upgrade Eks to 1.23/1.24 and there will be compatibility issues but we will not be able to revert.
Appreciate any input regarding this...
Thanks, Sagi