search

aws-samples / eks-kubeflow-workshop

Kubeflow workshop on EKS. Mainly focus on AWS integration examples. Please go check kubeflow website http://kubeflow.org for other examples
Apache License 2.0
96 stars 56 forks source link

Refactor the workshop to further simplify user setup #37

Open Jeffwan opened 4 years ago

Jeffwan commented 4 years ago

We have a few challenges for this project, the major issue

  1. some commands have to be run outside the notebook. image

Some of them are aws commands, some of them are kubeflow namespace level operation. image

  1. We ask user to restart kernel to make new installed command take affect. image

  2. Resource are not cleaned up after experiment, we should keep this in mind when we develop the codes.

  3. People may jump to different step without following tutorial in order, we add a few duplicate steps like creating S3 bucket, etc in multiple places. This needs to be refactor and make sure we have a prerequisite to setup everything before they jump into the real tutorial.

TODO: If multiple notebook share one bucket, how to pass bucket name into different notebooks?

cfregly commented 4 years ago

These are big issues, indeed.

Re: S3 bucket - I force users to create the same S3 Bucket that would be created by SageMaker's Default S3 bucket (sagemaker.Session().default_bucket). This is mainly because our workshop combines SageMaker with Kubeflow. And we typically use Event Engine which creates a new, ephemeral account for each user.

Here's the notebook code:

%%bash

export S3_BUCKET=sagemaker-$(aws configure get region)-$(aws sts get-caller-identity | jq -r '.Account') echo "export S3_BUCKET=${S3_BUCKET}" | tee -a ~/.bash_profile

Create a new S3 bucket and upload the dataset.

aws s3 ls s3://$S3_BUCKET || aws s3 mb s3://${S3_BUCKET}

echo "Completed"

The other stuff mentioned here is a sore point, for sure. I don't have a better solution. Curious to see what you come up with.

On Mar 29, 2020, at 4:35 PM, Jiaxin Shan notifications@github.com wrote:

We have a few challenges for this project, the major issue

some commands have to be run outside the notebook. https://user-images.githubusercontent.com/4739316/77863909-a7306f80-71da-11ea-81af-2d09733b2dd2.png Some of them are aws commands, some of them are kubeflow namespace level operation. https://user-images.githubusercontent.com/4739316/77863928-cfb86980-71da-11ea-8b2e-891ddbfae024.png We ask user to restart kernel to make new installed command take affect. https://user-images.githubusercontent.com/4739316/77863792-d1cdf880-71d9-11ea-8415-70087c9c378a.png Resource are not cleaned up after experiment, we should keep this in mind when we develop the codes.

People may jump to different step without following tutorial in order, we add a few duplicate steps like creating S3 bucket, etc in multiple places. This needs to be refactor and make sure we have a prerequisite to setup everything before they jump into the real tutorial.

TODO: If multiple notebook share one bucket, how to pass bucket name into different notebooks?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/eks-kubeflow-workshop/issues/37, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK7C4E4CJLT4ZLNMVYIAG3RJ7LNRANCNFSM4LWFK3WQ.

Jeffwan commented 4 years ago

@cfregly This is great. We should use more like this rather than a random string for bucket name, etc.