feat: Add support for Amazon EKS

aws / aws-step-functions-data-science-sdk-python

Step Functions Data Science SDK for building machine learning (ML) workflows and pipelines on AWS

Apache License 2.0

285 stars 87 forks source link

feat: Add support for Amazon EKS #156

Closed ca-nguyen closed 3 years ago

ca-nguyen commented 3 years ago

Issue #, if available: #106

Description of changes: Added:

EksRunJob to create a Task state that allows you to run a job on your Amazon EKS cluster
EksCall to create a Task state that allows you to use the Kubernetes API to read and write Kubernetes resource objects via a Kubernetes API endpoint
EksCreateClusterStep to create a Task state that creates an Amazon EKS cluster
EksDeleteClusterStep to create a Task state that deletes an Amazon EKS cluster
EksCreateFargateProfileStep to create a Task state that creates a Fargate profile
EksDeleteFargateProfileStep to create a Task state that deletes a Fargate profile
EksCreateNodeGroupStep to create a Task state that creates a node group
- EksDeleteNodeGroupStep to create a Task state that deletes a node group

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ca-nguyen commented 3 years ago

go for it - please be sure to update the PR description to include runJob and call APIs that were added in the most recent commit.

Done! Thank you!

ca-nguyen commented 3 years ago

LGTM besides some minor updates to the docs.

Thanks for reviewing! I applied your suggested changes

I see there are unit tests, but did you do any manual testing to ensure each step can be created with Step Functions through the SDK?

I did not do any testing outside of unit tests. I'll make sure to manual test before merging

StepFunctions-Bot commented 3 years ago

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-sEHrOdk7acJc
Commit ID: 1cef855fb1a8fa7ea93123c012b6c039fe66e21c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ca-nguyen commented 3 years ago

I see there are unit tests, but did you do any manual testing to ensure each step can be created with Step Functions through the SDK?

Tested manually and uncovered a few bugs:

eks::call does not have option to run a job (.sync)
removed LogOptions param from eks::runJob example as it does not allow to retrieve of logs (only possible with eks::runJob.sync) Made the changes in the last commits

shivlaks commented 3 years ago

I see there are unit tests, but did you do any manual testing to ensure each step can be created with Step Functions through the SDK?

Tested manually and uncovered a few bugs:

eks::call does not have option to run a job (.sync)

removed LogOptions param from eks::runJob example as it does not allow to retrieve of logs (only possible with eks::runJob.sync) Made the changes in the last commits

glad we caught those. be sure to expand on testing methodology and the steps we followed.

not for this PR:

thought: how can we automate that testing (via unit/simple integ tests). It will be helpful to establish so that we responsibly introduce service integrations. More importantly, regressions are easy to introduce if we don't have automated tests in place.