search

aws / aws-step-functions-data-science-sdk-python

Step Functions Data Science SDK for building machine learning (ML) workflows and pipelines on AWS
Apache License 2.0
289 stars 88 forks source link

Feature request: nested Step Functions #125

Closed kmiller96 closed 3 years ago

kmiller96 commented 3 years ago

Hi! I love your package. Until now I've been programmatically generating Step Function definitions using my own library (which was pretty hacky), so I love being able to migrate over to a better written one instead.

I think a really handy feature would be including nested Step Functions as a compute task. This is a common pattern that I use to help manage complexity and reuse high-level execution flows. This functionality is fully supported in AWS Step Functions already: https://docs.aws.amazon.com/step-functions/latest/dg/connect-stepfunctions.html

If you can point me in the right direction I'm happy to make the contribution to the source myself.

wong-a commented 3 years ago

Glad you're liking it! All state types and service integration steps are defined in src/stepfunctions/steps/.

Here's an example of a Lambda step: https://github.com/aws/aws-step-functions-data-science-sdk-python/blob/main/src/stepfunctions/steps/compute.py#L19-L45

It's fairly trivial to add new service integrations. We might consider some additional functionality to make it easier to use with Workflow instances, but all you'd really need to pass is the state machine ARN and execution input.

tomaszdudek7 commented 3 years ago

Hello @shivlaks. I see that this issue is in progress. Are there any chances of merging it any time soon?

shivlaks commented 3 years ago

hey @tomaszdudek7 - intending to have a PR out for this this week and targeting the next released version to include this support!

tomaszdudek7 commented 3 years ago

Great to hear.

shivlaks commented 3 years ago

@tomaszdudek7 - our wonderful team member @ca-nguyen will be implementing, but targets remain the same - shooting for end-of-week PR and shipping with the next release

AustinGomez commented 3 years ago

Hi, any updates here? I saw that the PR has been opened but haven't seen anything since. Thank you!

ca-nguyen commented 3 years ago

Hi @AustinGomez, We are aiming to have traction on the PR and merge the fix by end of week next week Thanks for your interest in this feature!