Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
7.88k stars 750 forks source link

Triggering Production Step Functions (AWS) #1602

Open jdonahue94 opened 9 months ago

jdonahue94 commented 9 months ago

Team - I have created a AWS step function with the following command.

insert_name_here.py --production step-functions create

Is there a way to trigger this production step function manually without using the AWS UI?

Thanks, Jacob

savingoyal commented 9 months ago

yep - insert_name_here.py --production step-functions trigger

jdonahue94 commented 9 months ago

Great. I thought so as well.

Though, the problem I run into is that the above seems to run whatever I have currently saved locally vs what was saved when I ran create.

For example, say I had the following when I ran create:

param = True

I then saved the following:

param = False

I then ran:

insert_name_here.py --production step-functions trigger

And the production step function ran but with param = False rather than True.

I was expecting param = True to be used since that was what the production step was created with.

savingoyal commented 9 months ago

trigger also passes in the parameters to the step-function. you can always do insert_name_here.py --production step-functions trigger --param True

jdonahue94 commented 9 months ago

Agreed.

I'm saying this param was hard coded ininsert_name_here.py and not passed as a param in the cli command.

The production run ran what was most recently saved locally vs what was saved when create was called.

That is:

param = True was hardcoded and saved locally when create was called.

vs.

param = False was hardcoded and saved locally when trigger was called.

My thinking was that insert_name_here.py --production step-functions trigger would run the code that was snapshotted when create was called regardless of what was saved locally.

jdonahue94 commented 9 months ago

Friendly bump of the above. Thanks.