Open MrDataPsycho opened 3 years ago
Hi @DataPsycho!
Currently, the only way to use Placeholders with container_arguments
is to define the container arguments entirely as a Placeholder.
Something like this:
execution_input = ExecutionInput(
schema={
"PATH_INPUT": str,
"DESTINATION_OUTPUT": str,
"container_arguments": list,
"ProcessingJobName": str,
"input_code": str,
"job_pk": str,
"job_sk": str,
}
)
processing_step = steps.ProcessingStep(
"SageMakerProcessingJob1",
processor=get_processing_container_config(),
job_name=execution_input["ProcessingJobName"],
inputs=input_meta,
outputs=output_meta,
container_arguments=execution_input["container_arguments"],
container_entrypoint=["python3", "/opt/ml/processing/code/main.py"]
)
Being able to use Placeholder values for the individual arguments within the container_arguments
would be a great enhancement to add and I can imagine many use cases for this. Thank you for bringing this to our attention! Tagging this as an enhancement and putting it on our radar.
Hope this helps!
Hi! Im very glad that @DataPsycho raised this issue. I was wondering the same, since I found weird that individual arguments couldn't be specified nor directly referencing the execution input or with the step functions json referencing.
I've tried both (the first similar to the original issue and the json referncing) :
1)
container_arguments=["--metrics-type", execution_input['MetricsType'], "--metrics-name", execution_input['MetricsName'], "--label-name",execution_input['LabelName']]
2)
container_arguments=["--metrics-type", "$$.Execution.Input['MetricsType']", "--metrics-name", "$$.Execution.Input['MetricsName']", "--label-name","$$.Execution.Input['LabelName']"]
It would be really nice to count with this enhancement! It expands and makes step creation more flexible.
Thank you for showing interest in this feature @Liks96! We are keeping this on our radar
Hi, Thanks for considering feature. Looking forward to use it when it is available. Thanks
I got here after having the same issue. I confirm it would be really useful having this flexibility!
I have the same issue, it would be wounderfull to have execution inputs as container arguements
Hi, Lets say I have a execution schema as follows:
How can I use the execution_input values in the Container Argument part bellow:
There the study name should come from the the execution input schema. But when trying to create the workflow graph it throughs following errors. Though in the jobname part it except the value from ExecutionInput
Error: