When running a pipeline with a step operator in SageMaker I may get:
An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'generic-training-pipeline-2023-04-12-15-36-41-124330-model-trainer' at 'trainingJobName' failed to satisfy constraint: Member must have length less than or equal to 63
If the pipeline name is medium sized.
Reproduction steps
Create a pipeline with only one step
Give it a medium size name, such as 'generic-training-pipeline'
Define a step operator in SageMaker
Use the step decorator to run the step on SageMaker
...
Relevant log output
Si è verificata un'eccezione: ClientError (note: full exception trace is shown but execution is paused at: _run_module_as_main)
An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'generic-training-pipeline-2023-04-13-06-05-26-177259-model-trainer' at 'trainingJobName' failed to satisfy constraint: Member must have length less than or equal to 63
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/session.py", line 611, in submit
self.sagemaker_client.create_training_job(**request)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/session.py", line 4344, in _intercept_create_request
return create(request)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/session.py", line 613, in train
self._intercept_create_request(train_request, submit, self.train.__name__)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/estimator.py", line 2042, in start_new
estimator.sagemaker_session.train(**train_args)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/estimator.py", line 1125, in fit
self.latest_training_job = _TrainingJob.start_new(self, inputs, experiment_config)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py", line 272, in wrapper
return run_func(*args, **kwargs)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/integrations/aws/step_operators/sagemaker_step_operator.py", line 207, in launch
estimator.fit(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/step_launcher.py", line 430, in _run_step_with_step_operator
step_operator.launch(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/step_launcher.py", line 375, in _run_step
self._run_step_with_step_operator(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/step_launcher.py", line 198, in launch
self._run_step(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/base_orchestrator.py", line 186, in run_step
launcher.launch()
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/local/local_orchestrator.py", line 82, in prepare_or_run_pipeline
self.run_step(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/orchestrators/base_orchestrator.py", line 166, in run
result = self.prepare_or_run_pipeline(
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/stack/stack.py", line 864, in deploy_pipeline
return self.orchestrator.run(deployment=deployment, stack=self)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/site-packages/zenml/pipelines/base_pipeline.py", line 599, in run
stack.deploy_pipeline(deployment=deployment_model)
File "/home/francesco/repos/shipamax-vulcan-review/data-science/src/modelling_tools/zenml/run_standard_training_pipeline.py", line 46, in <module>
pipeline.run(config_path="standard_training_config.yaml")
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/francesco/miniconda3/envs/zenml/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
return _run_code(code, main_globals, None,
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'generic-training-pipeline-2023-04-13-06-05-26-177259-model-trainer' at 'trainingJobName' failed to satisfy constraint: Member must have length less than or equal to 63
Code of Conduct
[X] I agree to follow this project's Code of Conduct
Thanks for this report. @christianversloot's #1505 was merged into develop very recently which fixes this (and the previous issue #1502). I'll be in the next release. Thanks for reporting it, though!
Contact Details [Optional]
pfrank995@gmail.com
System Information
What happened?
When running a pipeline with a step operator in SageMaker I may get:
An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'generic-training-pipeline-2023-04-12-15-36-41-124330-model-trainer' at 'trainingJobName' failed to satisfy constraint: Member must have length less than or equal to 63
If the pipeline name is medium sized.Reproduction steps
Relevant log output
Code of Conduct