Describe the bug
Create Experiment and Trial smexperiment. And configure PipelineExperimentConfig.
Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script, Tracker.load() return exception about this
Traceback (most recent call last):
File "train.py", line 133, in <module>
main()
File "train.py", line 66, in main
tracker = Tracker.load()
File "/miniconda3/lib/python3.8/site-packages/smexperiments/tracker.py", line 161, in load
_ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session),
AttributeError: 'NoneType' object has no attribute 'trial_component_name'
Maybe this function return None.
But TrialComponentEnvironment.source_arn is defined.
So, I guess is this line wrong?
Because environment["TRAINING_JOB_ARN"] contains uppercase when using sagemaker pipeline.
Sorry for my poor English.
To Reproduce
configure Experiment, Trial and PipelineExperimentConfig
run training job using pipeline
Tracker.load() in training script
Expected behaviorTracker.load() load trial_component in pipeline
Environment:
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:
I'm facing the same issue with Sagemaker Pipelines. When I remove the ".lower" in the code you mentioned here then it works. Is it necessary to convert the source arn to lowercase?
Describe the bug Create Experiment and Trial
smexperiment
. And configurePipelineExperimentConfig
. Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur) In training script,Tracker.load()
return exception about thisMaybe this function return
None
. ButTrialComponentEnvironment.source_arn
is defined. So, I guess is this line wrong? Becauseenvironment["TRAINING_JOB_ARN"]
contains uppercase when using sagemaker pipeline.Sorry for my poor English.
To Reproduce
Tracker.load()
in training scriptExpected behavior
Tracker.load()
load trial_component in pipelineEnvironment: Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): Framework Version: Python Version: 3.9.11 CPU or GPU:CPU Python SDK Version:
Are you using a custom image: yes