Open fjpa121197 opened 1 year ago
@fjpa121197 thanks for reaching out sagemaker! It seems like we are setting enable_sagemaker_metrics = True
when calling create_training_job api from SageMaker PySDK. Can you provide SageMaker training job ARN for further debug?
I have the same problem. Is it safe to display the ARN here so that you can debug? Any other information that you need?
Describe the bug SageMaker Experiments console, showing not data to create charts for metrics defined in job.
I'm currently running a HPO jobs, created by using the HyperparameterTuner object and Tensorflow estimator. This is the code portion for creating the HPO job, and estimator to be used:
I wait for the training of the jobs to finished, and they appeared in the Experiments console (in SageMaker studio). These are the metrics for one of the two jobs:
However, when trying to create a chart, to see what is the train loss over the epochs, I get a message that there is not data.
When I look at the training job settings, in the SageMaker console, I see that the "SageMaker metrics time series" is disabled, eventhough in my estimator, Tensorflow, I have it as True.
Not sure why the estimator configuration is not kept, when using the HyperParameterTuner object. When calling the .fit() method from the estimator, it keeps the enable_sagemaker_metrics = True.
System information A description of your system. Please provide: