Closed jmgray24 closed 3 years ago
Hi @jmgray24, yes this is intentional. log_metric
works by first writing the metrics to a file, which is then picked up by a metrics agent, which inserts the metrics into SageMaker. This agent only runs on training job hosts. Message me via slack if you would like more detail.
It would be useful for processing jobs as well.
Currently, I am having issues writing metrics using the Tracker class log_metric from a Processing Job
It appears that metrics_writer class only creates a metrics writer if it detects it is within a Training Job. Is this intentional?
https://github.com/aws/sagemaker-experiments/blob/1f1a0b780bdee42c8293d032d99279af6e44ded0/src/smexperiments/tracker.py#L108