Closed BaoshengHeTR closed 2 years ago
Is it really true that log_confusion_matrix
, etc., do not work in training jobs? I've been debugging for hours wondering why these methods weren't returning anything in my training jobs. If this is the problem, then these docs are very misleading, as it makes it appear that a training job generated the output charts. Furthermore, these docs state that
Note that this method must be run from a SageMaker context such as studio or training job due to restrictions on the CreateArtifact API.
which also implies that they could be run from a training job. It would be great if these charts and other artifacts could be generated/logged from within training jobs.
Hi Jordan,
Approches to load artifacts during the training job would be helpful.
You should be able to call Tracker.load()
in train.py
and then you can call tracker methods. In non-training job host contexts you can still use the Tracker
but need to supply the training job name, for example: Tracker.load(training_job_name=...)
but keep in mind log_metric
will not work outside of a training host.
tracker.load_artifact('report.txt')
I assume you mean tracker.upload_artifact
?
to upload the evaluation results on dev dataset; since the evaluation should be done in the training job, so we don't need to redo it after the training job
You should be able to upload artifacts during a training job, including a confusion matrix Also, some pipelines may break the training and evaluation steps into two different steps, for example Orchestrating Jobs with Amazon SageMaker Model Building Pipelines.
Is it really true that log_confusion_matrix, etc., do not work in training jobs?
It does work in training job. Are you using Tracker.load()
in your train.py
?
weren't returning anything in my training jobs.
log_confusion_matrix
will create a lineage Artifact entity with an association to the Trial Component that represents the training job. For example you should be able to see the created association by listing output associations: Associations.list(source_arn=<trial component arn>)
Note that this method must be run from a SageMaker context such as studio or training job due to restrictions on the CreateArtifact API.
This limitation was lifted recently, Ill update that documentation.
It would be great if these charts and other artifacts could be generated/logged from within training jobs.
If you could send me some psuedo code on what you are doing in your train.py
that could be great. Normally Tracker
works during the training job.
If this is the problem, then these docs are very misleading, as it makes it appear that a training job generated the output charts.
Ya, I can see that all it says is The graphs were produced using the Tracker APIs.
which doesn't give the full picture.
load artifacts during training jobs
Tracker
is created to upload/track experiment meta data, i.e., it is contructed before we callestimator.fit(**kwargs)
job, and so does the upload of metadata. However, some results are only available when a model is trained byestimator.fit(**kwargs)
job. Approches to load artifacts during the training job would be helpful. Describe the solution you'd like In endpoint scripttrain.py
used to createestimator
, if we can call:By doing so, artifacts generated during the training job can be tracked.
Describe alternatives you've considered
Additional context I know
tracker
provides methods likelog_confusion_matrix
, .etc. However, I think these methods assume we deploy the model (i.e., after the training job is done) and run inference and then log the results. But consider such a scenario that we want to callload_artifact
to upload the evaluation results on dev dataset; since the evaluation should be done in the training job, so we don't need to redo it after the training job.