[feature] Sagemaker training job metrics

awslabs / benchmark-ai

Anubis (formerly known as Benchmark AI), measures the goodness of machine learning workloads

Apache License 2.0

16 stars 6 forks source link

[feature] Sagemaker training job metrics #1047

Closed ryansteakley closed 4 years ago

ryansteakley commented 4 years ago

Issue #, if available: N/A Description of changes:

Allows user to supply a Name and a Regex which will be passed to a dictionary and supplied to the estimator function.
Sagemaker will use this to parse the training jobs logs and create metrics if the regex matches any output. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

arjkesh commented 4 years ago

Can you mark the code that's different from #1044 ? Is the code form #1044 necessary for this implementation?

ryansteakley commented 4 years ago

Nice! Could you please add some information to the descriptor docs to explain how this works? We use this file as the reference of what fields a descriptor can have.

(Sorry to insist on that but otherwise we end up creating tribal knowledge and the descriptor becomes an obscure file)

Will do