microsoft / MLOpsPython

MLOps using Azure ML Services and Azure DevOps
MIT License
1.2k stars 1.1k forks source link

run.parent.log fails randomly and silently and breaks Evaluate step in pipeline #393

Closed github-cloudlabsuser-1011 closed 2 years ago

github-cloudlabsuser-1011 commented 2 years ago

With the latest releases of AML SDK, we find line 157 in diabetes_regression/training/train_aml.py failing randomly and silently. This breaks line 122 in diabetes_regression/evaluate/evaluate_model.py. While the fix in evaluate_model.py solves the problem of the missing metric in the parent run, it does not address the root cause of the problem, which is the random failure of run.parent.log.