search

usnistgov / dioptra

Test Software for the Characterization of AI Technologies
https://pages.nist.gov/dioptra/
Other
210 stars 30 forks source link

[FEATURE] It would be useful if the task engine's YAML validation errors were printed to the logs (or other places where developers would see them) #604

Open jtsextonMITRE opened 1 month ago

jtsextonMITRE commented 1 month ago

In the task engine, the YAML validation actually presents a list of validation issues, which are useful for debugging the YAML.

However, in this workflow when using them, the validation issues are effectively thrown out. https://github.com/usnistgov/dioptra/blob/dev/src/dioptra/restapi/v1/workflows/lib/run_dioptra_job.py.tmpl#L63

It would be helpful for entrypoint developers to have the issues sent to the logs.

chisholm commented 1 month ago

My thinking was that YAML construction/submission and validation should occur long before it is passed to a worker. The worker is not (normally) the place to help users debug their declarative experiment description. The validation done in the worker is kind of a last-chance error check. But I think normally an invalid description should not have made it that far.

But I guess not all dioptra systems for debugging/diagnosing the descriptions are in place yet. If extra verbosity is added to the worker, I think it should be a temporary measure. If you have an experiment description in hand, you could just run the CLI validator on it. If you don't because the YAML is being automatically generated, and you would like to get your own hands on it to inspect, run, and/or validate it, perhaps a temporary endpoint could be added to download it?

So I guess to me, the simple "invalid YAML" diagnostic in the worker indicates that something upstream didn't catch the invalid description before it wound up being sent to the worker. One needs to figure out what happened there.