Description of changes:
Currently, the evaluate_dataset utility function (which gets called by the majority of the evaluation algorithms' evaluate method) assumes that if the ModelRunner passed to it is non-null, this means that model outputs should be obtained. However, this assumption does not hold for some users who provide a dataset with model outputs, but call evaluate with a non-null ModelRunner anyway.
This PR changes the conditional logic in evaluate_dataset such that the BYO inference outputs use case above is supported.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Description of changes: Currently, the
evaluate_datasetutility function (which gets called by the majority of the evaluation algorithms'evaluatemethod) assumes that if theModelRunnerpassed to it is non-null, this means that model outputs should be obtained. However, this assumption does not hold for some users who provide a dataset with model outputs, but callevaluatewith a non-nullModelRunneranyway.This PR changes the conditional logic in
evaluate_datasetsuch that the BYO inference outputs use case above is supported.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.