openml / openml-python

Python module to interface with OpenML
https://openml.github.io/openml-python/main/
Other
279 stars 143 forks source link

Add summary of locally computed metrics to representation of run #1214

Closed LennartPurucker closed 1 year ago

LennartPurucker commented 1 year ago

This addresses #1144 and might be a solution for the issue.

It is important to note that the representation already included a "Result" field and thus could have reported results in the printout. However, this only happens if the evaluation of the run is filled.

This implementation now also covers locally computed results. The locally computed results are only shown if no task/run-specific results already exist.

Example

from sklearn import ensemble
from openml import tasks, runs

clf = ensemble.RandomForestClassifier()
task = tasks.get_task(3954)
run = runs.run_model_on_task(clf, task, avoid_duplicate_runs=False)
print(run)

Old Output:

OpenML Run
==========
Uploader Name: None
Metric.......: None
Run ID.......: None
Task ID......: 3954
Task Type....: None
Task URL.....: https://www.openml.org/t/3954
Flow ID......: None
Flow Name....: sklearn.ensemble._forest.RandomForestClassifier
Flow URL.....: https://www.openml.org/f/None
Setup ID.....: None
Setup String.: Python_3.7.13. Sklearn_1.0.2. NumPy_1.21.6. SciPy_1.4.1.
Dataset ID...: 1120
Dataset URL..: https://www.openml.org/d/1120

New Output (ignore the difference in Setup String):

OpenML Run
==========
Uploader Name...................: None
Metric..........................: None
Local Result - Accuracy (+- STD): 0.8790 +- 0.0054
Local Runtime - ms (+- STD).....: 9662.5000 +- 2621.0648
Run ID..........................: None
Task ID.........................: 3954
Task Type.......................: None
Task URL........................: https://www.openml.org/t/3954
Flow ID.........................: None
Flow Name.......................: sklearn.ensemble._forest.RandomForestClassifier
Flow URL........................: https://www.openml.org/f/None
Setup ID........................: None
Setup String....................: Python_3.9.10. Sklearn_1.2.1. NumPy_1.23.5. SciPy_1.10.0.
Dataset ID......................: 1120
Dataset URL.....................: https://www.openml.org/d/1120

Test

I am not sure if we need to write a test for this. I did not find any tests for the representation so far.

mfeurer commented 1 year ago

Before merging this, could you please have a look at the pre-commit? That one is currently failing.

LennartPurucker commented 1 year ago

I fixed the doc string for the run that was malformed as a result of another PR. The pre-commit should work now.