EducationalTestingService / skll

SciKit-Learn Laboratory (SKLL) makes it easy to run machine learning experiments.
http://skll.readthedocs.org
Other
552 stars 67 forks source link

Stop using sklearn's private `_scorer` API for custom metrics in SKLL. #751

Closed desilinguist closed 1 year ago

desilinguist commented 1 year ago

This PR closes #748 and closes #750.

To review this PR, please try to create some custom metrics for both the titanic and California examples and try to use them via both the API and the configuration file.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.05 :tada:

Comparison is base (5fab3a5) 95.24% compared to head (075ab79) 95.30%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #751 +/- ## ========================================== + Coverage 95.24% 95.30% +0.05% ========================================== Files 29 29 Lines 3578 3576 -2 ========================================== Hits 3408 3408 + Misses 170 168 -2 ``` | [Impacted Files](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService) | Coverage Δ | | |---|---|---| | [skll/\_\_init\_\_.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9fX2luaXRfXy5weQ==) | `100.00% <ø> (ø)` | | | [skll/data/writers.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9kYXRhL3dyaXRlcnMucHk=) | `94.11% <ø> (+0.91%)` | :arrow_up: | | [skll/experiments/\_\_init\_\_.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9leHBlcmltZW50cy9fX2luaXRfXy5weQ==) | `94.66% <100.00%> (-0.03%)` | :arrow_down: | | [skll/learner/\_\_init\_\_.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9sZWFybmVyL19faW5pdF9fLnB5) | `97.19% <100.00%> (+<0.01%)` | :arrow_up: | | [skll/learner/utils.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9sZWFybmVyL3V0aWxzLnB5) | `93.40% <100.00%> (+0.03%)` | :arrow_up: | | [skll/metrics.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/751?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9tZXRyaWNzLnB5) | `97.16% <100.00%> (+0.08%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

mulhod commented 1 year ago

Tests passed for me just now. Will review shortly.