EducationalTestingService / skll

SciKit-Learn Laboratory (SKLL) makes it easy to run machine learning experiments.
http://skll.readthedocs.org
Other
550 stars 69 forks source link

Include model fit times in learning curves #745

Closed desilinguist closed 1 year ago

desilinguist commented 1 year ago

Last year, scikit-learn added functionality to include model fit times when computing learning curves since – in addition to the model's performance – it's also quite useful to know how the long the model takes to train as more training data was added. This PR now adds the same functionality to SKLL.

As always, the best way to review is to try this out in the examples. As a starting point, if you want to replicate the same example, you can modify the Titanic example's learning_curve.cfg file as shown below and then look at the Titanic_Learning_Curve_all.png and Titanic_Learning_Curve_all_times.png files in the output directory.

CleanShot 2023-06-26 at 11 16 33@2x

This PR closes #556.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.05 :tada:

Comparison is base (143ff09) 95.19% compared to head (10469c3) 95.24%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #745 +/- ## ========================================== + Coverage 95.19% 95.24% +0.05% ========================================== Files 29 29 Lines 3538 3578 +40 ========================================== + Hits 3368 3408 +40 Misses 170 170 ``` | [Impacted Files](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService) | Coverage Δ | | |---|---|---| | [skll/experiments/\_\_init\_\_.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9leHBlcmltZW50cy9fX2luaXRfXy5weQ==) | `94.69% <100.00%> (ø)` | | | [skll/experiments/output.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9leHBlcmltZW50cy9vdXRwdXQucHk=) | `97.86% <100.00%> (+0.40%)` | :arrow_up: | | [skll/learner/\_\_init\_\_.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9sZWFybmVyL19faW5pdF9fLnB5) | `97.18% <100.00%> (ø)` | | | [skll/learner/utils.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9sZWFybmVyL3V0aWxzLnB5) | `93.37% <100.00%> (+0.05%)` | :arrow_up: | | [skll/learner/voting.py](https://app.codecov.io/gh/EducationalTestingService/skll/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=EducationalTestingService#diff-c2tsbC9sZWFybmVyL3ZvdGluZy5weQ==) | `98.54% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.