Additional Error Methods (Closes #510)

[teubert]

This idea comes from something Juan was doing with his work.

This PR adds the ability to select different metric types when calculating error. The default configuration was MSE, but this PR adds RMSE, Max error (what Juan was using), MAE, and MAPE. The change was propagated into estimate_parameters so we can change what method we're using for error population when estimating parameters.

Partially closes #510 .

It conflicts with Aditya's changes to the tests, so the tests will be added in a separate PR

[github-actions[bot]]

Thank you for opening this PR. Each PR into dev requires a code review. For the code review, look at the following:

• [ ] Reviewer should look for bugs, efficiency, readability, testing, and coverage in examples (if relevant).
• [ ] Ensure that each PR adding a new feature should include a test verifying that feature.
• [ ] All tests must be passing.
• [ ] All errors from static analysis must be resolved.
• [ ] Review the test coverage reports (if there is a change) - will be added as comment on PR if there is a change
• [ ] Review the software benchmarking results (if there is a change) - will be added as comment on PR
• [ ] Any added dependencies are included in requirements.txt, setup.py, and dev_guide.rst (this document)
• [ ] All warnings from static analysis must be reviewed and resolved - if deemed appropriate.

[github-actions[bot]]

Benchmarking Results From:	Test	Time (s)
import main	0.12450959999999989
import thrown object	0.5034851
model initialization	0.10814729999999995
set noise	0.6647426999999999
simulate	0.29425420000000013
simulate with saving	0.9067192000000004
simulate with saving, dt	1.041127
simulate with printing results, dt	1.2501384
Plot results	13.8756545
Metrics	0.039065799999999484
Surrogate Model Generation	3.1775205
surrogate sim	0.9665627999999984
surrogate sim, dt	2.5974693

To:	Test	Time (s)
import main	0.12900820000000013
import thrown object	0.5149081
model initialization	0.10627190000000009
set noise	0.6525286000000001
simulate	0.2948086000000001
simulate with saving	0.912353
simulate with saving, dt	1.0314177
simulate with printing results, dt	1.2612556000000001
Plot results	13.956509899999999
Metrics	0.03957639999999785
Surrogate Model Generation	3.281892899999999
surrogate sim	0.970258900000001
surrogate sim, dt	2.621425000000002

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.13332580000000005
import thrown object	0.5165169000000001
model initialization	0.10797599999999985
set noise	0.6647277
simulate	0.2958470000000002
simulate with saving	0.9068472999999999
simulate with saving, dt	1.0297329000000004
simulate with printing results, dt	1.247783
Plot results	13.8320134
Metrics	0.0390864999999998
Surrogate Model Generation	3.144150100000001
surrogate sim	0.9770746000000017
surrogate sim, dt	2.6025332

To:

Test	Time (s)
import main	0.1265449999999999
import thrown object	0.5042186
model initialization	0.10630849999999992
set noise	0.6412257000000001
simulate	0.3001212999999998
simulate with saving	0.9232532999999998
simulate with saving, dt	1.0504479999999998
simulate with printing results, dt	1.2806007
Plot results	13.7719918
Metrics	0.03928099999999901
Surrogate Model Generation	3.245565599999999
surrogate sim	0.9719778999999988
surrogate sim, dt	2.6214157

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.13147189999999997
import thrown object	0.5097252999999999
model initialization	0.11355079999999984
set noise	0.6377637
simulate	0.29351800000000017
simulate with saving	0.9409421
simulate with saving, dt	1.0400573999999998
simulate with printing results, dt	1.2372381
Plot results	14.530191700000001
Metrics	0.03623879999999957
Surrogate Model Generation	3.2939246000000004
surrogate sim	1.0751139000000016
surrogate sim, dt	2.7217170999999993

To:

Test	Time (s)
import main	0.12855550000000004
import thrown object	0.5056787
model initialization	0.1122668
set noise	0.6439989999999998
simulate	0.29316580000000014
simulate with saving	0.9360018999999999
simulate with saving, dt	1.0303980999999998
simulate with printing results, dt	1.2400145
Plot results	14.538938700000001
Metrics	0.03572540000000046
Surrogate Model Generation	3.3994209999999967
surrogate sim	1.0672311000000008
surrogate sim, dt	2.7004731

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.15702020000000005
import thrown object	0.6000137000000001
model initialization	0.13299639999999968
set noise	0.7462393999999999
simulate	0.35671609999999987
simulate with saving	1.1202105000000002
simulate with saving, dt	1.2594957999999998
simulate with printing results, dt	1.5033404
Plot results	17.3797922
Metrics	0.047261999999999915
Surrogate Model Generation	3.7935338000000023
surrogate sim	1.2269794999999988
surrogate sim, dt	3.127027400000003

To:

Test	Time (s)
import main	0.15752290000000002
import thrown object	0.6013116999999999
model initialization	0.1283660000000002
set noise	0.7684725000000001
simulate	0.34231560000000005
simulate with saving	1.084244
simulate with saving, dt	1.1998195999999997
simulate with printing results, dt	1.4522743999999994
Plot results	17.5534926
Metrics	0.042994999999997674
Surrogate Model Generation	3.9395884999999993
surrogate sim	1.2209287999999994
surrogate sim, dt	3.128122099999999

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.15561559999999997
import thrown object	0.6042581
model initialization	0.1309174999999998
set noise	0.7429891999999998
simulate	0.3390025000000003
simulate with saving	1.0738017000000002
simulate with saving, dt	1.1982933999999998
simulate with printing results, dt	1.4425410999999997
Plot results	17.444236500000002
Metrics	0.042586299999999966
Surrogate Model Generation	3.7883990999999995
surrogate sim	1.2173727999999997
surrogate sim, dt	3.1213704000000035

To:

Test	Time (s)
import main	0.16062489999999996
import thrown object	0.6137723999999998
model initialization	0.12951290000000037
set noise	0.7403966
simulate	0.3448675000000003
simulate with saving	1.0914207999999999
simulate with saving, dt	1.1974139
simulate with printing results, dt	1.4575692
Plot results	17.3157056
Metrics	0.04332210000000103
Surrogate Model Generation	3.9980814000000002
surrogate sim	1.2176911999999973
surrogate sim, dt	3.106106900000004

[codecov-commenter]

Codecov Report

Merging #512 (9e6a1ee) into dev (9e6a1ee) will not change coverage. The diff coverage is n/a.

:exclamation: Current head 9e6a1ee differs from pull request most recent head ab086f9. Consider uploading reports for the commit ab086f9 to get more accurate results

@@           Coverage Diff           @@
##              dev     #512   +/-   ##
=======================================
  Coverage   83.77%   83.77%           
=======================================
  Files          28       28           
  Lines        2213     2213           
=======================================
  Hits         1854     1854           
  Misses        359      359           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.12869489999999995
import thrown object	0.5065140000000001
model initialization	0.10723130000000003
set noise	0.6450965
simulate	0.2938771
simulate with saving	0.9094457
simulate with saving, dt	1.0363105999999997
simulate with printing results, dt	1.2559538000000003
Plot results	13.95476
Metrics	0.04021060000000176
Surrogate Model Generation	3.141329500000001
surrogate sim	0.9649602000000002
surrogate sim, dt	2.609476000000001

To:

Test	Time (s)
import main	0.12795350000000005
import thrown object	0.504513
model initialization	0.10645340000000014
set noise	0.6418508
simulate	0.2933416000000002
simulate with saving	0.9088200999999998
simulate with saving, dt	1.0230317999999996
simulate with printing results, dt	1.2484859000000004
Plot results	13.816634500000003
Metrics	0.03985799999999884
Surrogate Model Generation	3.2648642000000017
surrogate sim	0.9740898000000016
surrogate sim, dt	2.614570399999998

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.12740680000000004
import thrown object	0.5006301
model initialization	0.10855689999999996
set noise	0.6510296000000002
simulate	0.29322700000000035
simulate with saving	0.9063340999999996
simulate with saving, dt	1.0320585000000002
simulate with printing results, dt	1.2526025999999995
Plot results	13.995472099999999
Metrics	0.0386995999999975
Surrogate Model Generation	3.2039936000000004
surrogate sim	0.9692317999999993
surrogate sim, dt	2.621046700000001

To:

Test	Time (s)
import main	0.12451790000000007
import thrown object	0.49894170000000004
model initialization	0.10552570000000006
set noise	0.6505607
simulate	0.2964973999999998
simulate with saving	0.9167208000000002
simulate with saving, dt	1.0300156999999999
simulate with printing results, dt	1.2505685
Plot results	14.055973199999999
Metrics	0.04038840000000121
Surrogate Model Generation	3.3194632000000013
surrogate sim	0.9742576000000014
surrogate sim, dt	2.6181884999999987

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.1297469000000001
import thrown object	0.5096121
model initialization	0.11333520000000008
set noise	0.6555544000000002
simulate	0.29800210000000016
simulate with saving	0.9487875999999997
simulate with saving, dt	1.0479884
simulate with printing results, dt	1.2752955000000004
Plot results	14.917508600000001
Metrics	0.03549880000000272
Surrogate Model Generation	3.2353214
surrogate sim	1.0836848000000003
surrogate sim, dt	2.7364555000000017

To:

Test	Time (s)
import main	0.13098560000000004
import thrown object	0.5057533999999999
model initialization	0.11148139999999995
set noise	0.6371884000000001
simulate	0.29707110000000014
simulate with saving	0.9339995999999999
simulate with saving, dt	1.0334322
simulate with printing results, dt	1.2447912000000008
Plot results	14.5561672
Metrics	0.03622769999999775
Surrogate Model Generation	3.3463122999999975
surrogate sim	1.0640010999999987
surrogate sim, dt	2.6991899000000004

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.14887359999999994
import thrown object	0.5825176000000001
model initialization	0.12963409999999986
set noise	0.7443090999999997
simulate	0.32598079999999996
simulate with saving	1.0402907
simulate with saving, dt	1.1592900999999998
simulate with printing results, dt	1.3749713000000003
Plot results	16.4516719
Metrics	0.04090460000000107
Surrogate Model Generation	3.6579739999999994
surrogate sim	1.1910820000000015
surrogate sim, dt	3.0062800000000003

To:

Test	Time (s)
import main	0.14828660000000005
import thrown object	0.5983623999999998
model initialization	0.12205000000000021
set noise	0.7428472000000004
simulate	0.3420057000000001
simulate with saving	1.0528736999999997
simulate with saving, dt	1.1558156000000004
simulate with printing results, dt	1.3831797000000003
Plot results	16.303109900000003
Metrics	0.039936899999997166
Surrogate Model Generation	3.748984499999999
surrogate sim	1.1637151999999986
surrogate sim, dt	2.9630921000000043

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.17830199999999996
import thrown object	0.6047550000000002
model initialization	0.1630587000000001
set noise	0.8341522000000001
simulate	0.42423719999999987
simulate with saving	1.4135505
simulate with saving, dt	1.5280622
simulate with printing results, dt	1.9957743
Plot results	23.3397465
Metrics	0.047467700000005664
Surrogate Model Generation	4.819684000000002
surrogate sim	1.738457099999998
surrogate sim, dt	3.9084536000000014

To:

Test	Time (s)
import main	0.1620549
import thrown object	0.6336939000000001
model initialization	0.15328279999999994
set noise	0.8219324000000001
simulate	0.4268091000000003
simulate with saving	1.4089353000000004
simulate with saving, dt	1.4676437
simulate with printing results, dt	1.8261308000000005
Plot results	22.7019248
Metrics	0.04711650000000134
Surrogate Model Generation	4.881522199999999
surrogate sim	1.7781419000000014
surrogate sim, dt	4.0331740999999965

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.14352539999999991
import thrown object	0.6145551999999999
model initialization	0.1279106000000001
set noise	0.7987723999999998
simulate	0.33812880000000023
simulate with saving	1.1793772999999996
simulate with saving, dt	1.2688426000000002
simulate with printing results, dt	1.6956779000000006
Plot results	19.578307799999997
Metrics	0.043262699999999654
Surrogate Model Generation	4.334689599999997
surrogate sim	1.359919099999999
surrogate sim, dt	3.369251300000002

To:

Test	Time (s)
import main	0.18246319999999994
import thrown object	0.5772513000000001
model initialization	0.1337098000000001
set noise	0.7518601999999999
simulate	0.35903779999999985
simulate with saving	1.2877605
simulate with saving, dt	1.3359237000000004
simulate with printing results, dt	1.5950474999999997
Plot results	18.0502256
Metrics	0.040120099999999326
Surrogate Model Generation	4.1618981999999995
surrogate sim	1.3295431999999998
surrogate sim, dt	3.4034255000000044

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.13215310000000002
import thrown object	0.5087630000000001
model initialization	0.11412670000000014
set noise	0.6261198000000001
simulate	0.2978023000000003
simulate with saving	0.9462525999999998
simulate with saving, dt	1.05097
simulate with printing results, dt	1.2486714
Plot results	14.853579399999997
Metrics	0.03546520000000086
Surrogate Model Generation	3.214946099999999
surrogate sim	1.076985999999998
surrogate sim, dt	2.7273239999999994

To:

Test	Time (s)
import main	0.12917690000000004
import thrown object	0.5181997
model initialization	0.11207509999999998
set noise	0.6320406000000001
simulate	0.29655660000000017
simulate with saving	0.9524668000000003
simulate with saving, dt	1.0460657000000002
simulate with printing results, dt	1.2559529000000005
Plot results	14.8616785
Metrics	0.035805500000002155
Surrogate Model Generation	3.3589456999999996
surrogate sim	1.0846353000000022
surrogate sim, dt	2.708164400000001

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.12907590000000002
import thrown object	0.4992247000000001
model initialization	0.11308070000000003
set noise	0.6436177999999999
simulate	0.2933593000000001
simulate with saving	0.9375727
simulate with saving, dt	1.0454218000000002
simulate with printing results, dt	1.2475520000000007
Plot results	14.919246299999998
Metrics	0.03632070000000098
Surrogate Model Generation	3.2158434999999983
surrogate sim	1.0778693999999973
surrogate sim, dt	2.698953799999998

To:

Test	Time (s)
import main	0.13069169999999986
import thrown object	0.5084447999999999
model initialization	0.11112119999999992
set noise	0.635386
simulate	0.2978196999999998
simulate with saving	0.9530767999999998
simulate with saving, dt	1.0512408
simulate with printing results, dt	1.2596072999999999
Plot results	14.725115899999999
Metrics	0.035882499999999595
Surrogate Model Generation	3.3439669999999992
surrogate sim	1.0741069000000003
surrogate sim, dt	2.709088099999999

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.1606921
import thrown object	0.5564872999999999
model initialization	0.11386869999999982
set noise	0.6358193999999999
simulate	0.29341099999999987
simulate with saving	0.9413105000000002
simulate with saving, dt	1.0355509
simulate with printing results, dt	1.2360366999999997
Plot results	14.6795191
Metrics	0.03620960000000295
Surrogate Model Generation	3.196184200000001
surrogate sim	1.0513948000000006
surrogate sim, dt	2.691355699999999

To:

Test	Time (s)
import main	0.12897480000000017
import thrown object	0.5101811999999999
model initialization	0.11164010000000002
set noise	0.6433632
simulate	0.30003829999999976
simulate with saving	0.9453652000000003
simulate with saving, dt	1.0480146999999995
simulate with printing results, dt	1.2999644000000004
Plot results	14.531374600000003
Metrics	0.035428700000000646
Surrogate Model Generation	3.311279299999999
surrogate sim	1.0765127999999997
surrogate sim, dt	2.7012909999999977

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.1296946
import thrown object	0.5156741999999999
model initialization	0.11013760000000006
set noise	0.6442623000000001
simulate	0.29975130000000005
simulate with saving	0.9218259
simulate with saving, dt	1.0478780999999997
simulate with printing results, dt	1.2686918999999994
Plot results	14.419728
Metrics	0.040473200000000986
Surrogate Model Generation	3.2076629000000025
surrogate sim	0.9757654000000002
surrogate sim, dt	2.614509400000003

To:

Test	Time (s)
import main	0.1270696
import thrown object	0.5059393999999999
model initialization	0.10641260000000008
set noise	0.6465179000000001
simulate	0.2983635999999996
simulate with saving	0.9218061
simulate with saving, dt	1.0429227
simulate with printing results, dt	1.2654664000000002
Plot results	14.3628925
Metrics	0.04000689999999807
Surrogate Model Generation	3.2950207999999996
surrogate sim	0.9750540999999977
surrogate sim, dt	2.5954441999999993

[github-actions[bot]]

Benchmarking Results [Update] From:	Test	Time (s)
import main	0.1628467
import thrown object	0.5805527000000001
model initialization	0.15362240000000016
set noise	0.8156570999999997
simulate	0.42709209999999986
simulate with saving	1.2538291999999998
simulate with saving, dt	1.4172789000000003
simulate with printing results, dt	1.7836318999999996
Plot results	22.4540497
Metrics	0.050594600000000156
Surrogate Model Generation	4.943468099999997
surrogate sim	1.7987149000000002
surrogate sim, dt	4.0266205

To:

Test	Time (s)
import main	0.16446579999999988
import thrown object	0.6046745999999998
model initialization	0.14960130000000005
set noise	0.7594457000000001
simulate	0.4430187000000001
simulate with saving	1.3379906000000004
simulate with saving, dt	1.4164179999999993
simulate with printing results, dt	1.8444229000000005
Plot results	23.7056844
Metrics	0.047385099999999625
Surrogate Model Generation	4.8742296
surrogate sim	1.770607400000003
surrogate sim, dt	3.980792700000002