search

APSIMInitiative / APSIM.PerformanceTests

0 stars 4 forks source link

Use a tolerance on R2 when comparing against accepted. #30

Open hol353 opened 4 years ago

hol353 commented 4 years ago

Look at my pull request:

https://github.com/APSIMInitiative/ApsimX/pull/4971

The R2 is coming up different but it is only different in the 5 decimal place I think.

hol430 commented 4 years ago

From memory I think there is already a tolerance, but it's a relative tolerance set to something like 10%. Of course, 10% of a small number is an even smaller number, so it can be a bit misleading.

hol353 commented 4 years ago

Yes you are probably right. This is the R2 difference:

Capture
hol430 commented 4 years ago

Wow, an r2 of 9e-5. I can think of faster ways to generate white noise!

I guess we should change the r2 test to check if r2 has changed by an absolute amount -e.g. 0.05.

hol353 commented 4 years ago

We are quite happy with an R2 of 9e-5. We want to make sure we maintain that level of fit.