search

pambsoftware / beefcake

Training planner and logger for Android
1 stars 0 forks source link

Rounding error #10

Open windmueller opened 8 years ago

windmueller commented 8 years ago

For certain combination of weights, there seems to be a rounding error.

screenshot_2015-11-18-18-46-11

My best guess is that you use a data type like "float" despite the fact that you do not require high precision floating point operations. See for example "Effective Java", Item 48.

ripern commented 8 years ago

You are correct. It's because we're using float. What would you recommend instead in sqlite? We need 2 decimals. A simple solution now would be to round the numbers in a logical way in the UI.

JimRorie commented 8 years ago

You should use decimal(10,2) (or appropriate precision) to avoid issues assoc with floats.

------ Original Message ------ From: "Magnus Bondesson" notifications@github.com To: "pambsoftware/beefcake" beefcake@noreply.github.com Sent: 11/29/2015 12:21:03 PM Subject: Re: [beefcake] Rounding error (#10)

You are correct. It's because we're using float. What would you recommend instead in sqlite? We need 2 decimals. A simple solution now would be to round the numbers in a logical way in the UI.

— Reply to this email directly or view it on GitHub.

This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

windmueller commented 8 years ago

Since you just need two decimals, simply multiply the value by 100 and store integer values.

ripern commented 8 years ago

Actually the data in db looks good. However, the additions of weights to calculate the totalWeight was causing the rounding error. Have fixed it there now.

Regarding decimal(10,2) it doesn't exist in Sqlite, https://www.sqlite.org/datatype3.html, and will be seen as a NUMERIC type. People seemed to be confused about how to grasp it (so am I), http://stackoverflow.com/questions/21757722/how-to-use-sqlite-decimal-precision-notation @JimRorie Do you know if the affinity is stored somewhere in Sqlite so it actually uses that information (2 decimals) when storing a value in such a column?

Thanks for your input guys!

JimRorie commented 8 years ago

My sqlite is a little rusty, so I led you astray. There doesn't appear to be bcd handling. The two paths I have seen are storage of values as text (not a fan) and storage as integer with implied decimals (x*100 in, x/100 out). I don't think explicitly casting will work.

------ Original Message ------ From: "Magnus Bondesson" notifications@github.com To: "pambsoftware/beefcake" beefcake@noreply.github.com Cc: "JimRorie" jfrorie@gmail.com Sent: 12-06-2015 7:11:08 AM Subject: Re: [beefcake] Rounding error (#10)

Actually the data in db looks good. However, the additions of weights to calculate the totalWeight was causing the rounding error. Have fixed it there now.

Regarding decimal(10,2) it doesn't exist in Sqlite, https://www.sqlite.org/datatype3.html, and will be seen as a NUMERIC type. People seemed to be confused about how to grasp it (so am I), http://stackoverflow.com/questions/21757722/how-to-use-sqlite-decimal-precision-notation @JimRorie Do you know if the affinity is stored somewhere in Sqlite so it actually uses that information (2 decimals) when storing a value in such a column?

Thanks for your input guys!

— Reply to this email directly or view it on GitHub.