number precision/text storage

[jedwards1211]

@vpicaver I just realized, JSON parsing will probably ruin the precision of numbers like 0.3; kind of a big problem! We need to investigate this...

[jedwards1211]

Yeah this is a rather ugly problem with JSON. Perhaps we should use strings everywhere? :/

[jedwards1211]

unless parsers that can return the exact number literal from the JSON are common for most languages...

[vpicaver]

Yea. I was running into a precision issues with cavewhere storing double instead of strings. I know realize that it much better to store any number the user enters as a string. This will preserve what they type in. I was hope to change all values to strings. I think it's a good idea for the JSON format to just use strings for user entered data.

Phi|ip

On Mon, Jan 25, 2016 at 3:06 AM, Andy Edwards notifications@github.com wrote:

unless parsers that can return the exact number literal from the JSON are common for most languages...

— Reply to this email directly or view it on GitHub https://github.com/jedwards1211/metacave/issues/10#issuecomment-174432331 .

[jedwards1211]

Yeah, I'm going to convert it. I have to say, it bums me out that it doesn't seem more common for JSON parsers to support getting the original number literal, because it feels a lot less elegant to use strings...oh well.

[vpicaver]

Yea. Floating point values are way faster to use and use less memory...

On Wednesday, January 27, 2016, Andy Edwards notifications@github.com wrote:

Yeah, I'm going to convert it. I have to say, it bums me out that it doesn't seem more common for JSON parsers to support getting the original number literal, because it feels a lot less elegant to use strings...oh well.

— Reply to this email directly or view it on GitHub https://github.com/jedwards1211/metacave/issues/10#issuecomment-175790082 .

Phi|ip

Sent from Gmail Mobile

[jedwards1211]

all right, I just fixed this :)

[vpicaver]

Cool :) now I need to update it in the meta cave branch in cavewhere

On Wednesday, January 27, 2016, Andy Edwards notifications@github.com wrote:

all right, I just fixed this :)

— Reply to this email directly or view it on GitHub https://github.com/jedwards1211/metacave/issues/10#issuecomment-175805493 .

Phi|ip

Sent from Gmail Mobile

[jedwards1211]

revisiting this...

I realized some time ago in unrelated work that my anxiety about floating point precision was completely off base. Double precision floating point numbers are beyond sufficient for most human-entered numbers. I had somehow mistakenly arrived at the conclusion that a lot of numbers would get mangled when saved back to the file, but I'm beginning to wonder if Java had shitty number to string conversion or something that mangled 0.2 to 0.19999999 etc where JavaScript number to string conversion would just output 0.2. Either that or my experience with single precision led to my misconception about double precision.

Double precision is just fine even for lat/long values:

> String(parseFloat('139.288381234'))
"139.288381234"

So, I think we should either

• go back to just using numbers for measurements
• or allow either numbers or strings

@vpicaver

I was running into a precision issues with cavewhere storing double instead of strings

Are you sure about this? Do you have an example of the problem? As I mentioned above, unless the builtin number to string conversion function you were using was subpar, there's no way you would run into this problem with human-entered numbers, especially cave survey measurements.

[jedwards1211]

Maybe the problem you were having was due to the fact that floating-point error from calculations often shows up in number formatting?

> 0.1
0.1
> 0.2
0.2
> 0.3
0.3
> 0.1 + 0.2
0.30000000000000004

[vpicaver]

I think anything that's user enterable should be a string therefore preserve measurement data as collected. I'm probably going to change Cavewhere to store data in memory as strings and convert it when needed to floating points (doubles). It's a bit slower but it will not loose precision or have rounding issues. Any calculated values will be stored as a double and then displayed to a particle decimal point. I think most or all values in MetaCave are user enterable values and should be stored as strings?

On Mon, Jan 7, 2019 at 5:10 AM Andy Edwards notifications@github.com wrote:

Maybe the problem you were having was due to the fact that floating-point error from calculations often shows up in number formatting?

0.1 0.1 0.2 0.2 0.3 0.3 0.1 + 0.2 0.30000000000000004

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jedwards1211/metacave/issues/10#issuecomment-451885401, or mute the thread https://github.com/notifications/unsubscribe-auth/AFShg1Ej_Mn4cMWVe5KYQNebkEQbAt9dks5vAx0ZgaJpZM4HLZ5J .

[jedwards1211]

@vpicaver my point is that a JSON number literal like 253.2843911 does preserve the measurement as collected.

And actually, JSON itself places no limits on the precision of numbers. Parsing and stringifying will lose precision if the JSON api converts to finite-precision numbers, but double precision is sufficient for cave measurements and GPS coordinates:

> JSON.stringify(JSON.parse('{"lat": 28.81823412, "lng": 92.83124234}'))
"{"lat":28.81823412,"lng":92.83124234}"

[jedwards1211]

@vpicaver here's an example of a lossless JSON parser in JS: https://github.com/josdejong/lossless-json Google's GSON also parses numbers losslessly. Unfortunately, Qt's JSON parser does not :( https://bugreports.qt.io/browse/QTBUG-28560?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true

[jedwards1211]

However, https://github.com/spotify/spotify-json is a C++ JSON parser that preserves number precision

[vpicaver]

I'm using Qt JSON parser. I curious if it preserves number precision. Anyways, I think measurements and user input should be strings instead of numbers just to guarantee no loose in precision.

On Tue, Jan 8, 2019 at 3:55 PM Andy Edwards notifications@github.com wrote:

However, https://github.com/spotify/spotify-json is a C++ JSON parser that preserves number precision

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jedwards1211/metacave/issues/10#issuecomment-452448113, or mute the thread https://github.com/notifications/unsubscribe-auth/AFShg1eDqcJio87VEDZd2UoVrVeNZFJ1ks5vBQXQgaJpZM4HLZ5J .