Precision in grid export

[stephanmg]

I'm wondering why in the UGX file export the default precision for the vertex coordinates is set to 18 by using std::set_precision(18). What's the reasoning behind this?

Wouldn't it be better to set it to this: std::numeric_limits<number>::digits10 + 1? This will, I guess, also be effected by the CMake option for the precision then, and I'm not sure if this should happen or not, or if this is the reason why one sets the precision above manually. It just seems a bit odd.

More a comment then a question: Also in UG file export we have 12 or 24 (LGM) digits of precision and for ART file export it's 20.

[sreiter]

According to https://en.cppreference.com/w/cpp/types/numeric_limits/max_digits10, at least max_digits10 (not digits10) digits should actually be used to uniquely store distinct floating point values in text form. A simple program on my machine told me that the concrete value of max_digits10 for double is 17, and 9 for float. Note that max_digits10 is c++11, which wasn't available/used at the time the required precision was fixed. While 18 is off by one for doubles and a bit too high for float values, std::numeric_limits<double>::digits10 + 1 would be a bit too small (my program reported 16).

I agree that the number of digits in LGM and ART files seem a bit high. Please note that those are rarely (if ever) used... so maybe this is not a big issue, I guess.

[stephanmg]

Thank you @sreiter. So, vertex coordinates are always guaranteed to be double precision? In particular the Vertex Attachment aPosition has double precision always? I couldn't find where the type is specified for aPosition (double/float).

[bsumirak]

So, vertex coordinates are always guaranteed to be double precision?

If you refer to the positions given in the file, the answer is no. How could they be? They are only guaranteed to have enough digits to support double precision. Whether these digits are actually used depends on who/whatever produces the file: If they use float coordinates, the precision is only float. And any domain imported from that file will only have float precision as well, obviously.

If you refer to the values contained in aPosition (see common_attachments.cpp), the answer is still no. The positions are given as ug::MathVector<3, number> and number is float if the preprocessor symbol UG_SINGLE_PRECISION is defined (cf. ugbase/common/types.h).

[stephanmg]

Ouch! I totally oversaw that APosition3 is typedef'ed further in commons_attachment.cpp. That's were my confusion arose. Thanks!