Internationalisation problem when parsing floats (comma vs point)

[GoogleCodeExporter]

When I open verletSim.crl, gravity is set to zero.

But in the file I can read:

    executeCommand('CreateNode', className = 'Vec3', name = 'gravity', specializationPreset = 'single', parentNode = topNode)
    executeCommand('SetAttributeValue', attribute = topNode + '.gravity.x', value = '[0] 3')
    executeCommand('SetAttributeValue', attribute = topNode + '.gravity.y', value = '[-0.8] 3')
    executeCommand('SetAttributeValue', attribute = topNode + '.gravity.z', value = '[0] 3')

Don't know why it do that...

Same for "endPoint". When I open the y value is set to 12.000, but in the file 
it is 12.7:

    executeCommand('CreateNode', className = 'Vec3', name = 'endPoint', specializationPreset = 'single', parentNode = topNode)
    executeCommand('SetAttributeValue', attribute = topNode + '.endPoint.x', value = '[10] 3')
    executeCommand('SetAttributeValue', attribute = topNode + '.endPoint.y', value = '[12.7] 3')
    executeCommand('SetAttributeValue', attribute = topNode + '.endPoint.z', value = '[0] 3')

bad convertion to integer? :(

Original issue reported on code.google.com by dorian.f...@gmail.com on 10 Dec 2011 at 12:01

[GoogleCodeExporter]

I can't repro on Os X, there must be some Os dependant issue when reading 
values from string.
What Os were you running from? 

Original comment by aintergu...@gmail.com on 10 Dec 2011 at 12:22

[GoogleCodeExporter]

Arf! I'm on Linux.

So I have to investigate on this. Can you just tell me where the "string to 
float" conversion is supposed to be done? I will try to find where the problem 
come. :)

Original comment by dorian.f...@gmail.com on 10 Dec 2011 at 12:50

• Changed state: Accepted

[GoogleCodeExporter]

Yes, look for Numeric::setFromString and also stringUtils::parseFloat and 
parseInt.

Original comment by aintergu...@gmail.com on 10 Dec 2011 at 1:33

[GoogleCodeExporter]

So, I've made a lot of modifications to know what was the problem:

else if(type == Numeric::numericTypeFloat || type == 
Numeric::numericTypeFloatArray){
    std::vector<std::string> values;
    stringUtils::split(valuesStr, values, ",");
    _floatValues.clear();
+   std::cout<< "valuesStr " << valuesStr << std::endl;
    for(int i = 0; i < values.size(); ++i){
+       std::cout<< "values[i] " << values[i] << std::endl;
+       std::cout<< "atof(values[i].c_str()) " << atof(values[i].c_str()) << 
std::endl;
        float value = stringUtils::parseFloat(values[i]);
+       std::cout<< "value " << value << std::endl;
        _floatValues.push_back(value);
+       std::istringstream instr;
+       instr.str(values[i]);
+       float x;
+       instr >> x;
+       std::cout<< "istringstream " << x << std::endl;
    }

    _size = _floatValues.size();
}

executeCommand('SetAttributeValue', attribute = 'root.gravity.y', value = 
'[-0.8] 3')
valuesStr -0.8
values[i] -0.8
atof(values[i].c_str()) -0
value -0
istringstream -0.8

But istringstream seems to be slower.

There is a lot of interesting conversation about atof problem:
http://www.velocityreviews.com/forums/t280779-why-doesnt-atof-work.html

http://ubuntuforums.org/showthread.php?t=1710535
"A quick google indicates that atof is deprecated"

http://cboard.cprogramming.com/c-programming/94806-convert-string-double.html
"atof is deprecated in favor of strtod"

But after some tests strtod or strtof have the same problems. So I was thinking 
this was not an atof problem. But I've tryed this:

std::cout<< "atof("-0.8") " << atof("-0.8") << std::endl;

and get:

atof("-0.8") -0

Crazy...

EDIT:

Ok, I've found something interesting:

double dx = strtod("-0.8",NULL);
std::cout<< "dx " << dx << std::endl;
> dx -0

double dx = strtod("-0,8",NULL);
std::cout<< "dx " << dx << std::endl;
> dx -0.8

There was not a point "." here but a comma ",".

Same for atof:

float fx = atof("-0.8");
std::cout<< "fx " << fx << std::endl;
> dx -0

float fx = atof("-0,8");
std::cout<< "fx " << fx << std::endl;
> dx -0.8

Same for sscanf:

sscanf("-0.8", "%f", &dx);
std::cout<< "sscanf " << dx << std::endl;
> dx -0

sscanf("-0,8", "%f", &dx);
std::cout<< "sscanf " << dx << std::endl;
> dx -0.8

The problem seems to came from "localisation"....

And after test 
(http://www.cplusplus.com/reference/clibrary/clocale/localeconv/):

struct lconv *local = localeconv();
std::cout<< "local->decimal_point " << local->decimal_point << std::endl;

I've got this:
> local->decimal_point ,

I'm almost sure if you guys copy this two lines you will have a point ".", not 
a comma ",".

So the problem is about location. If I search "atof locale" in google I see 
many people that try to get a "locale independent atof"...

The simplest solution seems to be using istringstream...

What you guys do you think of this? :(

Original comment by dorian.f...@gmail.com on 14 Dec 2011 at 10:25

• Changed title: Internationalisation problem when parsing floats (comma vs point)

[GoogleCodeExporter]

After adding:

std::cout<< "current local " << setlocale(LC_NUMERIC, NULL) << std::endl;

This give the current local for LC_NUMERIC stuff. And I get:

current local fr_FR.UTF-8

So as I was thinking, I'm in "locale lang".

If I add:
else if(type == Numeric::numericTypeFloat || type == 
Numeric::numericTypeFloatArray){
+   setlocale(LC_NUMERIC, "C");

...to make every numeric operations of C to the default "C" locale, everything 
is well loaded.

I now have -0.8 in gravity of verletSim example. :)

But I've done another test: I've renammed the node: "gravity" to "gravityé", 
save the .crl file and try to reopen it. coral has removed the "é".

Anyway, this is a problem... Set a locale to "C" just to parse files is a bad 
idea I think...

I think we need a "real utf8 string library" to do conversions.

https://developer.apple.com/library/mac/#documentation/Cocoa/Reference/Foundatio
n/Classes/NSString_Class/Reference/NSString.html

Boost seems to have interesting stuff:
http://stackoverflow.com/questions/7548564/float-stdstring-conversion-alternativ
e
http://www.boost.org/doc/libs/1_47_0/libs/spirit/doc/html/spirit/qi/reference/nu
meric/real.html

boost::lexical_cast seems to be exactly what we need:
http://www.boost.org/doc/libs/1_48_0/doc/html/boost_lexical_cast.html
Seen here: 
http://social.msdn.microsoft.com/forums/en-US/vclanguage/thread/ca0218a7-1290-4d
de-a32f-4c7d331f5dbc:

float f;
std::string s;

f  = boost::lexical_cast<float>(s);
s = boost::lexical_cast<std::string>(f);

I know IBM also have a big string library to deal with strings...

I will test tomorr....

Ok, I'd just test with boost:lexical_cast and it work like a charm....

float f  = boost::lexical_cast<float>(values[i]);
std::cout<< "boost lexical " << f << std::endl;
boost lexical -0.8

*Boost seems to be a big world no one can explore in a single life...*

Ok so this seems to be a very elegant way to deal with string/numerics 
conversions. Maybe we should do this for the whole parser?...

I mean, replace stringUtils parsers by boost::lexical_cast?

If Andrea is ok with that I will submit a patch tomorrow to solve this problem 
using boost::lexical_cast (and removing every stringUtils::parseFloat instance 
in the code) but I think we could think about replace some stringutils in the 
futur by others boost::lexical_cast.

What do you think of all of this Andrea?

Original comment by dorian.f...@gmail.com on 14 Dec 2011 at 11:09

[GoogleCodeExporter]

This seems to be all fixed, thanks Dorian.

Original comment by aintergu...@gmail.com on 3 Jan 2012 at 11:08

• Changed state: Done