wdas / partio

C++ (with python bindings) library for easily reading/writing/manipulating common animation particle formats such as PDB, BGEO, PTC. https://wdas.github.io/partio
https://www.disneyanimation.com/open-source/partio/
Other
458 stars 133 forks source link

Partio file I/O is locale dependent #78

Closed lgritz closed 3 years ago

lgritz commented 4 years ago

The stream input/output for GEO files will default to the global system locale. For users in non-US countries who are running with locales set to use , as the decimal separator, this will result in incorrect reads of typical geo files (since the . decimal point will TERMINATE parsing of a number rather than merely indicate the fractional part), and writing will produce floating point numbers with commas which will then be unreadable when in the US style locale.

I noticed this reading GEO files, but I suspect there may be similar issues scattered all through the code base, since grep turns up not a single instance of the word "locale." I feel your pain, it took me a long time to root these issues out of my code bases a couple years ago. (In fact, looking for one more wayward locale bug in OSL is how I stumbled onto this problem in partio, though it was not ultimately the cause of my woes.)

I think that the reasonable way to handle this is to decide that all parsing and output in your library should use the "C" locale (US conventions, also known in as "locale independent") so that file I/O will behave identically regardless of how a user's system-wide locale is set, or how an app sets the locale.

Some DCC apps set the locale to "C" to force locale independence, but others prefer to keep the system locale in order to have various UI elements use the local/foreign customary presentation format. A library like partio will inherit the locale set by the app (or the global one) and is expected not to mess with it. So you just have to deal with it and handle for each kind of input or output.

For stream I/O, this is fairly straightforward. In GEO.cpp, for example, you see something like

unique_ptr<istream> input(Gzip_In(filename,ios::in));
if(!*input){
    if(errorStream) *errorStream<<"Partio: Can't open particle data file: "<<filename<<endl;
    return 0;
}

and what you want to do is subsequently (but before any reads)

input->imbue (std::locale::classic());  // force C locale

and the same for output streams.