search

VowpalWabbit / vowpal_wabbit

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.
https://vowpalwabbit.org
Other
8.48k stars 1.93k forks source link

Model files are not portable #2079

Open peterychang opened 5 years ago

peterychang commented 5 years ago

Model files contains bit patterns for certain types of data (like floats). Since the size and the precise bit-wise format of a float is not guaranteed by standards, the resulting model files are technically not portable

kumpera commented 5 years ago

Floats are ieee 754, which works all over the place and are well specified.

peterychang commented 5 years ago

Thats not guaranteed

float - single precision floating point type. Usually IEEE-754 32 bit floating point type double - double precision floating point type. Usually IEEE-754 64 bit floating point type

They're usually IEEE754, but not guaranteed to be. This won't be a problem for most systems, but I know some embedded systems for example, do not use that standard

lokitoth commented 4 years ago

Can we define the stored model file as IEEE-754, and any system on which we do not read correctly due to using the wrong float type is buggy?

(Alternatively, converging on an IDL-based serializer would also work here)

thealphadollar commented 4 years ago

@lokitoth @peterychang What do you think is the progress on this issue?