open-flash / swf-parser

Rust and Typescript parsers for SWF
GNU Affero General Public License v3.0
28 stars 8 forks source link

[rs] Fix `transmute_u16_to_f16` precision #54

Closed demurgos closed 5 years ago

demurgos commented 5 years ago

The Rust implementation of transmute_u16_to_f16 is losing precision.

Example:

3    7    a    3
0011 0111 1010 0011
seee eemm mmmm mmmm
s: 0          -> 1
e: 01101      -> 13
m: 1110100011 -> 931
normal form: s * 2**(e-15) * (1 + m/1024)
-> 2**(-2) * (1 + 931/1024)
-> 1955 / 4096
-> 0.477294921875

But Rust returns 0.47729492.

demurgos commented 5 years ago

The Rust precision works fine in fact. The issue is that V8 treats all floats as f64 and serializes with a high precision while serde serializes as an f32 and uses less digits. Here is the neighborhood of the example value with serde:

0.4772949  prev
0.47729492 value
0.47729495 next

This is not a parser issue but an AST one.