[Docs] hex values for pixel data in the "Sample File" in the docs don't match the floating point values

[jportway]

On this page the hex data listed for the file doesn't seem to match the floating point values written below it. Neither the FP16 or the FP32 values seem to match exactly their hex representations : eg.

The first FP32 value (pixel (0,0) in the Z channel) is listed as

5C	28	81	3A
0.000985395

but the little endian IEEE-754 representation of 0.000985395 should be 0x 5F 28 81 3A unless I'm very mistaken.

The first non zero FP16 value (pixel (1,0) in the G channel) is listed as :

54	29
0.042

but by my calculations 0.042 in FP16 should be 0x 60 29

There seem to be two possibilities for how this has happened : either this is an exotic binary FP representation (which seems unlikely?), or the floating point numbers in the table represent some original pixel data that has undergone some kind of transformation prior to being encoded (maybe log-> linear ?)- in which case the "sample file" documentation is a bit misleading / confusing because all the other values in the description relate to the data in the file, rather than original pixel data.

If this discrepancy in the floating point encoding represents something deliberate it should probably be noted somewhere in the document - if it is noted somewhere, then I can't find it. I've been using this "sample file" data in test to check conformance of a simple Scala EXR codec, and it's taken me many hours to work out why its been failing - hopefully someone else can be saved the frustration if the docs are updated.

[cary-ilm]

I see the same thing you're seeing, I would expect 5F where the table says 5C. I somehow doubt it's a typo since all the first bytes are off, but I don't have an explanation. That document predates me, and it probably dates to the first release of the library 20 years ago. I'll try contacting the original author and ask for any insight.

[peterhillman]

Isn't this just rounding? 5C 28 81 3A is 0.000985394697636 which rounds to 0.000985395 to 6 significant figures 54 C9 as a half is 0.041626 , which is 0.042 to 2 significant figures. So if you convert from the hex to the decimal, they are correct, but you get different values if you convert the rounded decimal representations back into hex.

[jportway]

Rounding seems like a plausible explanation. Nonetheless, because it's presented as a "sample file" in the documentation, maybe at least there should be an explanation of it? I know the documentation isn't really meant as guidance for people creating alternative implementations, but it's de facto going to be used that way, and this is really confusing for anyone using the document (as I was) as a guide for implementing the format. Maybe just add a note?

[cary-ilm]

I will look into updating the document to be less confusing.

[annguyen-ilm]

I have checked the documentation values with the actual sample.exr file generated and it does seem that the documentation round up values to some decimal places with the comparison of all the values view in Nuke. I think the hex values should be okay with the represented numeric values. The sample.exr will be included for those wanting to see the comparison.