search

Semi-ATE / STDF

STDF Library
MIT License
43 stars 19 forks source link

Floating Point Precision #57

Open seimit opened 1 year ago

seimit commented 1 year ago

Discussed in https://github.com/Semi-ATE/STDF/discussions/56

Originally posted by **Mike-Purtell** December 22, 2022 I have been using the Semi-ATE STDF library for about 1 year, to process STDF files from Advantest 93K SmartTest 8 (java) OS. Works very well for me, could not be happier for having found them. After the STDF conversion, the test measurement results are stored as float-64. The STDF spec lists measurement results and test limits as float-32. In this sense, measurement results take up twice as much memory as they should, which may slow down the conversion, and lower the maximum STDF file size possible. I have been optimizing my data analysis scripts by converting all floats from 64 to 32 bits after the STDF conversion, and am seeing impressive performance improvements My idea/suggestion is for an option to change all of the 64-bit floating point results to 32-bit as early as possible during the STDF conversion. 32-bit floats have similar resolution to 7 decimal digits of resolution, more than enough for my work on RF transceivers ICs. I have no complaints about the speed of the conversion tool, but if this makes it faster, who wouldn't take it? Also, I have had very large stdf files ( > 1G) crash when my PC ran out of memory. Perhaps reducing the size of value allocated to float-64 would allow conversion of larger STDF files.
seimit commented 1 year ago

Dear Mike, Thank you for your idea! I think it is worth to implement this request and I created an issue from your idea. Not sure when I will have free time to make it, but you will be notified, once I start working on it. Kind Regards, Seimit