I see that from #19 the decision was made to match the Perl implementation by truncating integers. This was perhaps in part due to an "accumulation of error" from #18
For example, if you have 23.4 as a sample count you can upscale it to 234, then set factor to 0.1
Which implies that decimal points are supported.
In firestorm the units are in nanoseconds. At least on my computer, the minimum resolution for a sample is 100ns. To "unclutter" the text, I would like to display microseconds instead with fractional values. Eg: instead of 17,100ns, show 17.1μs. This would require fractional values to do well. The current behavior sometimes ends up truncating values to 0.
The accumulation of error can be avoided by not using floating point for calculations throughout, but instead only use the factor at the last moment when formatting the SVG text.
jonhoo
commented
2 years ago
I see that from #19 the decision was made to match the Perl implementation by truncating integers. This was perhaps in part due to an "accumulation of error" from #18
But, the documentation specifies that
Which implies that decimal points are supported.
In firestorm the units are in nanoseconds. At least on my computer, the minimum resolution for a sample is 100ns. To "unclutter" the text, I would like to display microseconds instead with fractional values. Eg: instead of 17,100ns, show 17.1μs. This would require fractional values to do well. The current behavior sometimes ends up truncating values to 0.
The accumulation of error can be avoided by not using floating point for calculations throughout, but instead only use the
factorat the last moment when formatting the SVG text.