sunspec / pysunspec2

SunSpec Python library for interfacing with SunSpec devices.
Apache License 2.0
57 stars 21 forks source link

Computed values contain floating point error #72

Closed jmmgenerac closed 1 year ago

jmmgenerac commented 1 year ago

When a point has a nonzero scale factor, its cvalue (computed, scaled value) can contain floating point error. This causes problems when an application wants to verify a write by comparing the scaled value it has written to the device with the scaled value it reads back from the device.

For example, take a data point having an sf_value of -1. When the value is 302, the cvalue will read back (on my machine) as 30.200000000000003. The cvalue for this point really should be rounded to 30.2

Suggested change from (device.py#L220):

            if self.sf_value:
                sfv = self.sf_value
                if sfv:
                    v = v * math.pow(10, sfv)

To this:

            if self.sf_value:
                sfv = self.sf_value
                if sfv:
                    v = round(v * math.pow(10, sfv), -1 * sfv)