DHT11 Sensor - inaccurate data

superdj4life commented 7 years ago

Mycodo Issue Report:

Specific Mycodo Version: 4.0.26
git commit hash:

Problem Description

Please list: incorrect sensor data using DHT11

what were you trying to do
add sensor DHT11 and begin reporting temp/humidity data

Errors

List any errors you encountered. no errors encountered, the readings are incorrect here's a paste from the current readings Measurement | Timestamp 192 °C (377.60 °F) | 2016/11/30 18:39:04 192 % | 2016/11/30 18:39:04 222.88 °C (433.18 °F) DP | 2016/11/30 18:39:04
Copy and pasting crash logs, or link to any specific Not a great developer, sorry I can't site specific lines of code. if it helps the DHT11.py sensor class states: "The DHT11 class is a stripped version of the DHT22 sensor code by joan2937. You can find the initial implementation here: https://github.com/joan2937/pigpio/blob/master/EXAMPLES/Python/DHT11_SENSOR/dht11.py"

I ran this example code and was able to get good results from the sensor, output below:

pi@raspberrypi:~ $ sudo python dht11test.py temperature: 22 humidity: 23 temperature: 22 humidity: 23 temperature: 22 humidity: 23 temperature: 22 humidity: 23

Steps to Reproduce the issue:

How can this issue be reproduced?

add DHT11 sensor
Choose correct GPIO (I'm using 04)
Save and activate the sensor
go to the "Live" view and you will the sensor data is inaccurate

Additional Notes

No, thank you all for the hard work, this looks amazing.

kizniche commented 7 years ago

Hmm. Could it be that the decimal is misaligned?

192 °C 192 %

22 °C 23 %

Can you provide more data from Mycodo? If you check "Enable Export" from the graph options (plus symbol at the top-right of a graph), refresh the page, then you can export the data to CSV from the drop-down menu at the top-right of the graph.

kizniche commented 7 years ago

Also, do these values change when the sensor is heated or cooled and do they change proportionally?

kizniche commented 7 years ago

I've been reviewing the differences between the file you linked (dht11.py) vs what's used in Mycodo, (dht11.py), but I haven't found any differences I would attribute to the error. I don't have a DHT11 sensor to test, so I'm sort of driving blind at the moment.

superdj4life commented 7 years ago

Sure thing, That would make sense but No, the values stay high on every reading. CSV export attached as requested.

On Wed, Nov 30, 2016 at 7:22 PM, Kyle Gabriel notifications@github.com wrote:

Also, do these values change when the sensor is heated or cooled and does it change proportionally?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kizniche/Mycodo/issues/158#issuecomment-264060516, or mute the thread https://github.com/notifications/unsubscribe-auth/AXJmlehwN0oUlIS86X6X5m1yuIe4pcxFks5rDi97gaJpZM4LA6eE .

kizniche commented 7 years ago

I don't see a file attached.

superdj4life commented 7 years ago

Here you go sorry

On Wed, Nov 30, 2016 at 7:29 PM, Nathan Johnson superdj4life@gmail.com wrote:

Sure thing, That would make sense but No, the values stay high on every reading. CSV export attached as requested.

On Wed, Nov 30, 2016 at 7:22 PM, Kyle Gabriel notifications@github.com wrote:

Also, do these values change when the sensor is heated or cooled and does it change proportionally?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kizniche/Mycodo/issues/158#issuecomment-264060516, or mute the thread https://github.com/notifications/unsubscribe-auth/AXJmlehwN0oUlIS86X6X5m1yuIe4pcxFks5rDi97gaJpZM4LA6eE .

kizniche commented 7 years ago

Still no attachment. I don't think you can attach files to issues by email. I believe you have to use the browser.

superdj4life commented 7 years ago

sorry about that, here's the raw output of the .csv

"DateTime","DHT11 (BgApD4EA) Temperature","DHT11 (BgApD4EA) Humidity","DHT11 (BgApD4EA) Dew Point" "2016-11-30 17:46:00",0,0,0 "2016-11-30 17:48:00",52.666666666666664,52.733333333333334,59.428 "2016-11-30 17:50:00",192,192,222.88000000000008 "2016-11-30 17:52:00",192,192,222.88000000000008 "2016-11-30 17:54:00",192,192,222.88000000000008 "2016-11-30 17:56:00",192,192,222.88000000000008 "2016-11-30 17:58:00",192,192,222.88000000000008 "2016-11-30 18:00:00",192,192,222.88000000000008 "2016-11-30 18:02:00",192,192,222.88000000000008 "2016-11-30 18:04:00",192,192,222.88000000000008 "2016-11-30 18:06:00",192,192,222.88000000000008 "2016-11-30 18:08:00",192,192,222.88000000000008 "2016-11-30 18:10:00",192,192,222.88000000000008 "2016-11-30 18:12:00",192,192,222.88000000000008 "2016-11-30 18:14:00",192,192,222.88000000000008 "2016-11-30 18:16:00",192,192,222.88000000000008 "2016-11-30 18:18:00",192,192,222.88000000000008 "2016-11-30 18:20:00",192,192,222.88000000000008 "2016-11-30 18:22:00",192,192,222.88000000000008 "2016-11-30 18:24:00",192,192,222.88000000000008 "2016-11-30 18:26:00",192,192,222.88000000000008 "2016-11-30 18:28:00",192,192,222.88000000000008 "2016-11-30 18:30:00",192,192,222.88000000000008 "2016-11-30 18:32:00",192,192,222.88000000000008 "2016-11-30 18:34:00",192,192,222.88000000000008 "2016-11-30 18:36:00",192,192,222.88000000000008 "2016-11-30 18:38:00",192,192,222.88000000000008 "2016-11-30 18:40:00",192,192,222.88000000000008 "2016-11-30 18:42:00",192,192,222.88000000000008 "2016-11-30 18:44:00",192,192,222.88000000000008 "2016-11-30 18:46:00",192,192,222.88000000000008 "2016-11-30 18:48:00",192,192,222.88000000000008 "2016-11-30 18:50:00",192,192,222.88000000000008 "2016-11-30 18:52:00",192,192,222.88000000000008 "2016-11-30 18:54:00",192,192,222.88000000000008 "2016-11-30 18:56:00",192,192,222.88000000000008 "2016-11-30 18:58:00",192,192,222.88000000000008 "2016-11-30 19:00:00",192,192,222.88000000000008 "2016-11-30 19:02:00",192,192,222.88000000000008 "2016-11-30 19:04:00",192,192,222.88000000000008 "2016-11-30 19:06:00",192,192,222.88000000000008 "2016-11-30 19:08:00",192,192,222.88000000000008 "2016-11-30 19:10:00",192,192,222.88000000000008 "2016-11-30 19:12:00",192,192,222.88000000000008 "2016-11-30 19:14:00",192,192,222.88000000000008 "2016-11-30 19:16:00",192,192,222.88000000000008 "2016-11-30 19:18:00",192,192,222.88000000000008 "2016-11-30 19:20:00",192,192,222.88000000000008 "2016-11-30 19:22:00",192,192,222.88000000000008 "2016-11-30 19:24:00",192,192,222.88000000000008 "2016-11-30 19:26:00",192,192,222.88000000000008

On Wed, Nov 30, 2016 at 7:30 PM, Nathan Johnson superdj4life@gmail.com wrote:

Here you go sorry

On Wed, Nov 30, 2016 at 7:29 PM, Nathan Johnson superdj4life@gmail.com wrote:

Sure thing, That would make sense but No, the values stay high on every reading. CSV export attached as requested.

On Wed, Nov 30, 2016 at 7:22 PM, Kyle Gabriel notifications@github.com wrote:

Also, do these values change when the sensor is heated or cooled and does it change proportionally?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kizniche/Mycodo/issues/158#issuecomment-264060516, or mute the thread https://github.com/notifications/unsubscribe-auth/AXJmlehwN0oUlIS86X6X5m1yuIe4pcxFks5rDi97gaJpZM4LA6eE .

kizniche commented 7 years ago

Well, I don't have a DHT11 to test, but if you're willing to let me run some test scripts on your Pi connected to your DHT11, I could probably find the issue pretty quick. If so, you can contact me privately at http://kylegabriel.com/contact

etiology commented 7 years ago

I assume this is unrelated to the behavior but I wanted to mention something strange about how the setup() method is being used. In this vesion of the DHT11 class the self.setup() method is called each time a reading is taken where the pigpio example only calls it in the ___init()___. The callbacks only need to be registered once.

kizniche commented 7 years ago

I noticed that also, which is odd because I did put setup() in __init()__ for the DHT22 class, which is very similar to the DHT11's.

zsole2 commented 7 years ago

It looks to me that the sensor may be broken. I had similar issues with several DHT sensors. After a 4-6 months (albeit I use them above 90% RH), they started to give this kind of steady readings (the normal value where they stopped, though), and either returned to normal measurements after reboot, or not, but only foe a few more hours. Heating, drying, other reconditioning attempts also have failed. So finally I have changed my sensors to HTU21D, and t.hey work no for quite OK. Although time will only tell if they can keep up..

superdj4life commented 7 years ago

thanks for all the input. i'll be available tomorrow if you'd like to remote in and run some test with the sensor. anytime after 10am MDT will work for me

On Wed, Nov 30, 2016 at 10:52 PM, zsole2 notifications@github.com wrote:

It looks to me that the sensor may be broken. I had similar issues with several DHT sensors. After a 4-6 months (albeit I use them above 90% RH), they started to give this kind of steady readings (the normal value where they stopped, though), and either returned to normal measurements after reboot, or not, but only foe a few more hours. Heating, drying, other reconditioning attempts also have failed. So finally I have changed my sensors to HTU21D, and t.hey work no for quite OK. Although time will only tell if they can keep up..

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kizniche/Mycodo/issues/158#issuecomment-264085287, or mute the thread https://github.com/notifications/unsubscribe-auth/AXJmldlSxw09eMk53tWi54z-Zb7n_T-Iks5rDmCugaJpZM4LA6eE .

etiology commented 7 years ago

I have the DHT11 sensor up and running using this library and I'm getting reasonable numbers.

@superdj4life I think you might be right in suspecting the hardware. Still it would be nice to understand the issue so I'll keep looking into it. It would be great if we could log suspected hardware errors.

kizniche commented 7 years ago

@superdj4life, did you receive readings from the script following Mycodo? That is, did you run the tests back-to-back, and if so, were these the results you wrote above? I'd be curious to see if you get erroneous readings from the Mycodo class but not from the other class, if you run them near the same time.

I'm going to be busy the next few days, so I probably won't be able to ssh in until the weekend (maybe). But, if you can verify that after running each back-to-back and Mycodo still returns erroneous readings, then I think an investigation is due. However, as @zsole2 pointed out, it may be giving temporary good readings and then start to become erroneous after a short while.

kizniche commented 7 years ago

It would be great if we could log suspected hardware errors.

@etiology, that's a good idea. We could integrate the range of the sensor into the class (just one idea) to know when readings are not as expected.

etiology commented 7 years ago

Ok I was able to produce an error with this. Here is where I see the issue

                # is checksum ok ?
                if not (total & 255) == self.checksum:
                    raise Exception

Do you see that raise Exception part. First is should be way more verbose about the exception by passing a message but this crashes the sensor thread. Previously I ran the sensor all night without any issues but here is what I got today when running the sensor for 30 seconds using this setup in a console:

from sensors.dht11 import DHT11Sensor
import pigpio
from time import sleep

pi = pigpio.pi()
sensor = DHT11Sensor(pi, 4)  # plugged into GPIO pin #4

for resp in sensor:
    """" run sensor readings in a loop """
    print("Temperature: {}".format(resp['temperature']))
    print("humidity: {}".format(resp['humidity']))
    sleep(5)

The output:

Temperature: 16.0
humidity: 50.0
Exception in thread Thread-279:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/pigpio.py", line 1129, in run
    cb.func(cb.gpio, newLevel, tick)
  File "sensors/dht11.py", line 165, in either_edge_callback
    handler(tick, diff)
  File "sensors/dht11.py", line 186, in _edge_rise
    raise Exception
Exception

Temperature: 192.0
humidity: 195.0
Temperature: 192.0
humidity: 195.0
Temperature: 192.0
humidity: 195.0
Temperature: 192.0
humidity: 195.0

So the checksum fails, throws a blank exception, the thread crashes and I suppose some setup values are now ganked. I could see handling this in the DHT11Sensor class by logging the issue and reinitializing the sensor or by handling it in the thread.

Anyone have any thoughts on what might be the best way to handle this? I think as long as we log this error in the DTH11Sensor class we aren't just making the disappear (which is good), and we might be able to save the system from responding to bad sensor conditions by running the setup again.

Perhaps we should think of using a Context Manager for the sensor's setup method. Since upon reinitializing the sensor under an error condition would require cleaning up the connection by dropping pins and such, and then reattaching the sensor.

superdj4life commented 7 years ago

I was able to recreate the output using your code, thanks. I think the hijacked value is coming from lines 171-172: if diff >= 200: # Bad bit? self.checksum = 256 # Force bad checksum

As far as i can tell I think this confirms my sensor is bad, i'll be replacing it tonight. Thanks for the quick responses from everyone. even though the issue was ultimately on my end it seems to have started some new conversations.

etiology commented 7 years ago

That's great news in a way. It opens the possibility of having a failing DHT11 sensor being detected and logged.

Thinking of the implementation details of such a behavior I still like the idea of a context manager for the sensor. Still needs further reflection though. It might be a behavior that all sensors and relays should do.

superdj4life commented 7 years ago

I have the new sensor. if it helps it's the OSEPP temp/humidity sensor, and it's producing the same results.

pi@nathonpi:~/Mycodo/mycodo $ sudo python testdht11.py Temperature: 24.0 humidity: 30.0 Temperature: 24.0 humidity: 30.0 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pigpio.py", line 1129, in run cb.func(cb.gpio, newLevel, tick) File "/home/pi/Mycodo/mycodo/sensors/dht11.py", line 162, in either_edge_callback handler(tick, diff) File "/home/pi/Mycodo/mycodo/sensors/dht11.py", line 183, in _edge_rise raise Exception Exception Temperature: 0.0 humidity: 192.0 Temperature: 0.0 humidity: 192.0 Temperature: 0.0 humidity: 192.0 Temperature: 0.0 humidity: 192.0 Temperature: 0.0 humidity: 192.0 ^CTraceback (most recent call last): File "testdht11.py", line 12, in sleep(5) KeyboardInterrupt

superdj4life commented 7 years ago

I ran the code example here the output was normal for a moment, then the same error, but then the sensor seems to be reset and readings continue as normal

temperature: 22 humidity: 27 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/local/lib/python2.7/dist-packages/pigpio.py", line 1129, in run cb.func(cb.gpio, newLevel, tick) File "testpigpio.py", line 68, in either_edge_callback handler(tick, diff) File "testpigpio.py", line 90, in _edge_RISE raise TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

temperature: 22 humidity: 31 temperature: 22 humidity: 31 temperature: 22 humidity: 31 temperature: 22 humidity: 31

kizniche commented 7 years ago

So the difference between the working example and ours is theirs uses merely raise where ours uses raise Exception?

If I had the sensor, I would test whether there is merely momentary bad data received (which I believe is the case, as this is checking the checksum), or if there was an unrecoverable error in communication. How the other sensors handle a recoverable error is simply throw out the measurement and wait until the next measurement period to try again. I suspect if we do this, we won't have to resetup any pins/connections.

etiology commented 7 years ago

I noticed that ours used atexit to register the close() method. Close never gets called. At least I don't see it. Again there is this setup-run-destroy behavior in get_measurement that isn't working.

From what I can tell the thread runs, maybe gets a good measurement, crashes, and only gets cleared and setup again on the next thread setup.

etiology commented 7 years ago

So context manager seems like the right move here. The atexit function doesn't work in the was it is used. It doesn't get called when the run method exits scope. I think that our use of this code in a threaded run loop is what is really different between the example code and our class. The loop means it doesn't exit in a way that calls close. Close removes the callback but without it being cleared the data gets malformed.

kizniche commented 7 years ago

I'm trying to figure out where that atexit came from. I can't seem to find the code I originally used to create the sensor class. It seems very close to the example code we're looking at but with slight changes, like with atexit.

kizniche commented 7 years ago

I don't have the sensor to test this, but I came up with a possible solution using a context manager. Change get_measurements() to this:

    def get_measurement(self):
        """ Gets the humidity and temperature """
        if self.power is not None:
            print("Turning on sensor at GPIO {}...".format(self.gpio))
            self.pi.write(self.power, 1)  # Switch sensor on.
            time.sleep(2)
        with self.dht11_read():
            self.pi.write(self.gpio, pigpio.LOW)
            time.sleep(0.017)  # 17 ms
            self.pi.set_mode(self.gpio, pigpio.INPUT)
            self.pi.set_watchdog(self.gpio, 200)
            time.sleep(0.2)
            self._dew_point = dewpoint(self._temperature, self._humidity)

and add this import and function to the DHT11Sensor class:

from contextlib import contextmanager

    @contextmanager
    def dht11_read(self):
        self.setup()
        yield
        self.close()

kizniche commented 7 years ago

Perhaps these variables in init should be reset in setup():

self.high_tick = 0
self.bit = 40
self.either_edge_cb = None

etiology commented 7 years ago

If I get a chance tonight or tomorrow I'll test it for you.

kizniche commented 7 years ago

For some reason the context manager wasn't working in my tests. In the meantime, I'm using try/except/finally in the DHTx modules:

        try:
            if self.power is not None:
                logger.debug("Turning on sensor at GPIO {}...".format(self.gpio))
                self.pi.write(self.power, 1)  # Switch sensor on.
                time.sleep(2)
            self.setup()
            self.pi.write(self.gpio, pigpio.LOW)
            time.sleep(0.017)  # 17 ms
            self.pi.set_mode(self.gpio, pigpio.INPUT)
            self.pi.set_watchdog(self.gpio, 200)
            time.sleep(0.2)
            self._dew_point = dewpoint(self._temperature, self._humidity)
        except Exception as e:
            logger.error("{cls} raised an exception when taking a reading: "
                         "{err}".format(cls=type(self).__name__, err=e))
        finally:
            self.close()

kizniche / Mycodo