dtxx: Implement driver for DHT-11 / DHT-22

[skyflyer]

Hello!

Are there any plans on supporting DHT-22 sensors? The project looks very nice (no C dependency).

[maruel]

The devices are fairly trivial, shouldn't be too hard to implement a driver. Want to try?

[skyflyer]

Challenge accepted :)

I played with the sensor a bit, and I see from the datasheet that reading is very timing-sensitive (20 us).

So I set up an input with edge detection on both sides (d.pin.In(gpio.PullUp, gpio.BothEdges)) and I'm reading the value in a for loop with a preallocated slice with measurement result objects:

start := time.Now()
for i := range res {
    if !d.pin.WaitForEdge(1 * time.Millisecond) {
        res[i].level = d.pin.Read()
        res[i].nsSinceStart = time.Since(start).Nanoseconds()
        res = res[:i+1]
        break
    }
    res[i].level = d.pin.Read()
    res[i].nsSinceStart = time.Since(start).Nanoseconds()
}

Based on the output that I'm getting, I'm guessing the code does not execute fast enough (?). I'm measuring elapsed time with time.Since(start).Nanoseconds() which might not be optimal:

t - time in us since start
s - signal
d - signal duration
---
...
t=   479  s=Low    d= 95
t=   575  s=Low    d= 78
t=   653  s=Low    d= 79
t=   733  s=Low    d= 79
t=   813  s=Low    d= 80
t=   893  s=Low    d= 80
t=   974  s=Low    d= 79
t=  1053  s=Low    d= 81
t=  1134  s=Low    d= 81
t=  1216  s=High   d= 72
...

Sometimes, I get double amount of readings, with values of 11 microseconds in between (running the same program multiple times).

Since I'm getting a lot of consecutive readings with the same level (Low), I suppose there are High levels in between which happen too fast to be actually read by the code. According to the datasheet, the length of high-level for bit value 0 lasts only 26-28us. and the code apparently spends too much time in the processing of the reading.

I could "assume" that the level was high in between, but that seems fragile.

Calling C code from go doesn't seem like the right approach for this library (pure go, if I understand this correctly), and that approach already exists -- the authors mention:

Originally attempt was made to write whole library in Golang, but during debugging it was found that Garbage Collector (GC) "stop the world" issue in early version of Golang sometimes freeze library in the middle of sensor reading process, which lead to unpredictable mistakes when some signals from sensor are missing. Starting from Go 1.5 version GC behaviour was improved significantly, but original design left as is since it has been tested and works reliably in most cases.

So I needed an excuse to fire up an oscilloscope, to verify the timings and they seem inline with what the datasheet says:

@maruel, what kind of approach do you suggest?

[maruel]

If you want more reliably timing measurements, https://periph.io/x/periph/host/bcm283x#Nanospin is good but it requires the app to run as root.

I think the only way to make it work somewhat reliably is to use a CPU-less communication mechanism.

I started designing structures to allow this at https://periph.io/x/periph/conn/gpio/gpiostream but one known limitation is that it is unidirectional; there's no way to switch from input to outputs, so it'll need an higher level similar to https://periph.io/x/periph/conn/spi#Packet.

There's two potential implementation. The first is DMA base communication, there's one somewhat implemented at https://periph.io/x/periph/host/bcm283x#Pin.StreamOut but it is not reliable yet. The second will be via a ft232h https://periph.io/x/extra/experimental/devices/ftdi (I'll rename the package to ftd2xx in the coming days) which supports paced data including input/output. The FTDI proprietary protocol is named MPSSE. The nice thing about this is that it'll just work on anything over USB.

[skyflyer]

I just tested bcm283x.ReadTime() in a loop without doing anything else expect WaitForEdge, but it still gives inconsistent results. Sometimes, the results are really promising, but it is very random (sometimes it detects 24 edges, sometimes 54, 51, 50, 48, etc).

Tested with StreamIn, but I'm not sure I'm doing it right and that the gpiostream would be the right approach, since the signal length differs between 0 and 1 bit values (as you can see from the oscilloscope screenshot above).

I'm not at all familiar with DMA access, so I can't comment on that. I suppose interrupts wouldn't work as well? Or would they (and how to enable them)? I'm thinking of collecting interrupts with timestamps if that would be possible. Otherwise, I guess I'll just have to resort to C (interestingly enough, I just tested the go-dht library and it is not getting the correct readings every time -- whereas Adafruit's python library (with C bindings) reads them every time (it calculates average pulse length and then uses that to detect 1s and 0s).

[maruel]

My thinking is to oversample just enough to reconstruct the signal at a ~25μs resolution. It won't be doable right now because of the output->input switch but this is something I want to support for v3.

[maruel]

The spec calls for max 5ms transmission duration so a 5μs sample rate is only 1000 samples, which should be doable.

[skyflyer]

It won't be doable right now because of the output->input switch

Why isn't it doable? The output is at the beginning, then it is switched to input for the rest of the reading. It is not alternating between input and output?

I'll try to setup 5us Resolution on the OutputStream and see what I can come up with. A question though: it is returning bytes - I suppose they represents bits or am I mistaken?

[skyflyer]

FWIF, I started getting the following errors:

return errors.New("bcm283x-dma: no channel available")

Are the channels left in a busy state?

After reboot, the StreamIn can read a couple of times (apx. 5 times), and then returns to this error. dmesg output shows only one DMA channel, and that one is supposed to be used by the display?

[    0.070149] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[    0.150852] bcm2835-dma 3f007000.dma: DMA legacy API manager at bb80f000, dmachans=0x1
[    0.252437] BCM2708FB: allocated DMA memory fad10000
[    0.252490] BCM2708FB: allocated DMA channel 0 @ bb80f000
[    0.769685] Using Buffer DMA mode
[    0.786622] WARN::dwc_otg_hcd_init:1032: FIQ DMA bounce buffers: virt = 0xbad04000 dma = 0xfad04000 len=9024
[    1.051234] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[    1.060552] mmc-bcm2835 3f300000.mmc: DMA channel allocated
[    4.813560] graphics fb1: fb_ili9486 frame buffer, 480x320, 300 KiB video memory, 32 KiB DMA buffer memory, fps=33, spi0.0 at 16 MHz

[maruel]

I think this is related to #226, something isn't right in the bcm283x-dma driver. Make sure you Halt() the pin before terminating the process.

I fear that there's some significant work that needs to be happening before it's possible to support a device like the DHT-xx in periph due to lack of paced I/O. I've been thinking about (ab)using the PCM pin, which would work great but I forget what was the state with this.

[skyflyer]

@maruel,

since I don't need this support urgently, I'll just close the issue. Perhaps I'll try to play with it in my free time and if something useful comes out of it, I'll post it here and we can see if it is pull-request-worthy.

I'll just purchase DS18B20 and continue with that (or use an ffi interop for dht-xx).

I appreciate your input, I've definitely learned something new! Thanks!

[maruel]

I'll keep the issue open since there's valuable insight in here.

[skyflyer]

FWIW, I tested with pin.Halt() as well, and I'm still getting bcm283x-gpio (GPIO21): bcm283x-dma: no channel available error. Even though the resolution is 5us, I'm getting only 3 bits for the "high" signal, but that's close enough. I'm afraid the no channel available is a showstopper for this to work, so I'll try with the DS18B20.

[maruel]

One quick question: what distro did you use? I only test on Raspbian Stretch Lite; e.g. without GUI. This may be important.

[skyflyer]

I am using the official Raspbian with GUI enabled - the final project will require the GUI.

[maruel]

Only let's continue the diagnostics on #226.

[c1tt1]

Question! Was this driver implemented??

[maruel]

No at the moment.

[c1tt1]

Is there a plan to do so or has someone started working on it?

[maruel]

Not as far as I know.

[c1tt1]

@maruel I came across this go-dht package it seems a fit implementation for this issue.

[maruel]

It is.

[c1tt1]

so can it be merged? Also the example could be good for the main website examples.

[maruel]

Oh sure if @MichaelS11 wants to! I'd start with adding to experimental, then make a page on the website, then upgrade to stable.

[MichaelS11]

Welcome to take/have as much of https://github.com/MichaelS11/go-dht as you want.

Could not get WaitForEdge to work, note this this part: https://github.com/MichaelS11/go-dht/blob/be44b9ee7fec8f81d57dea89c17d26961183266e/dhtNotWindows.go#L75-L90

[c1tt1]

@maruel and @MichaelS11 if you're both happy with that I could submit a PR with it.

[maruel]

Sadly WaitForEdge is at the mercy of the Linux kernel, and when calling into it it may take the occasion to swap the process out, which may mean several ms of delay. That's why you couldn't make it reliable. On the other hand on a RPi, Read() and Out() talks to the gpio bypassing the kernel, so the odds of being preempted is lower (but not nil).

@kaskerd thanks for volunteering!

[MichaelS11]

@maruel Yeah, found that Read sometimes missed a level change, but with all the checks, it almost always is caught when it does.

@kaskerd Sounds great, thank you! :)

Side note, welcome to these as well: https://github.com/MichaelS11/go-ads https://github.com/MichaelS11/go-hx711

[schlamar]

There is a device tree overlay for dht11/22:

Name:   dht11
Info:   Overlay for the DHT11/DHT21/DHT22 humidity/temperature sensors
        Also sometimes found with the part number(s) AM230x.
Load:   dtoverlay=dht11,<param>=<val>
Params: gpiopin                 GPIO connected to the sensor's DATA output.
(default 4)

I guess this would be the simplest and most reliable approach getting data from a DHT sensor into golang without using cgo.

Here is the driver (which is using Linux's IIO): https://github.com/raspberrypi/linux/blob/e2d2941326922b63d722ebc46520c3a2287b675f/drivers/iio/humidity/dht11.c

Here are some instructions on how to enable the overlay: https://community.openhab.org/t/reading-a-dht11-22-with-overlay-and-send-result-via-rest-api/44827

BTW, @MichaelS11's solution didn't reliably work for me on a PiZero. I was constantly getting timing related errors like Read error: missing some readings - high level duration too long: 102µs (probably due to kernel interrupts or context switches).

[MichaelS11]

@schlamar Was curious, did you use dht.ReadRetry(11) ?

[schlamar]

Yes, I tried your ReadRetry example and it didn't work for multiple runs. I guess there is too much preemption on the single core CPU.

Working implementations like Adafruit_Python_DHT use the sched_set_scheduler syscall to give the process realtime priority (https://github.com/adafruit/Adafruit_Python_DHT/blob/5ead51d/source/common_dht_read.c#L57).

[MichaelS11]

@schlamar What Go version? What Linux version are you running? (uname -a) I tested on Linux 4.14.34-v7+

Would you like to test setting the priority? Would be added to the two places where debug.SetGCPercent is called in: https://github.com/MichaelS11/go-dht/blob/master/dhtNotWindows.go

import "golang.org/x/sys/unix"

err = unix.Setpriority(unix.PRIO_PROCESS, 0, -19)

err = unix.Setpriority(unix.PRIO_PROCESS, 0, 0)

Notes:

Setpriority docs: http://man7.org/linux/man-pages/man2/getpriority.2.html

unix docs: https://godoc.org/golang.org/x/sys/unix

[MichaelS11]

@maruel

Trying to get a little more performance out of read. Do you think seekRead can be changed from Seek and then Read to ReadAt?

https://github.com/google/periph/blob/589c98b7c4542bc2f651f6dcee83546ef70b8ec1/host/sysfs/sysfs.go#L49-L52

Was looking at the lower level Go os code and looks like it would at least remove one checkValid.

Seek: https://github.com/golang/go/blob/131eb8fbf80fd8b51ae8b5c5220d566582a41e71/src/os/file.go#L204-L214

Read: https://github.com/golang/go/blob/131eb8fbf80fd8b51ae8b5c5220d566582a41e71/src/os/file.go#L109-L113

ReadAt: https://github.com/golang/go/blob/131eb8fbf80fd8b51ae8b5c5220d566582a41e71/src/os/file.go#L121-L139

The DHT-x changes levels so fast every little optimization helps.

[MichaelS11]

Or if anyone else can look at that change? If it is a good change, anyone else can do a PR?

[schlamar]

I just found out that there is a new GPIO character device and the sysfs interface is deprecated. Maybe this is worth investigating.

A lot of context including useful links can be found here: https://github.com/adafruit/adafruit-beaglebone-io-python/issues/157

An example using /dev/gpiochipX with Go can be found here: https://github.com/stapelberg/hmgo/blob/master/internal/gpio/reset.go

There is the GPIO_GET_LINEEVENT_IOCTL request to get event data from GPIO including timestamps. Update: Here is a C example with this ioctl call: https://github.com/torvalds/linux/blob/master/tools/gpio/gpio-event-mon.c

[maruel]

@schlamar the new GPIO interface is tracked as #371. I haven't made traction on it since I was #398 to be in first, and I dragged my feet a bit.

As for the issues here, I wish I could come up with a DMA based solution, but that will have to wait for v4 sadly. :/ I'm fine with an alternative in the meantime, please submit a PR. :)

[MichaelS11]

I would be unable to do a PR because of the contributing requirements. Would someone else be able to do it?

[maruel]

Ported to https://github.com/periph/devices/issues/60.