GPIO output feature - Githubissues

marc1de commented 2 years ago

Did you ever think about adding GPIO-output capabilities?

Thought, this might be a great feature to:

create individual status-lights/displays with LED's (eg. DHW is on, Heating active, etc....)
turn on relays to control valves/pumps/motors (eg. to control Floor-Heating Valves etc.)
or just to replace some simple actors in that place (maybe switching lights or turning on an power socket)

MichaelDvP commented 2 years ago

Actually there is a pin command to set a output pin remotely. If you are using relais, there are some modules available that have much too less insulation distance for 230V, double check before using. But there is also project #127, adding some common scenarios as logic routines.

bbqkees commented 2 years ago

I have some more information in my wiki: https://bbqkees-electronics.nl/wiki/gateway/using-gateway-gpio.html

I still strongly recommend NOT using the Gateway GPIO but just get another ESP board with Tasmota etc for relay functions. The Gateway is directly connected to the EMS bus which is part of the critical heating system in your house and by adding a relay and switching 230V with it is a safety risk.

For instance this Wemos relay shield is the most dangerous thing you can use: https://www.wemos.cc/en/latest/d1_mini_shield/relay.html

The GPIO line is about 1mm away from main voltage. If it makes contact, you WILL blow up the Gateway and the EMS bus.

marc1de commented 2 years ago

@MichaelDvP thanks for your hints but actualy I was thinking over a more "user-friendly-implementation", so that the GPIO's could easily be "visualized or changed" in the WebUI or maybe in HA or IOBroker. (as you already did with eg: "Boiler heating activated" or the"Analog Sensor")

For example:

I was able to build an automation in HA that lights the LED on GPIO12 of my raspberry if "Boiler heating" is active - but I thought, it could be a nice feature to have that on the ems-esp directly (cause it's in a better place for that).

(btw: your idea in #324:

Adding a generic aux_function with text input of a simple logic function, something like " " e.g. "boiler/wwtemp > 50"

sounds very interesting - but I think, this would requires some major changes to ems-esp so that I suppose thats not very likely to see that getting implemented)

Meanwhile I tried to experiment with the pin command, but unluckily I failed with that. I tried to set the GPIO's with http-commands via Browser.....but obviosly I was unable to find the right syntax: The manual said:

so I tried different variations like: http://ems-esp/api/system/pin{"id":36, "value":0} or http://ems-esp/api/system/pin&id=36&value=0

but still after several hours I didn't succeed with that. (security authentication was disabled in the settings while I tested).

I'd be very happy if someone could give me an full example for the correct command (preferably with and without the use of the authentication token)

@MichaelDvP @bbqkees: thanks for your warnings and safety hints. I fully agree, that someone, who is not certified to work with higher voltages should never experiment with such things (and especially not with that sort of dangerous relay-shields without optocouplers and proper insulation or reliable ems-protection). (btw: that wemos-relay-shield you've linked is indeed a really shocking example of dangerous hardware-crap that someone should never use)

bbqkees commented 2 years ago

With the latest API changes all commands that can set something are supposed to be done via a HTTP POST or PUT. The pin command is such a POST command, not a GET request you can just type into the URL.

The Wemos relay shield is 'shocking' figuratively but also in real life. But if you don't know anything about electronics, you would just buy it and connect it.

MichaelDvP commented 2 years ago

@marc1de The v3 api is POST with url http://ems-esp/api/system/pin and payload {"id":36, "value":0} or http://ems-esp/api/system and payload {"cmd":"pin","id":36, "value":0} If you want to use GET you can use the v2 syntax: http://ems-esp/api?device=system&cmd=pin&id=36&data=0 Or send from mqtt (mqtt-explorer or HA) topic ems-esp/system payload {"cmd":"pin","id":36,"value":0} As a workaround you can create a analog sensor input on pin36, then use the pin command and the pin changes to output, showing the state in web-ui. After reboot or reconfigure it's input again (or use pin command with value in, input or -1. But we can also add a analog-mode "OUTPUT".

marc1de commented 2 years ago

thanks for the attempts to help me out.....unfortunaly I must have made a stupid error, because none of the approaches that I tried, did help:

My System: EMS-ESP Version: latest development release v3.4.0b4 Device (Platform / SDK): ESP32 / v3.3.5-1-g85c43024c

LED (with 270Ohm Resistor) between Vcc and GPIO25 (so it will light up if GPIO is Low).
(double checked that this is an allowed output-pin for the ems32 and security authentication was disabled in the settings of ems-esp)

Because I still have some trouble implementing the "v3 api is POST-Solution" into HA my first attempt was this:

as far as I understood Michael, it should be still possible to use the old v2-syntax. So I first tried to type the urls "http://ems-esp/api?device=system&cmd=pin&id=25&data=0" or
"http://ems-esp/api?device=system&cmd=pin&id=25&data=1" into my browser. Browser reports: {message": "OK"} but nothing else happend!

Next I configured GPIO25 as Digital input in the web-ui, saved it and tried to enter the urls again.

Again nothing happend: LED stays off, no changes in the web-ui):

Ok, next try: started the mqtt-explorer and tried to enter the values there:

Result: no LED changes, no web-ui changes, no changes in HA....only the changing values in mqtt-explorer

Same happens when I try to use the mqtt-publish in HA or try to start the scripts that I entered in HA.

Any clues, what I am doing wrong? Maybe, it would be the best solution, if the OUTPUT-feature could be added in ems-esp......I really don't know, why your suggestions didn't work in my environment. But thanks anyway for your great support.

proddy commented 2 years ago

Isn't GPIO36 only for digital input, not output, as they don't have any push/pull resistors? worth checking...

MichaelDvP commented 2 years ago

Yes, io36 is input only, and io25 is DAC output, it needs value 255 to set it high. I've tested with io19 and it works.

marc1de commented 2 years ago

aargh, so in fact, my problems were mostly related to a wrong choice of Pins!

@proddy: you're right: Pin 36 is input only. That was a typo in my post. In fact I used Pin 25 and 26 for my test's. (not knowing, that this was a bad choice too....)

I now found some other sources that pointed out, which ports should be avoided: As far as I know now, the following Pins should be avoided on a ESP32-Dev Kit V4

GPIO: 34,35,36,39: only Inputs, no Pullups 6,7,8,9,10,11 : used for internal Flash 2,4,12,13,14,15,25,26,27: ADC2 - cannot be used when WIFI is used 2,5,18,23: used by ems-esp (=ESP32-BBQ-GatewayS32 with Dallassensor on 18) 0,1,3,12: not recommended / unexpected behavior at boot (Pin 2 is somehow special. In the sheets it has ADC2 and should therefore be avoided, but it is also used as internal LED on some ESP-Boards, so it should be safe to use that as StatusLED, just like in the BBQ-Gateway S32)

so that leaves only GPIO 16,17,19,21,22,32,33 free for safe use. (maybe this information could be added in the manual to prevent other user to make the same mistake)

after changing to Pin 19 (32 and 33 are also verified) I was able to set the values via URL (api-V2) "http://ems-esp/api?device=system&cmd=pin&id=19&data=0" and via mqtt. topic ems-esp/system payload {"cmd":"pin","id":19,"value":0}

.... and after correcting the script in HA it also worked.

but maybe I'll give it a try to check out the api-v3-Method as Michael suggested. url http://ems-esp/api/system/pin and payload {"id":19, "value":0}

Thanks @all for your great help!!!

(but still would be happy, if the OUTPUT-feature could be implemented in the code, so that the values could be set in the web-ui of ems-esp - or even better: if some of the GPIO's could be connected directly with some of the (binary-)entities in the web-ui so that they can act as StatusLEDS's without needing an other program)

eg: LED1 - internal StatusLED for EMS-ESP LED2 - Heating activated LED3 - Heating active

LEDx - Boiler/Thermostat critical Error (ok, the last one will be difficult, because there is no defined Errorstatus, so it will probably neccessary to set this status in an external Program (eg: if statuscode = xxx or xxy or xxz set LED on)

proddy commented 2 years ago

We'll mention the pins in the wiki/doc - good to point that out.

MichaelDvP commented 2 years ago

so that leaves only GPIO 16,17,19,21,22,32,33 free for safe use.

nearly all right, ADC2 is used for Wifi, but the pins can be used We have

1, 3 for serial monitor
2, 4, 5, 16, 17, 18, 33, 34, 36 used for different board types
0, 2, 4, 5, 12, 14, 15 used for strapping bootloader, can be used with care in normal operation (see boards)

i think safe to use is:

digital output: 13, 19, 21, 22, 26, 27, 33, 37, 38
digital input/counter: 13, 19, 21, 22, 25, 26, 27, 33, 35, 37, 38, 39, no pullup on 35, 39
ADC input: 13, 19, 21, 22, 25, 26, 27, 33, 35, 37, 38, 39
DAC output: 25
PWM output: 13, 19, 21, 22, 26, 27, 33, 37, 38

I think if i enhance the analog config to digital output, i can also add DAC and PWM channel 0,1,2. And counter can also measure the time between two impulses (Timer mode), or reciprocal 1/sec (Rate mode) Rename the analogsensor/counter command to analogsensor/setvalue (making system/pin obsolet) Settings in Web-UI: Inputs:

Digital in: no parameters
ADC: offset, factor, uom,
Counter: offset, factor, uom
Timer or Rate: factor, uom Outputs:
Digital_out: value 0-1
DAC: (digital out on io25): value 0-255
PWM_0-PWM_2: Frequency 1-5000 Hz, Dutycycle(value): 0-8192

proddy commented 2 years ago

that would be a great enhancement and also help out @Sunbuzz (#127)

proddy commented 2 years ago

We'll mention the pins in the wiki/doc - good to point that out.

added. https://emsesp.github.io/docs/#/Configure-firmware?id=adding-external-sensors

bbqkees commented 2 years ago

Can we add a big warning notice in the wiki and a permanent warning notice in EMS-ESP for this feature? I am not at all happy with people opening up their Gateway and adding all kinds of unknown stuff to the circuit board.

We need to keep in mind that the circuit board is also directly connected to the EMS bus of expensive boilers and very expensive heat pumps. Most users are of course not electrical engineers and thus lack thorough knowledge of electronics.

In the next revision of the S32 there will be some additional protection but I can build in as much protection as I want, there will always be better fools out there.

Aside from the risk of blowing stuff up, some electronics will also introduce noise or other disturbances into the system.

The more features we enable, the more people will use those features. We need to stay aware of the consequences.

marc1de commented 2 years ago

@MichaelDvP:

after the last update to v3.4.0b6 my scripts for changing the DigitalOutputs didn't work any more. According to your last update in the docs (commit 3d3bfe0) I tried this (and several similar varitions) in the mqtt-explorer:

but that didn't work: Could you please give me an example for the correct syntax?

Same problems occured with the browser-commands. I tried: http://ems-esp/api/analogsensor/setvalue&value=1&id=19 or http://ems-esp/api/analogsensor/setvalue?value=1&id=19

but also failed!

as a suggestion: I think, the provided information is very difficult to understand for "non-programmers". Maybe it would be easier, if you could add some more examples in the docs for this commands.

... and a second suggestion: I think the name"analogsensor" is meanwhile a little bit misleading in case of "digital In/Outputs, Counters or PWM's". Maybe "gpio" is a better term that covers all the possible cases (including the analogsensors).

MichaelDvP commented 2 years ago

You're right, mqtt does not work due to missing subscriptions. I'll add them for dallassensor and analogsensor.

For api you have to use the v2 OR v3 api, you can not combine the syntax. v2: http://ems-esp/api?device=analogsensor&cmd=setvalue&id=19&data=18.5 v3: http://ems-esp/api/analogsensor/setvalue payload: {"id":19,"value":18.5} telnet: call analogsensor setvalue 18.5 19

marc1de commented 2 years ago

Ok, meanwhile I tried to implement the ApiV3-method in HA but didn't succeed with that either:

Does anybody know, whats wrong with this syntax in configuration.yaml?

rest_command:
  emsesp_set_gpio16_on:
    url: 'http://ems-esp/api/analogsensor/setvalue/'
    method: POST
    content_type: 'application/json'
    payload: '{"id":16,"value":1}'

Whenever I try to Call the service from developer tools, I get this error: 2022-02-19 18:45:50 ERROR (MainThread) [homeassistant.components.rest_command] Client error http://ems-esp/api/analogsensor/setvalue/

or does this "Client error" indicate, that this is no HA-Problem but a Problem in ems-esp? my ems-esp-log reports:

My System: EMS-ESP Version: latest development release v3.4.0b6 HA-Core: 2022.2.3, HA.OS: 7.4

proddy commented 2 years ago

Make sure you have the bearer token with POSTs as in the example. I just tested and it works with

rest_command:
   emsesp_set_gpio39_on:
    url: http://ems-esp.local/api/analogsensor/setvalue
    method: POST
    headers:
      authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6ImFkkWluIiwiYWRtaW4iOnRydWV9.2bHpWya2C7Q12WjNUBD6_7N3RCD7CMl-EGhyQVzFdDg"
    content_type: "application/json"
    payload: '{"id":39,"value":1}'

Note., I created a Digital Out on GPIO 39 in EMS-ESP

marc1de commented 2 years ago

@proddy: Your example did the job! Thanks for checking! But after that it finaly worked, I got really curios what might have caused the trouble, so I did some more tests. Couldn't believe that it was the Bearer-Authorisation, since I tried that before and my autorisation checking in ems-esp was disabled for the tests. DigitalOutput already was defined before and I I did severals tests with different combinations of slashes and quotation marks....but never succeeded.

But now I found the reason for the fault: It was the missing ".local" in the url that caused the trouble!!!

(never thought, that this could be the problem, since that was never needed before. I can usualy reach the gateway with http://ems-esp or http://ems-esp.local)

proddy commented 2 years ago

ah great! EMS-ESP starts a multicast DNS server (mDNS) which itself uses the .local domain prefix. Unless you added an entry in your router you can always resort to .local. Also if you turn off authentication in the settings you won't need the Bearer token for POSTs. It's only there to add a little more security.

marc1de commented 2 years ago

@proddy, @MichaelDvP, thanks to your help, my DigitalOutputs now work as desired. But since the last update to v3.4.0b6 I realized some crashes of ems-esp during the night. The first time I' didn't pay much attention to this situation (thought, that it might be related to my intense testing), but again this morning my ems-esp was frozen during the night.

[Fault Description:]

ems-esp showed a red flag with "network error" and didn't respond to any commands
ping from my desktop-PC to "ems-esp" not working (timeout) [ping to ems.esp.local never works]
ems-esp did not respond to any HA-commands

After a reset of ems-esp everything worked fine again. But while investigating the situation, I noticed two remarkable things.

the situation is different from a power-down situation. To my big surprise, HA was still receiving mqtt-updates from ems-esp!!! (in the first diagramm, you can see, when I restarted the ems-esp [orange arrow] But as you can see in the second diagramm, it was still receiving continously changing values [see green curve]).

The second remarkable point was:

after the restart, the leds on my ems-esp initially showed the wrong status (well, I certainly expected this, since the were controlled by the HA-Automations - but to my very big surprise the automation-script of HA had detected the correct status directly after the restart and got triggered- but without effect to the ems-esp. In other words, HA "tried" to turn on the status-leds......but apparently the ems-esp was just not ready to execute the Api-Command at that early moment.

[Hardware description] I don't think, that the fault is related to my hardware or power or elecromagnetic interference. Before I switched over to ems-esp with an esp32 and HA, my gateway (self build, based on this project: https://www.mikrocontroller.net/topic/309075?page=single) was operational several years with iobroker. I have no relays or other critical stuff connected (only two leds with 3mA and resistor on gpio 16 and 19 -directly soldered onto the gateway). External Power source is good, stabilized and free of spikes (checked with Oszilloscop). Didn't realize any "freezing errors" with versions prior to v3.4.0b6.

-------------------------------------------------------------------------------------------- EDIT: Statusupdate 23.02.2022: updated to v3.4.0b7 yesterday evening:

Result:

no freezing of ems-esp this morning
DigitalOutput with Api V2: works
DigitalOutput with Api V3: works
DigitalOutput with mqtt: works

-------------------------------------------------------------------------------------------- EDIT 2: Statusupdate 24.02.2022:

again "freezing" of ems-esp!

it was still sending data via mqtt but no ping to ems-esp and webinterface was not reachable!

marc1de commented 2 years ago

@proddy, @MichaelDvP Update 26.02.2022: DNS-Failure

Again having the same issue again yesterday and last night, I did some experiments and finally found the reason for the "partial freezing".

I found out, that I was still able to change the digitalOutputs in this "freezing-state" via mqtt but not with the restful ApiV3-commands, After some more experiments I found, that the problem was related to DNS! After changing my scripts to the IP instead of dns-names the script worked again.

Summary: After resetting ems-esp, the following script works

but after a while the dns-entry on my HA-raspberry seems to be outdated and the script stops working. After changing the urls in my script to: url: http://192.168.xxx.xxx/api/analogsensor/setvalue

it worked again (without reset of ems-esp). while url: http://ems-esp/....... or url: http://ems-esp.local/....... did not work.

I have no clue, why the dns-entry gets outdated and I am a little bit confused, why the mqtt-command: ems-esp/analogsensor payload {"cmd":"setvalue","id":16, "value":1} still works in that state.

proddy commented 2 years ago

ok, so EMS-ESP wasn't crashing, the router was just losing the cache of the mdns entries. I've read the native MDNS can be a bit flakey on the ESP32. If this problem persists I'll look into an alternative like https://github.com/khoih-prog/MDNS_Generic

marc1de commented 2 years ago

@proddy

...so EMS-ESP wasn't crashing, the router was just losing the cache of the mdns entries.

well, or to be more correct: I think it wasn't the router but my HA-raspberry, that lost the entry (which should be perfectly ok, since the ttl will run out after some time - so I suppose it might be a problem whith the renewal of this mdns-entry).

If this problem persists... I'll keep an eye on that and will keep you informed.

I understand the mechanism of mdns (and that it only works with .local) but what confuses me, is: In my HA I also used the mqtt-command: And as you could see, I use the regular DNS with that mqtt-entries: And this script works all the time so I know, that HA is definitely able to use the regular DNS of my DSL-Router. Therefore it's a big mystery for me, that the same expression doesn't work in my ApiV3-Call where I had to use url: http://ems-esp.local/....... instead of url: http://ems-esp/.......

marc1de commented 2 years ago

@proddy The mdns problem still persists. I repeatedly checked the mdns-feature in the last days. To verify the issue I wrote a little script in HA that can be executed manually to set a gpio output.

after a restart of ems-esp the script is working (also it works after several hours)
but the next day the script is no longer functional (while the same script with the IP-adress of ems-esp still works).
Restart of HA or reboot of my raspberry doesn't help - only a restart of ems-esp makes it working again.

proddy commented 2 years ago

Best if we create a new issue for this and close this one (related to GPIOs). I looked the MDNS code again and compared to Tasmota and esphome and its similar so I'm not convinced its a s/w issue. But I'll try the same as you and check with the DNS service can resolve the ems-esp.local in a cron job.

MichaelDvP commented 2 years ago

I've tested this and it seems the mDNS is broken after a network reconnect. To reproduce simply go to network settings and click save, after reconnect the mDNS stops working. Is seems to be a solution to insert before this lline a MDNS.end(); to stop older instances.

proddy commented 2 years ago

genius! didn't think of that. @MichaelDvP feel free to push a change direct to the repo

proddy commented 2 years ago

closing this issue, as it was implemented in 3.4

emsesp / EMS-ESP32

GPIO output feature #355