search

esphome / feature-requests

ESPHome Feature Request Tracker
https://esphome.io/
413 stars 26 forks source link

Create Native API Server #46

Open OttoWinter opened 5 years ago

OttoWinter commented 5 years ago

Problem with current native API is that the ESP is the server. That was an intentional decision that makes setup much easier (and so the default will stay that way).

However, for things like deep sleep the api needs to be turned around: A central server (kind of like MQTT broker) that all ESPs connect to. Current plan is to incorporate this new server directly into HA.

brandond commented 5 years ago

With both client and server support, will you add support for direct calls between nodes?

OttoWinter commented 5 years ago

@brandond At a later stage, but yes.

andreasfelder commented 5 years ago

Would it be possible to use the built-in API that home assistant already has? You could call it via a rest call and make state changes in it very easily. Authentication info would need to be provided through the config. Might be easier vs building your own server.

OttoWinter commented 5 years ago

No, that is not real time and a hacky solution. I want to build a solution that will last, not one that will have to be replaced in the next release. Besides, most of the API code more or less exists already, the "server" we're speaking about here would not even have to understand what's inside the messages it is sending/receiving, it just needs to read them from ESP and relay them to subscribed clients like HA

pdanilew commented 5 years ago

I'm trying to create battery based solution like in #174. Any ideas to push this forward?

freekode commented 4 years ago

Is there any ideas how it should looks like? I would like to implement it

tjschiffer commented 4 years ago

While I understand the main use case here is for deep sleep, another benefit would be faster time to HA frontend control for nodes that are powered on. For example, a light bulb that is powered off after a while and then powered back on takes some time to be picked up be HA (my understanding is that this is due to the exponential back off with the current API connection). For now I'll be moving to MQTT but I'm looking forward to this feature.

Lumpusz commented 4 years ago

Another consideration is security. If ESP is not the server anymore, but the client, it has to initiate the connection to the server, which means the firewall has to allow the subnet, which is used by IOT devices to initiate new connections to the main lan. This loosens up security a bit.

ski7777 commented 4 years ago

I think the best solution would be to have a home assistant integration that listens on a port and all esphome nodes can connect to it. Any transport layer security problems should be part of #133 and should not prevent this development. As far as I know it should be possible to pre-establish the socket connection and the use the existing gRPC inside. So the existing protocol can still be used. Any proposals to this idea?

Raphael

ski7777 commented 4 years ago

What I forgot to say: There should be some kind of an authentication of the clients. Therefore we could either use the proposed TLS-PSK from #133 or as long as this is non-existent one could use a key that is stored in HASS and esphome and when opening the connection this key is checked. It doesn´t sound ideal but without TLS there wont be a goodsolution

glmnet commented 4 years ago

@ski7777 Otto already has some ideas of how he's going to do it, however he is not doing it!

What makes this complicate for me is:

IMHO this api_server thing makes sense only for deep_sleep (or cold boots after a serial or OTA flash) so I'd go with a mechanism to ping ESPPlugin to connect to ESPNode, this should make the connection quicker, not as quick as running the comms in the connection initiated by the ESP, but much simple implementation, and if this fails for some reason all still works as usual.

A "server" on the ESPPlugin is also desired for OTA upgrades to sleeping nodes

kvn1351 commented 4 years ago

@ski7777 Otto already has some ideas of how he's going to do it, however he is not doing it!

@glmnet What do you mean with he's not doing it? Did the idea get dropped?

ski7777 commented 4 years ago

My ideas are just a proposal. I think we should have a full concept first and then start thinking about the implementation. Everything else will cause frustration to the developers who's pull requests get rejected

glmnet commented 4 years ago

@ski7777 Otto already has some ideas of how he's going to do it, however he is not doing it!

@glmnet What do you mean with he's not doing it? Did the idea get dropped?

He’s idle, fr is ok.

mcondren commented 3 years ago

Any progress on this?

davidschlegel commented 3 years ago

I would also be very happy to hear some news about this feature, as I think it is very crucial for low energy setups.

josvink66 commented 2 years ago

Hi, is there any new information about this?

eliasmainetti commented 2 years ago

Any news ?

GamerClassN7 commented 2 years ago

Any new seruarding this Feature request or any Workerounds ?

elupus commented 2 years ago

I think a simpler solution here would be a wakeup notification from the device to HA. Could be a simple http rest command that signal i'm awake, please connect to me. That way there is no major design changes required and we can quickly react to a woken up device.

I just relalize this is exactly what @glmnet proposed too. So what would be needed for this is to add a endpoint to the esphome component i HA that can trigger a connection to device.

elupus commented 2 years ago

Actually.. this ping function is likely already in place. The esp device will do a mdns broadcast on startup i think. So home assistant just need to react to that.

Chupaka commented 2 years ago

Well, I'm interested in "calling home" to allow devices to connect to HA from behind NAT (e.g. on cellular networks).

GamerClassN7 commented 2 years ago

Actually.. this ping function is likely already in place. The esp device will do a mdns broadcast on startup i think. So home assistant just need to react to that.

@elupus yeah but that is all home assistant still list all valuses as unavalilible sometimes due to deep sleep. so i am afrait i just dont think this is fully implemented partialy at best :( or i am dummes person and i dont know tohandle deep sleep with native api not with mqtt.

elupus commented 2 years ago

Well, i think deep sleep behaviour should be improved for native api. When device goes to deep sleep, sensor entities aught not go unavailable until the deep sleep period ends plus some margin. They should just retain last reported value.

Then i think config entities should ve added, that remain available and get synced to the device once it boots.

Nine of above us currently supported. But would make it much easier to make use of deep sleep.

LouDnl commented 2 years ago

Any status update or roadmap for this feature?

glmnet commented 2 years ago

Actually.. this ping function is likely already in place. The esp device will do a mdns broadcast on startup i think. So home assistant just need to react to that.

This was implemented some time ago, however as noted by other people it's not very reliable as mdns is involved.

ljan commented 2 years ago

I stumbled over this after a few days wondering why my deep sleep sensors do not report their values within 15s on time. This should be documented as it makes deep sleep verry inefficient.

AndreKR commented 1 year ago

It would also mean that you could set a hostname for the server in the configuration and then the nodes can be anywhere, on the internet, via mobile, anything.

I was quite surprised to see that ESPHome relies on broadcasts, nodes with open ports and all nodes being on the same subnet. Not really appropriate anymore IMO.

However, the even better way is probably to implement MQTT over TLS and to teach the dashboard to do OTA over MQTT. That would instantly open up the world of MQTT bridges, clustering and a whole bunch of tooling.

EternityForest commented 1 year ago

@ski7777 Otto already has some ideas of how he's going to do it, however he is not doing it!

What makes this complicate for me is:

  • The current model should still work, discovery and connections initiated from ESPPlugin
  • ESPNode is a server, and thus more than one ESPPlugin can connect to it simultaneously and control it, this makes it hard to say to which server the ESPNode should connect when it's in client mode, it can connect to a fixed one, the last one or a list.

IMHO this api_server thing makes sense only for deep_sleep (or cold boots after a serial or OTA flash) so I'd go with a mechanism to ping ESPPlugin to connect to ESPNode, this should make the connection quicker, not as quick as running the comms in the connection initiated by the ESP, but much simple implementation, and if this fails for some reason all still works as usual.

A "server" on the ESPPlugin is also desired for OTA upgrades to sleeping nodes

I kind of agree here. I'm not really a fan of having both client and server modes unless it's really needed, as per the "Narrow waist" principle, it's really nice to have fewer possible configurations to avoid tools getting fragmented. Protocols with tons of optional modes usually wind up only using one of them anyway.

A simple reverse connection protocol would be pretty easy. Not sure if the APIs allow it, but you could store the last 2 or 3 client IP addresses in RTC ram, and unicast them an MDNS packet to make things more reliable.

Or just create a new UDP protocol that could carry the same data as the native API, after the crypto is already set up with the TCP version. There are already several components like the camera and microphone that could benefit from unreliable transport, so it could be used for more than just on/off notifications.

lucius-the commented 1 year ago

Any news/progress with this ?

My use case is: in some locations I would like to have an esphome device (ONLY). Home Assistant installation would be hosted in a cloud VM. The network, the esphome device sits in, only has Internet. In such case, esphome node should connect TO home assistant, not the other way around.

I'm aware "home assistant" means "using at home", with everything working offline, etc. But the system (hass + esphome especially) is so usable for so many things, I think implementing an option to provide esphome devices sitting somewhere where there's no homeassistant, it would be nice if they could connect to hass over the Internet. MQTT client in esphome can't do TLS (also can't do client certs), so that's not an option.

Native API client could also provide 99% of everything needed to enable node-to-node communication as well, which is another great thing, for projects with no need for a central hass.

I really see a native API client as a great upgrade, that would enable a whole new set of use cases and projects. I see it as an option, to add alongside existing native API server for projects that need it (and not to replace the current native API server, as others have mentioned, because that also has it's own positive use cases).

I am a developer but unfortunately I have near-zero knowledge of C... so can't help much with implementation. But I'd like to try and help if there's any way I could, perhaps with advice or design, I do have a bit of experience (20+ years) and I'm good in networking. If someone wants my help, let me know. I'd really love to see this feature move forward.

(edit for typpos)

lucius-the commented 1 year ago

I mean, please have gone so far to try and implement a VPN client inside an ESP ! Talk about overkill... A use case for that is probably the feature we are talking about here, that is still missing in esphome.

More info on wireguard VPN in esphome: https://github.com/esphome/feature-requests/issues/1444

fakuivan commented 10 months ago

So far the reasons I've seen for this feature are the following:

Another approach that would satisfy these points would be to allow some nodes to act as relays to home assistant. The server stuff is already integrated into the nodes, home assistant would coordinate some sort of key for the two nodes to talk to each other, and then use that to pass messages to HA whenever the deep sleep node goes online.

An even simpler implementation would just notify HA that the node is up trough the always up node, no relaying. But I think most importantly this would allow nodes to talk to each other if HA goes down and achieve something like ZigBee binding, so that your light switches still work as you update or do maintenance on the HA server.