Support direct output to Time-Series databases

[fake-name]

Describe the problem you have/What new integration you would like

ESPHome is super handy for the framework it provides. However, the only way to get output at the moment, as far as I can determine is either:

• Write a custom MQTT forwarding agent
• Add-Ons to Hass.io

I have an existing time-series database (I actually have both graphite and am experimenting with an influxdb server, both of which have a fair bit of data in them already).

The hass.io extension for influxdb appears to be hard-coded to run it's own internal influxdb instance in docker, and (based on cursory reading) doesn't support using existing instances. Additionally, I'd really prefer not needing the hass.io server in the first place.

Basically, it'd be nice if there was a way to have ESPHome ship sensor metrics directly to the time-series database. Shipping metrics to graphite is super lightweight (just squirt a UDP packet). InfluxDB is still straightforard, though a bit more stateful (it's a HTTP POST).

By doing this, the logging system is dramatically simplified.

Please describe your use case for this integration and alternatives you've tried:

The current options are:

• ESPHome Sensor -> MQTT Broker -> custom forwarding agent -> TS Database
• ESPHome Sensor -> Hass.io server -> extension -> Containerized (separate) TS Database

By having the sensor ship directly to the server, you get:

• ESPHome Sensor -> TS Database

So there's much less complexity, and points of failure.

Additional context

I'm already running my hass.io instance in a LXC container, so I can't use docker. The hard dependency (?) on docker is super annoying.

I also have a existing grafana instance. The insistence of the plugins to shove what are basically completely independent services into the hass interface is annoying when you're trying to use esphome as a component of an existing system, and you'd like to avoid having 2 instances of everything.

[fake-name]

https://github.com/esphome/esphome/pull/719 would go a long ways towards supporting this. Most TS databases support simple HTTP(S) addition endpoints.

[brandond]

Otto has been pretty clear that he's only interested in supporting Home Assistant with this project, so this is probably not going to get added.

I use Prometheus, which Home Assistant has a built in exporter for. Works great.

[fake-name]

Even if other people provide the integration? That's..... spectacularly lame.

[brandond]

Maybe @ottowinter has changed his mind, but see: https://github.com/esphome/ESPHome/issues/55#issuecomment-398390637

You might also take a look at https://github.com/esphome/feature-requests/issues/40

[fake-name]

The MCU size argument is a bit silly. Minimal HTTP can be done on a 8-bit AVR with 2 kb of ram. Sure, it won't support some of the esoteric corner cases, but you can do minimal HTTP by hand with a telnet client. It's not substantially more complex then MQTT.

Other integrations (like StatsD) are simpler. It's a stateless UDP protocol, the hardest part is the DNS lookup, and that's already done elsewhere so it doesn't add any complexity.

---

If ESPHome is intended to be used only with Hass.io, that's fine, but it'd be really nice if the documentation reflected that. I wouldn't have pursued it if that's the case.

I already have extensive logging and automation infrastructure. I'd probably be less bothered about the hass requirement if I didn't have a graphite database with years of logs and a fully configured grafana front end already set up.

The little tinkering I've done so far has been that ESPHome is really nice for implementing bespoke sensors. I'd be sad to have to abandon it.

[OttoWinter]

Yeah I understand the use case. And I also thought about something similar some time ago. Most simple arduino projects also do just that: log directly into a time series database. Having the ability to set up a project with a small db system that is directly supported is something that is a potential goal for the future.

The reason I don't usually allow any HTTP stuff is because then people will try to pull in tons of web services. Maybe something for weather information, some for traffic conditions etc (there's an endless supply of that, see Home Assistant). Adding all those would clutter the codebase incredibly. (The argument here has shifted over time; yes HTTP can be compact, but if you add more things like response/error handling it quickly grows; but, esphome has since then gained an HTTP server which is kinda the same thing as a client).

That's why some work's being done on a generic HTTP request component: https://github.com/esphome/esphome/pull/719#issuecomment-545785181 It will allow the DIY stuff to work, but prevent cluttering the codebase with cloud integrations (at least that's the hope).

The other issue is how components share their data. HA has chosen a representation where everything is really just a simple json-able datastructure. In C++ that would not work so well, so ESPHome has base classes like Sensor etc.

Each "frontend" (stuff that sends data onward, like MQTT) needs to implement a sender for each component type (Sensor, etc). That process is not very scalable, meaning if a new component type is added significant work goes into adding an implementation for each sender type. A fix for that would be to have stuff like a time-series database sender be of lower priority, that are updated asynchronously - which will not be that popular with users.

The hard dependency (?) on docker is super annoying.

There is no hard dependency on docker anywhere. Yes Hass.io (note: not Home Assistant) uses docker, but you can use Home Assistant without Hass.io too.

[paravoid]

There is now a PR for review that adds statsd output: https://github.com/esphome/esphome/pull/6642 which may be relevant to the subscribers of this issue.

[kpfleming]

I have an external component which supports publishing measurements to InfluxDB (v2). I have considered proposing this for inclusion, and could probably do that now that http_request is supported on the three primary hardware architectures in ESPHome.