search

tgalarneau / bms

JBD BMS and Thornwave bluetooth data monitoring
43 stars 17 forks source link

MQTT Testing - 8s #6

Open the-butterfry opened 3 years ago

the-butterfry commented 3 years ago

Starting this as a clarification thread for setup and deploy of the MQTT transport option. I observed that I was using much more data than I'd hoped. Current setup is BMS(local) > Telegraf (local) > Influx (local) > Grafana (remote). This meant anytime Grafana tried to render for a client it pulls data from Influx over the VPN. I burned my 3g data up in no time. Current thinking is now, send the MQTT data once remotely and have the queried DB local to Grafana.

MQTT Setup: BMS(local) > MQTT Broker(vpn remote) > Telegraf(vpn remote) > Influx(vpn remote) > Grafana(vpn remote) > Home Assistant/client grafana view

Observations: For my setup and the unreliability of tunneled 3g cellular internet, MQTT QOS is important. In the .py file one request would be to have a single location to set QOS instead of at each message. topic ="data/bms" gauge ="data/bms/gauge" broker="192.168.xx.1" port=1883 qos=1

tgalarneau commented 3 years ago

It should be easy to use a global qos variable. Higher qos means more data will be sent.

If all data is sent to remote, it seems to me that it would increase your data use as now only sending the subset of data required to complete your graphs. Maybe could also run grafana locally and only send snapshots of web page graphs to remote and not the data and remove the need for sending any data. Or perhaps, just remotely view the local grafana instance as needed.

On Wed, May 26, 2021 at 6:18 AM Cory Verellen @.***> wrote:

Starting this as a clarification thread for setup and deploy of the MQTT transport option. I observed that I was using much more data than I'd hoped. Current setup is BMS(local) > Telegraf (local) > Influx (local) > Grafana (remote). This meant anytime Grafana tried to render for a client it pulls data from Influx over the VPN. I burned my 3g data up in no time. Current thinking is now, send the MQTT data once remotely and have the queried DB local to Grafana.

MQTT Setup: BMS(local) > Telegraf(vpn remote over 3g) > Influx(vpn remote) > Grafana(vpn remote) > Home Assistant/client grafana view

Observations: For my setup and the unreliability of the internet, MQTT QOS is important. In the .py file one request would be to have a single location to set QOS instead of at each message.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tgalarneau/bms/issues/6, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATWKVYNG6J3DGTUZ2JDRDALTPUNMTANCNFSM45SNCD5A .

-- Tom

tgalarneau commented 3 years ago

For my system I use multiple mqtt topics.

temps ="data/bms/temp" gauge ="data/bms/gauge" balance = "data/bms/balance" volt = "data/bms/volts" minmax = "data/bms/minmax" protecting = "data/bms/protect"

You could remotely subscribe only to say "gauge" for remote real time viewing and then use a local grafana for more detailed info.

On Wed, May 26, 2021 at 7:42 AM Tom Galarneau @.***> wrote:

It should be easy to use a global qos variable. Higher qos means more data will be sent.

If all data is sent to remote, it seems to me that it would increase your data use as now only sending the subset of data required to complete your graphs. Maybe could also run grafana locally and only send snapshots of web page graphs to remote and not the data and remove the need for sending any data. Or perhaps, just remotely view the local grafana instance as needed.

On Wed, May 26, 2021 at 6:18 AM Cory Verellen @.***> wrote:

Starting this as a clarification thread for setup and deploy of the MQTT transport option. I observed that I was using much more data than I'd hoped. Current setup is BMS(local) > Telegraf (local) > Influx (local) > Grafana (remote). This meant anytime Grafana tried to render for a client it pulls data from Influx over the VPN. I burned my 3g data up in no time. Current thinking is now, send the MQTT data once remotely and have the queried DB local to Grafana.

MQTT Setup: BMS(local) > Telegraf(vpn remote over 3g) > Influx(vpn remote) > Grafana(vpn remote) > Home Assistant/client grafana view

Observations: For my setup and the unreliability of the internet, MQTT QOS is important. In the .py file one request would be to have a single location to set QOS instead of at each message.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tgalarneau/bms/issues/6, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATWKVYNG6J3DGTUZ2JDRDALTPUNMTANCNFSM45SNCD5A .

-- Tom

-- Tom

the-butterfry commented 3 years ago

Yeah, good ideas. I'm just trying to get it going. There is something funky that I couldn't sort out so i fell back to sockets for now.

I was getting data, but it would only do a few iterations and then hang. I think it's my mqtt input config (only thing i've not looked at).. anyhow, seems to be fine with the following

BMS(local) > Telegraf(local) > Influx(vpn remote) > Grafana(vpn remote) > Home Assistant/client grafana view.

I haven't looked at the data size for the Telegraf to Influx communication, but it's vastly less impactful than multiple clients hitting the DB on the local side everytime the view refreshes.

Onsite i've got a RPi with a touchscreen that is running a version of the dashboard in kiosk mode all the time. It was fetching over VPN to grafana remote > Grafana was querying the DB back over the vpn > sending data back over to hand off to grafana and then finally back onsite to the RPi display... total waste. Now it's only going over VPN once to refresh. I can probably sort that out by running local copies... tomorrow.

tgalarneau commented 3 years ago

I put my mqtt broker on pi and only subscribe from remote.

On Wed, May 26, 2021 at 10:37 AM Cory Verellen @.***> wrote:

Yeah, good ideas. I'm just trying to get it going. There is something funky that I couldn't sort out so i fell back to sockets for now.

I was getting data, but it would only do a few iterations and then hang. I think it's my mqtt input config (only thing i've not looked at).. anyhow, seems to be fine with the following

BMS(local) > Telegraf(local) > Influx(vpn remote) > Grafana(vpn remote) > Home Assistant/client grafana view.

I haven't looked at the data size for the Telegraf to Influx communication, but it's vastly less impactful than multiple clients hitting the DB on the local side everytime the view refreshes.

Onsite i've got a RPi with a touchscreen that is running a version of the dashboard in kiosk mode all the time. It was fetching over VPN to grafana remote > Grafana was querying the DB back over the vpn > sending data back over to hand off to grafana and then finally back onsite to the RPi display... total waste. Now it's only going over VPN once to refresh. I can probably sort that out by running local copies... tomorrow.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tgalarneau/bms/issues/6#issuecomment-849100354, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATWKVYMVYQ5AGP6LL757EOLTPVLXZANCNFSM45SNCD5A .

-- Tom