Upgrade QoL isssues/enhancements

[BuongiornoTexas]

Hi @jasonacox,

I've just upgraded to the latest version via upgrade.sh and hit a couple of issues on the way - most of them due to modifying my own configuration and doing some preliminary development work. Based on these, I came up with a couple of very low priority quality of life enhancements. If you see value in any of these, I'm happy to work up PRs to gradually address them after I've got the first pass cumulative kWh PR in place.

• Could we add implement a user generated addition to telegraf.conf in setup and upgrade scripts - maybe appended from a telegraf.local file, which I think works OK with TOML? This would be helpful for maintaining personal customisations such as my three phase power monitoring, and may also be useful when developing new functionality.

• I was hit with my rootless docker and grafana permissions issue again. It would be good if we could customise powerwall.yml to replace the default user for grafana (maybe by environment variable)? This problem is mostly mine at the moment, but it'd be nice to sort out.

• Minor issue: Should there be a dashboard.json in both the root folder and in the dashboards folder? The one in the root folder has the latest changes to the temperature and sun and moon.

• Closed in latest update. Very minor issue: I got hit with a repeated UID error when importing the new dashboard json while keep the old dashboard in place for comparison. It may be worth changing the UID when new versions of the dashboard are released.

• Moved to new issue. On dashboards, I haven't had time to figure out library panels in detail, but maybe they can be used to provide clean up updates to small elements of the dashboard, and maybe allow easier roll-your-own dashboards?

• On to do list already I think the influxdb.sql import raised errors about CQs already existing - maybe these should be dropped before re-adding?

[youzer-name]

• On dashboards, I haven't had time to figure out library panels in detail, but maybe they can be used to provide clean up updates to small elements of the dashboard, and maybe allow easier roll-your-own dashboards?

I had typed up a reply and then closed the tab... argh! Hopefully typing this again will let me be more organized and concise, but don't bet on it :-)

Library panels could be useful for people making their own dashboards, plus they make it easier for the official project to have multiple dashboard variations for people to choose from. For example we could have a version for Powerwall+ and a version for regular Powerwall... (Powerwall Classic? OG Powerwall? Powerwall-?) and the user could set either one as their home dashboard based on their setup. I also think others might find some value in having multiple dashboards with different levels of detail, or ones showing current data vs. older data. That is how I've set up my Grafana.

I've been using library panels because I run a simplified dashboard set as my 'home' dashboard. I have the big dashboard that is closer to the project's default dashboard linked from there, but I don't really look at the big dashboard that often. More on that below.

On a mostly unrelated tangent... are the electricity prices for the 'savings' panel still hard-coded into the queries? Because that seems like a good place to use Grafana variables. If the variable is coded into the query, it is easy for the user to set and update the variable as needed (although since we're not storing the data, any changes to pricing will change the totals when looking at older data). This is another thing I might find easier to do using my mySQL copy of the data. It should be relatively easy to set up a table there with electricity costs for different time periods, and then get mySQL to return a cost per kWh that is accurate for each date. I may play with that if I get a chance. My panel is set up as a 'cost' panel instead of a 'savings' panel, since I have a solar PPA and pay for all of my power with one rate for grid and another for solar.

Back from the tangent... here are some screenshots of how I've set up my multiple dashboards:

The dashboard that I have set as 'home' in Grafana fits on one screen without scrolling on most of my devices. It just runs off the bottom of the screen on the laptop I'm using now:

At the top you can see a row with dashboard links. I have that set to show all starred dashboards. When expanded, it looks like this:

So current status is my simplified dashboard, Powerwall Monitor is a customized version of the project's standard dashboard, and I have two others that show data from other sources (one monitors my home automation and raspberry Pi hardware and one monitors my HVAC and some weather data).

The power flow and energy usage are displayed on both my simplified "Current Status" dashboard and on my Powerwall-Monitor dashboard, so they are library panels.

I was thinking of making a new home dashboard with just the data I look at the most from all 4 current dashboards. Any panels that I pull from the other dashboards that aren't already in the library will get put there first, and then added to the new home dashboard.

Just found out that deprecated visualization types don't work as library panels. Time to go down the rabbit hole of trying to convert the Savings (or in my case, cost) graph from Graph(old) to a Bar Chart.

[BuongiornoTexas]

Hopefully typing this again will let me be more organized and concise

I'm familiar with the problem myself :-).

Library panels could be useful for people making their own dashboards, plus they make it easier for the official project to have multiple dashboard variations for people to choose from.

This and your experience lines up with what I was hoping/expecting from the library panels. From what you are saying, it should be possible to transition (slowly) from custom dashboards to a modular panel library with dashboard templates that share elements from this library?

I like your summary dashboard layout! Could it be used as a test bed for this enhancement?

On a mostly unrelated tangent... are the electricity prices for the 'savings' panel still hard-coded into the queries?

Into the grafana queries? Yup. I'm mildly interested in this as well because I'm on a time of use tariff, so my value depends when usage occurs in the day. I'm pretty sure the way to hit this is with influx queries. The price data could be stored as an infrequently updated table in influx, or in grafana if the variables support array forms. This one is down the list of my priorities.

[BuongiornoTexas]

Answering one of my own questions: docker compose provides support for environment variable expansion in the .yml, and allows specification of these variables in a .env file. https://docs.docker.com/compose/environment-variables/

[youzer-name]

This and your experience lines up with what I was hoping/expecting from the library panels. From what you are saying, it should be possible to transition (slowly) from custom dashboards to a modular panel library with dashboard templates that share elements from this library?

Yes. I think we could pack all the options for panels into the library and then set up one or more standard dashboards, but users could easily build their own dashboards by adding panels from the library in any combination or arrangement they want. When something needs to be updated, only one edit needs to be made and it affects all the linked panels.

I like your summary dashboard layout! Could it be used as a test bed for this enhancement?

I just did some major revisions to it so it no longer looks like the screenshot above :-) But if @jasonacox is interested in going down this route, I'd be happy to work on it. I can start with the current standard dashboard, update any panels that are listed as "(old)" to current ones, and add them all to the library. I can make two dashboards... one as close as possible to the current standard one, and one 'lite' one, and then see if all of those changes can be merged into the project.

On a mostly unrelated tangent... are the electricity prices for the 'savings' panel still hard-coded into the queries?

Into the grafana queries? Yup. I'm mildly interested in this as well because I'm on a time of use tariff, so my value depends when usage occurs in the day. I'm pretty sure the way to hit this is with influx queries. The price data could be stored as an infrequently updated table in influx, or in grafana if the variables support array forms. This one is down the list of my priorities.

My experience over the last few days says this may be challenging in InfluxDB. The InfluxQL language doesn't allow for 'datepart' type queries. The Flux language reportedly does, but despite working with databases every day, trying to do anything in Flux gives me acid reFlux. I can handle syntax variations between InfluxQL, mySQL, and MSSQL, but Flux is a completely different thing.

If I were going to try what you're describing, I'd definitely copy the data from InfluxDB to mySQL and then handle the cost calculations in mySQL queries. It should be relatively easy to join a table with dates and times defining different rates to the kwh totals (or even the detailed autogen data) using mySQL. I expect it would be much more difficult if not impossible in InfluxDB.

[jasonacox]

There is a lot here. 😄

BuongiornoTexas - quality of life enhancements?

First, @BuongiornoTexas - these are gold and so are you! I love your writing... let me take some of these:

Could we add implement a user generated addition to telegraf.conf in setup and upgrade scripts

Great point - Taking the pypowerwall.env.sample approach we could have telegraf.conf.sample that setup copies to telegraf.conf which is not impacted in the upgrade process. And YES to submitting a PR if you want to take this on.

It would be good if we could customise powerwall.yml to replace the default user for grafana (maybe by environment variable)?

I know about the env files, but can you give details on how we could do that? Yes on PR here too if you can figure it out.

I got hit with a repeated UID error when importing the new dashboard json

Great point, we need to change the UID. Anyone here, feel free to update the UIDs and submit a PR. But also see my "Dashboard Market" note below...

influxdb.sql import raised errors about CQs already existing

I have that on my list. If/when we actually change he CQs that will be an issue. I'm hesitant to do anything much to InfluxDB because @youzer-name loves it so much. 😆 In all seriouness, the urgency on this is low and I prefer to do a > /dev/null 2>&1 on those commands to mute the non-impacting error noise for now.

panels

@youzer-name I'm interested but how do we make it easy and not create a lot of bloat? I have a slightly different proposal, but still open to the panel idea. So....

Dashboard Market?

I must say, I love your dashboard @youzer-name ! It occurs to me that we have no good way to share and store community built dashboards. I suspect the list and creativity will just keep going. I want to keep the project basic and simple on initial install with a default starter panel like we have. BUT, I would like to make it easy for the community to browse and shop for other dashboard (panels too?) that are easily imported.

What if we converted the /dashboards folder into a "User Contributed" store of dashboards (and panels)? It would have a README with screenshots and details for each, including "how to import" instructions. Community members here could submit their dashboard layouts via pull request. Thoughts?

[youzer-name]

Dashboard Market?

I must say, I love your dashboard @youzer-name ! It occurs to me that we have no good way to share and store community built dashboards. I suspect the list and creativity will just keep going. I want to keep the project basic and simple on initial install with a default starter panel like we have. BUT, I would like to make it easy for the community to browse and shop for other dashboard (panels too?) that are easily imported.

What if we converted the /dashboards folder into a "User Contributed" store of dashboards (and panels)? It would have a README with screenshots and details for each, including "how to import" instructions. Community members here could submit their dashboard layouts via pull request. Thoughts?

@jasonacox Having complete dashboards available for download could be very useful, but I still think having at least some of the panels in the library might help keep things easier to maintain. That way if someone wants to let others do updates and improvements, or put another way, if one wants to stay with the 'official' panels, they can use the library. Then when there is a change made, like adding weather to the energy use panel, any user-created dashboards that include that panel will also be updated. The library would also allow some built-in customization by having interchangeable panels, like a 'cost' panel for people who pay per kWh for their solar with a PPA and a 'savings' panel for people who don't pay for the solar per kWh.

As I see it the Market idea has (at least) two benefits. First, people can share those user-created dashboards, and other people might download them or use them for inspiration for their own custom dashboards. Second, people would be free to go off-script with the dashboards they upload to the Market, and if people (mainly you as the maintainer) like the ideas enough, they could flow back into the main project.

While trying to work around some issues caused by migrating the Graph (old) panels to the new Bar Graph panel... it seems to be impossible to get the numeric values to appear in the legend... I ended up with this (for now).

I'm still not sure if I like the horizontal bars on use and cost, but the one stat panel with house/solar/grid/battery totals replaced 4 different panels that were showing those numbers before.

And related to the topic about possibly changing the CQs or data sampling: I currently have the stat panel pulling the numbers from the integral and the bar graph using the sum of the kwh.http bucket. While they're close, they are not identical. Looking at yesterday's numbers, the integral matches what I get from the Tesla app for solar and grid. The kwh.http exactly matches for Home use, and neither matches the app-reported battery use.

The 'solar' page in my Tesla app shows 7.5 kWh produced yesterday but the page with the Powerwall info shows 6.8 kWh. I assume one number is being reported by the inverter and the other by the Powerwall Gateway. I looked at more days and the Powerwall page was higher on every one I looked at except yesterday. When looking at week or month intervals, the Powerwall page was always higher. Considering that I pay for what is reported on the Solar page, I'm surprised it's showing the lower number. My September bill shows 552.38 kWh, the solar part of the app shows 552.4 kWh, but the Powerwall part of the app shows 575.7 kWh! So in my case it's not worth chasing the difference between the integral and the kwh bucket, because my 'cost' in Grafana will always be off by 5% or so due to the difference between the gateway and the inverter.

[BuongiornoTexas]

From this discussion, it looks as though there are legs to do some form of modularising dashboards - can I suggest we raise it as a separate enhancement discussion and/or issue, rather than keeping it in this grab bag set?

I'll get on to the minor PRs when I get a break from other things in my life.

| I assume one number is being reported by the inverter and the other by the Powerwall Gateway

You'll need to dig into the specifics of your setup to see what is being monitored (system_status, site meters, vitals, and any separate neurio meter if you have one). For instance, I know I have CTs on supply phase, on each solar system and by the looks of things, one on the battery. Site load looks like it is being done by difference. But from my reading, other line ups can be quite different.

I've got a separate issue on reconciling the app and the local powerwall data #87 - adding information on these type of issues is likely to help me with progressing it. But I'm with you on mismatches between the app and the data we can pull directly - losing hair on trying to figure out what is happening there. To date, I have one (1) piece of good news: over a 9 day period, the powerwall CTs pretty well match the solar output my inverters say they are generating - but the app thinks I'm generating much less. Still waiting for my current meters to changed over into a single three phase meter, at which point I'm hoping to do a similar reconciliation on the grid import/export. Unfortunately, I think I'm going to need to spend some time with Teslapy to get to grips with this.

[BuongiornoTexas]

@jasonacox - I'm still learning my way around this - let me know if I have made any mistakes in making the PR.

PR closing this issue implements the following:

• Key change: adds compose-dash.sh utility script to run docker compose, as the command line becomes unwieldy with the updates implemented in this PR.
• Adds local compose.env file which provides environment variables to docker compose. Currently only implements GRAFANAUSER.
• Overrides the telegraf command in docker compose to read additional telegraf .conf files from a config directory. Maps telegraf.local to local.conf in this directory. Can add non-standard measurements and/or use for local development testing without modifying the main development tree.

Setup/upgrade will silently create the local compose.env or telegraf.local files if they do not exist, and these default settings should work for most users. Setup/upgrade will not modify these files if they exist already.

CAUTION! I've tested setup.sh, but don't know how to test upgrade.sh due to #104.