jasonacox
commented
4 months ago Thanks @BuongiornoTexas - running tests...
Closed BuongiornoTexas closed 3 months ago
jasonacox
commented
4 months ago Thanks @BuongiornoTexas - running tests...
jasonacox
commented
3 months ago Testing this for v4.1.0
mcbirse
commented
3 months ago Tested by simulating an upgrade using the new upgrade.sh script, new grafana.env.sample, etc. per the files for the merge.
Note the git pull may look different to a normal upgrade, as I tested this from a private repository to try to simulate a valid upgrade.
Plugin list before running upgrade:
# ls -l grafana/plugins/
drwxr-sr-x 3 admin admin 4096 Mar 10 18:23 fetzerch-sunandmoon-datasource
drwxr-sr-x 6 admin admin 4096 Mar 10 18:23 grafana-influxdb-flux-datasource
drwxr-sr-x 4 admin admin 4096 Mar 10 18:23 grafana-piechart-panel
drwxr-sr-x 3 admin admin 4096 Mar 10 18:23 simpod-json-datasource
drwxr-sr-x 8 admin admin 4096 Mar 10 18:23 yesoreyeram-boomtable-panelRan the upgrade:
# ./upgrade.sh
Upgrade Powerwall-Dashboard from 4.0.5 to 4.1.0
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.
Upgrade - Proceed? [y/N] y
Resetting Timezone to Default...
Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...
No local changes to save
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (1/1), done.
remote: Total 6 (delta 4), reused 6 (delta 4), pack-reused 0
Unpacking objects: 100% (6/6), 1.98 KiB | 337.00 KiB/s, done.
From https://github.com/mcbirse/testing
9350def..7c70119 main -> origin/main
Updating 9350def..7c70119
Fast-forward
Powerwall-Dashboard/VERSION | 2 +-
Powerwall-Dashboard/grafana.env.sample | 13 ++++++++++++-
Powerwall-Dashboard/upgrade.sh | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
3 files changed, 66 insertions(+), 13 deletions(-)
Updating 'grafana.env' structure. '.old' backup file created.
Grafana plugin list has changed.
You appear to be running an older Grafana configuration with
plugins that are different to the current recommended list.
This script can update your plugins to the current recommendations.
If you have installed custom plugins or want to keep it as
it is, answer 'n' below and the script will provide instructions for
manual checks/updates.
Do you want to update your Grafana plugin list? [y/N] y
grafana
grafana
Setting Timezone back to America/Los_Angeles...
Updating Powerwall-Dashboard stack...
Including powerwall.extend.yml
Running Docker Compose...
[+] Running 5/5
✔ Container pypowerwall Running 0.0s
✔ Container influxdb Running 0.0s
✔ Container weather411 Running 0.0s
✔ Container grafana Started 1.7s
✔ Container telegraf Running 0.0s
Waiting for InfluxDB to start...
up!
Add downsample continuous queries to InfluxDB...
2024/03/10 07:34:21 error: continuous query already exists
2024/03/10 07:34:21 error: continuous query already exists
Deleting old pyPowerwall...
pypowerwall
pypowerwall
Deleting old telegraf...
telegraf
telegraf
Deleting old weather411...
weather411
weather411
Restarting Powerwall-Dashboard stack...
Including powerwall.extend.yml
Running Docker Compose...
[+] Running 5/5
✔ Container pypowerwall Started 2.0s
✔ Container influxdb Running 0.0s
✔ Container weather411 Started 2.1s
✔ Container grafana Running 0.0s
✔ Container telegraf Started 3.5s
---------------[ Update Dashboard ]---------------
Open Grafana at http://localhost:9000/
From 'Dashboard\Browse', select 'New/Import', browse to /home/admin/testing/Powerwall-Dashboard/dashboards
and upload 'dashboard.json'.
Please note, you may need to select data sources
for 'InfluxDB' and 'Sun and Moon' via the
dropdowns and use 'Import (Overwrite)' button.The revised plugin list after upgrade no longer includes "grafana-influxdb-flux-datasource":
# ls -l grafana/plugins/
drwxr-sr-x 3 admin admin 4096 Mar 10 18:34 fetzerch-sunandmoon-datasource
drwxr-sr-x 4 admin admin 4096 Mar 10 18:34 grafana-piechart-panel
drwxr-sr-x 3 admin admin 4096 Mar 10 18:34 simpod-json-datasource
drwxr-sr-x 8 admin admin 4096 Mar 10 18:34 yesoreyeram-boomtable-panelPlugins were re-downloaded and registered when grafana container started:
# docker logs grafana
✔ Downloaded grafana-piechart-panel v1.6.4 zip successfully
Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.
✔ Downloaded yesoreyeram-boomtable-panel v1.5.0-alpha.3 zip successfully
Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.
✔ Downloaded fetzerch-sunandmoon-datasource v0.3.0 zip successfully
Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.
✔ Downloaded simpod-json-datasource v0.6.3 zip successfully
Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.
logger=plugin.manager t=2024-03-10T07:34:23.59679441Z level=info msg="Plugin registered" pluginId=input
logger=plugin.manager t=2024-03-10T07:34:23.715230233Z level=info msg="Plugin registered" pluginId=grafana-piechart-panel
logger=plugin.manager t=2024-03-10T07:34:23.715577783Z level=info msg="Plugin registered" pluginId=simpod-json-datasource
logger=plugin.manager t=2024-03-10T07:34:23.715957717Z level=info msg="Plugin registered" pluginId=yesoreyeram-boomtable-panel
logger=plugin.manager t=2024-03-10T07:34:23.716077987Z level=info msg="Plugin registered" pluginId=fetzerch-sunandmoon-datasourceAll worked okay from what I can tell. Plugin list was pruned and re-downloaded core plugins only, and Grafana + Dashboard working.
During the upgrade I see an error about some cq's already exists. I'm not sure why, and I don't think this will cause an issue... maybe this always happens and I've never noticed!?
EDIT - just thought should test an upgrade where user would already have latest grafana.env changes - all good.
WARNING: You already have the latest version (v4.1.0).
Upgrade Powerwall-Dashboard from 4.1.0 to 4.1.0
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.
Upgrade - Proceed? [y/N] y
Resetting Timezone to Default...
Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...
No local changes to save
Already up to date.
Grafana plugin list is current. No action required.Thanks @mcbirse !
During the upgrade I see an error about some cq's already exists. I'm not sure why, and I don't think this will cause an issue... maybe this always happens and I've never noticed!?
Yes, this is a known issue. We have always run the full influxdb.sql file during each upgrade without deleting the CQs first. Only new CQs in the file will be accepted with a warning about the existing ones.
jasonacox
commented
3 months ago Thanks, @BuongiornoTexas ! Good addition.
I merged this as version 4.1.0 since it removes plugins that could be breaking if someone made additions assuming flux datasource was available.
jasonacox
commented
3 months ago I have an old Linux box (Ubuntu 18.04.6 LTS) that I use for a final test. It did fail during the upgrade. I somehow didn't get the right PR during tests before the merge. 😞
Error: ✗ Plugin not found (Grafana v9.1.2 linux-amd64)
This Linux box uses docker-compose version 1.17.1. The issue appears that the old docker compose doesn't support variable substitution so that GF_INSTALL_PLUGINS=${PWD_CORE_PLUGINS} will always be empty. Only Docker-compose 1.5+ support variables substitution.
I keep this old box around because this was a fairly common distro when we first started the project. I would dismiss it but I wonder if there are others in the community with this version of docker-compose that will face this same problem (either on a Linux box, RPi or NAS). The only fix is to revert or to remove the substitution. I'm going to investigate the impact of removing the substitution.
jasonacox
commented
3 months ago It appears that anyone still stuck on the older docker-compose versus docker compose will be impacted by this. I'm going to revert for now until we can figure out best next steps.
jasonacox
commented
3 months ago Reverted the added upgrade logic but kept the new plugin list. Also, adjusted logic in case someone ran 4.1.0 on a docker-compose system. It will remediate it if they update. Right now the logic is simple: looks for tag in grafana.env "Updated v4.1.1" to trigger prompt to let user update to full grafana.env.sample. Not as elegant as it will remove any customizations.
Release v4.1.1: https://github.com/jasonacox/Powerwall-Dashboard/releases/tag/v4.1.1
BuongiornoTexas
commented
3 months ago @jasonacox - I do understand your desire to continue supporting as many OS versions as possible, but I'll also note that even with 18.04 being an LTS, it is now out of its extended support period for nearly a year and is only getting security maintenance. I think you could make the case that anyone sticking with 18.04 is probably doing it for good reason and should be capable of installing docker updates to get things running properly for their system (the current major LTS is due for release shortly and its up to version 24!).
I don't have any real skin in the game, but this is the second time I've been caught by long tail issues in this project (the last one was also related to docker compose V1 - while V2 hasn't been out quite as long as Ubuntu 18.04, it was released 3-4 years ago!). These issues are going to make it harder and harder to make useful contributions to the code base in the long term.
jasonacox
commented
3 months ago If it was just Ubuntu, I agree with you and would have kept as is, possibly just added a warning note to upgrade. However, as I investigated, I discovered it was due to the docker-compose which is not tied to Ubuntu 18.04. Latest version can include docker-compose (even if they don't need it). We currently accommodate it in our setup and ./compose-dash.sh. I debated doing the complete upgrade and removing docker-compose support with a warning that the user needs to update docker and remove docker-compose. That is still worthwhile effort (we need to do that) but I wanted to get your changes in as I think removing those unused plugins is overwhelmingly helpful and a needful cleanup.
Thanks for your help @BuongiornoTexas . I recognize the frustration, but I hope to convey that I genuinely appreciate your contributions. I'm also keen to ensure the upgrades effective (and mostly issue free) for our large community. And yes, we do have a growing tail. :)
BuongiornoTexas
commented
3 months ago I discovered it was due to the docker-compose which is
No problem - I understood the issue. It was much more a generic comment about the problems of maintaining LONG backward compatibility while adding new features (if influxdb 3.0 improves functionality dramatically, I can see it resulting in a major crunch - not holding my breath on that, but hopeful at this point).
It's also frustrating that the thing that is holding us back on this issue is the docker stack itself - which is kind of defeating the purpose of containerisation in the first place!
Improved handling of grafana plugin updates in upgrade.sh.
Closes #433