jasonacox / Powerwall-Dashboard

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems
MIT License
270 stars 57 forks source link

V3.0.0 - Integrate Solar Only Support #386

Closed mcbirse closed 7 months ago

mcbirse commented 7 months ago

v3.0.0 - Updates

Integrate Solar Only Support

Related

jasonacox commented 7 months ago

✅ Checking Upgrade on RPi Setup from 2.10.0 to 3.0.0 - Steps to force upgrade:

git pull
gh pr checkout 386
cp upgrade.sh tmp.sh
bash tmp.sh upgrade
...
./verify.sh

Output

WARNING: You already have the latest version (v3.0.0).

Upgrade Powerwall-Dashboard from 3.0.0 to 3.0.0
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.

Upgrade - Proceed? [y/N] Y

Resetting Timezone to Default...

Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...

No local changes to save
Already up to date.

Setting Timezone back to America/Los_Angeles...

Updating Powerwall-Dashboard stack...
Running Docker Compose...
[+] Running 5/5
 ✔ Container influxdb     Running                                                                                                                  0.0s 
 ✔ Container pypowerwall  Started                                                                                                                  3.5s 
 ✔ Container weather411   Started                                                                                                                  3.8s 
 ✔ Container grafana      Running                                                                                                                  0.0s 
 ✔ Container telegraf     Started                                                                                                                  3.2s 

Waiting for InfluxDB to start...
 up!

Add downsample continuous queries to InfluxDB...
2023/11/18 17:47:35 error: retention policy already exists
2023/11/18 17:47:35 error: retention policy already exists
2023/11/18 17:47:35 error: retention policy already exists
2023/11/18 17:47:35 error: continuous query already exists
2023/11/18 17:47:35 error: continuous query already exists

Deleting old pyPowerwall...
pypowerwall
pypowerwall

Deleting old telegraf...
telegraf
telegraf

Deleting old weather411...
weather411
weather411

Restarting Powerwall-Dashboard stack...
Running Docker Compose...
[+] Running 5/5
 ✔ Container pypowerwall  Started                                                                                                                  2.0s 
 ✔ Container influxdb     Running                                                                                                                  0.0s 
 ✔ Container grafana      Running                                                                                                                  0.0s 
 ✔ Container weather411   Started                                                                                                                  2.3s 
 ✔ Container telegraf     Started                                                                                                                  2.0s 

---------------[ Update Dashboard ]---------------
Open Grafana at http://localhost:9000/

From 'Dashboard/Browse', select 'New/Import', and
upload 'dashboard.json' located in the folder
/home/pi/Powerwall-Dashboard/dashboards/

Please note, you may need to select data sources
for 'InfluxDB' and 'Sun and Moon' via the
dropdowns and use 'Import (Overwrite)' button.
Verify Powerwall-Dashboard 3.0.0 on Linux - Timezone: America/Los_Angeles
----------------------------------------------------------------------------
This script will attempt to verify all the services needed to run
Powerwall-Dashboard. Use this output when you open an issue for help:
https://github.com/jasonacox/Powerwall-Dashboard/issues/new

Checking configuration
----------------------------------------------------------------------------
 - Dashboard configuration: default
 - EnvVar COMPOSE_PROFILES: default,weather411

Checking pypowerwall
----------------------------------------------------------------------------
 - Config File pypowerwall.env: GOOD
 - Container (pypowerwall): GOOD
 - Service (port 8675): GOOD
 - Version: 0.6.2 Proxy t28
 - Powerwall State: CONNECTED - Firmware Version: 23.28.2

Checking telegraf
----------------------------------------------------------------------------
 - Config File telegraf.conf: GOOD
 - Local Config File telegraf.local: GOOD
 - Container (telegraf): GOOD
 - Version: Telegraf 1.28.2 (git: HEAD@8d9cf395)

Checking influxdb
----------------------------------------------------------------------------
 - Config File influxdb.conf: GOOD
 - Container (influxdb): GOOD
 - Service (port 8086): GOOD
 - Filesystem (./influxdb): GOOD
 - Version: InfluxDB shell version: 1.8.10

Checking grafana
----------------------------------------------------------------------------
 - Config File grafana.env: GOOD
 - Container (grafana): GOOD
 - Service (port 9000): GOOD
 - Filesystem (./grafana): GOOD
 - Version: Grafana CLI version 9.1.2

Checking tesla-history
----------------------------------------------------------------------------
 - Skipped: Only required in 'solar-only' configuration

Checking weather411
----------------------------------------------------------------------------
 - Config File weather/weather411.conf: GOOD
 - Container (weather411): GOOD
 - Service (port 8676): GOOD
 - Weather: {"temperature": 15.86}
 - Version: 0.2.2

All tests succeeded.

New Install

Powerwall Dashboard (v3.0.0) - SETUP
-----------------------------------------
Select configuration profile:

 1 - default     (Powerwall w/ Gateway on LAN)
 2 - solar-only  (No Gateway - data retrieved from Tesla Cloud)

Select profile: Select profile: 1

Timezone (leave blank for America/Los_Angeles)
Enter Timezone: 

Enter credentials for Powerwall...
Password: !@#$!%%#@@!
Email: example@example.com
IP Address: 10.1.2.3

Using America/Los_Angeles timezone...
-----------------------------------------

Weather Data Setup
-----------------------------------------
Weather data from OpenWeatherMap can be added to your Powerwall Dashboard
graphs.  This requires that you set up a free account with OpenWeatherMap
and enter the API Key during this setup process.

Do you wish to setup Weather Data? [y/N] y
Forecast looks great!  Proceeding...

Set up a free account at OpenWeatherMap.org to get an API key
   1. Go to https://openweathermap.org/
   2. Create a new account and check your email to verify your account
   3. Click on 'API Keys' tab and copy 'Key' value and paste below.

Enter OpenWeatherMap API Key: 123456789012345678901234567890

Enter your location coordinates to determine weather in your location.
   For help go to https://jasonacox.github.io/Powerwall-Dashboard/location.html

Enter Latitude: 34.123456
Enter Longitude: -118.654321

Enter the desired units: M)etric, I)mperial or S)tandard where:
    M)etric = temperature in Celsius
    I)mperial = temperature in Fahrenheit
    S)tandard = temperature in Kelvin

Enter M, I or S: m
Units selected: metric

NOTE: The OpenWeatherMap key can take up to 2 hours to be valid.
      You may see errors or no data until it is fully activated.

Weather Configuration Complete

Running Docker Compose...
[+] Running 42/34
 ✔ grafana 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                  74.1s 
 ✔ influxdb 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                 75.9s 
 ✔ weather411 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                             19.2s 
 ✔ pypowerwall 8 layers [⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                             68.2s 
 ✔ telegraf 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                  84.3s                                                                                   
[+] Running 6/6
 ✔ Network powerwall-dashboard_default  Creat...                                       0.2s 
 ✔ Container pypowerwall                Started                                       22.8s 
 ✔ Container influxdb                   Started                                       22.8s 
 ✔ Container grafana                    Started                                        3.2s 
 ✔ Container telegraf                   Started                                        2.8s 
 ✔ Container weather411                 Started                                        3.0s 
-----------------------------------------
Waiting for InfluxDB to start...
... up!
Setup InfluxDB Data...
Executing single run query run-once-2.8.0.sql file...
Executing single run query run-once-2.8.3.sql file...
Executing single run query run-once-2.9.2.sql file...
Fetching local weather...
weather411
------------------[ Final Setup Instructions ]-----------------

Open Grafana at http://localhost:9000/ ... use admin/admin for login.

Follow these instructions for *Grafana Setup*:

* From 'Configuration\Data Sources' add 'InfluxDB' database with:
  - Name: 'InfluxDB'
  - URL: 'http://influxdb:8086'
  - Database: 'powerwall'
  - Min time interval: '5s'
  - Click "Save & test" button

* From 'Configuration\Data Sources' add 'Sun and Moon' database with:
  - Name: 'Sun and Moon'
  - Enter your latitude and longitude (tool here: https://bit.ly/3wYNaI1 )
  - Click "Save & test" button

* From 'Dashboard\Browse', select 'New/Import', and upload 'dashboard.json'
  from the /home/pi/Powerwall-Dashboard/dashboards folder.
jasonacox commented 7 months ago

🔴 Issue with a Ubuntu hosted instance during upgrade:

Upgrade Powerwall-Dashboard from 2.10.0 to 3.0.0
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.

Upgrade - Proceed? [y/N] y

Resetting Timezone to Default...

Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...

No local changes to save
remote: Enumerating objects: 111, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 111 (delta 68), reused 94 (delta 54), pack-reused 0
Receiving objects: 100% (111/111), 60.95 KiB | 770.00 KiB/s, done.
Resolving deltas: 100% (68/68), completed with 14 local objects.
From https://github.com/jasonacox/Powerwall-Dashboard
   6e26eee..faa3a81  main       -> origin/main
 * [new branch]      v3.0.0     -> origin/v3.0.0
Updating 6e26eee..faa3a81
Fast-forward
 .gitignore                                                 |    7 +
 README.md                                                  |   61 +-
 RELEASE.md                                                 |    8 +
 VERSION                                                    |    2 +-
 WINDOWS.md                                                 |  201 +++
 compose.env.sample                                         |   11 +-
 dashboards/README.md                                       |   16 +-
 .../dashboard.json => dashboards/dashboard-solar-only.json |  574 +++------
 dashboards/dashboard.json                                  |    4 +-
 powerwall.yml                                              |   26 +
 setup.sh                                                   |  286 ++++-
 tools/README.md                                            |   36 +-
 tools/solar-only/README.md                                 |  130 +-
 tools/solar-only/compose-dash.sh                           |   69 --
 tools/solar-only/compose.env.sample                        |   27 -
 tools/solar-only/grafana.env.sample                        |   34 -
 tools/solar-only/grafana/.keep                             |    1 -
 tools/solar-only/influxdb.conf                             |  154 ---
 tools/solar-only/influxdb/dropcq.sql                       |   25 -
 tools/solar-only/influxdb/influxdb.sql                     |   42 -
 tools/solar-only/influxdb/run-once-2.8.0.sql               |   15 -
 tools/solar-only/influxdb/run-once-2.8.3.sql               |   13 -
 tools/solar-only/influxdb/run-once-2.9.2.sql               |   18 -
 tools/solar-only/powerwall.yml                             |   71 --
 tools/solar-only/setup.sh                                  |  178 ---
 tools/solar-only/tesla-history/README.md                   |  286 -----
 tools/solar-only/tesla-history/tesla-history.py            | 1560 ------------------------
 tools/solar-only/tz                                        |    1 -
 tools/solar-only/tz.sh                                     |   31 -
 tools/solar-only/weather.sh                                |  149 ---
 tools/solar-only/weather/weather411.conf.sample            |   34 -
 tools/{solar-only => }/tesla-history/Dockerfile            |    0
 tools/tesla-history/tesla-history.py                       |  793 +++++++++---
 tools/{solar-only => }/tesla-history/upload.sh             |    0
 tz.sh                                                      |   18 +-
 upgrade.sh                                                 |  247 +++-
 verify.sh                                                  |  355 ++++--
 weather.sh                                                 |   86 +-
 38 files changed, 1964 insertions(+), 3605 deletions(-)
 create mode 100644 WINDOWS.md
 rename tools/solar-only/dashboard.json => dashboards/dashboard-solar-only.json (85%)
 delete mode 100755 tools/solar-only/compose-dash.sh
 delete mode 100644 tools/solar-only/compose.env.sample
 delete mode 100644 tools/solar-only/grafana.env.sample
 delete mode 100644 tools/solar-only/grafana/.keep
 delete mode 100644 tools/solar-only/influxdb.conf
 delete mode 100644 tools/solar-only/influxdb/dropcq.sql
 delete mode 100644 tools/solar-only/influxdb/influxdb.sql
 delete mode 100644 tools/solar-only/influxdb/run-once-2.8.0.sql
 delete mode 100644 tools/solar-only/influxdb/run-once-2.8.3.sql
 delete mode 100644 tools/solar-only/influxdb/run-once-2.9.2.sql
 delete mode 100644 tools/solar-only/powerwall.yml
 delete mode 100755 tools/solar-only/setup.sh
 delete mode 100644 tools/solar-only/tesla-history/README.md
 delete mode 100644 tools/solar-only/tesla-history/tesla-history.py
 delete mode 100644 tools/solar-only/tz
 delete mode 100755 tools/solar-only/tz.sh
 delete mode 100755 tools/solar-only/weather.sh
 delete mode 100644 tools/solar-only/weather/weather411.conf.sample
 rename tools/{solar-only => }/tesla-history/Dockerfile (100%)
 rename tools/{solar-only => }/tesla-history/upload.sh (100%)
 mode change 100644 => 100755
Current branch main is up to date.

Setting Timezone back to America/Los_Angeles...

Updating Powerwall-Dashboard stack...
Running Docker Compose...
ERROR: The Compose file './powerwall.yml' is invalid because:
Unsupported config option for services.pypowerwall: 'profiles'
Unsupported config option for services.telegraf: 'profiles'
Unsupported config option for services.tesla-history: 'profiles'
Unsupported config option for services.weather411: 'profiles'

$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~18.04.3
$ docker-compose --version
docker-compose version 1.17.1, build unknown
jasonacox commented 7 months ago

Issue corrected with v3.0.1.

jasonacox commented 7 months ago

Thanks for this upgrade @mcbirse !!! 🙏

Please feel free to propose an alternative to my v3.0.1 patch. I need to add the docker-compose V1 hosts to my testing plan going forward. In any case, it seems to work. Looking for any input from others.

mcbirse commented 7 months ago

Hi @jasonacox - I committed directly a bug fix for the v3.0.1 patch as I believe the order in which the checks for docker compose V1/V2 need to be the other way round.

i.e. check for V2 first using docker compose version and if that fails, then fall back to the V1 check docker-compose version and use the individual .yml files since "profiles" is likely not supported.

Why check for V2 first?

The docker-compose command is aliased when Docker Compose V2 is installed (at least it is on my system, and it is also mentioned in the docs: https://docs.docker.com/compose/migrate/#how-do-i-switch-to-compose-v2)

Running both commands on my Alpine Linux system produces the below:

$ docker-compose version
Docker Compose version v2.17.3
$ docker compose version
Docker Compose version v2.17.3

So running the first command (with the dash) is not a definitive test to say "only Docker Compose V1 is installed", so we should check for V2 first.

This is just a quick fix, hopefully there are no issues with it.

I would still like to look further for other alternatives to the patch to support Docker Compose V1 users, however the current solution is good I believe.

mcbirse commented 7 months ago

Looking into this issue further, Docker Compose V1 does actually support Profiles, as long as the version being used is 1.28.0 or newer...

I wonder if it would be better to check the version of Docker Compose before resorting to using the fix?

Or, perhaps, we could run something similar to below to check for errors instead, which validates the compose config.

if docker compose -f powerwall.yml $pwextend config > /dev/null 2>&1; then
    #all good
else
    #failed - assume no profiles support
fi

I do also wonder how many people are still using Docker version older than 1.28.0 (i.e. pre 2021-01-20) and should it be supported?

jasonacox commented 7 months ago

Good call/fix on testing for docker V2 first.

I do also wonder how many people are still using Docker version older than 1.28.0 (i.e. pre 2021-01-20) and should it be supported?

Fair point! I happen to still have an old Ubuntu 18.04.6 system running and it has docker-compose 1.17.1. I have intended to upgrade it but also seemed to find it mirroring some of the odd behaviors Synology NAS users would report. This is the curse of any project. When do you stop supporting old version when it has spread to so many platforms? :)

I ran this:

if docker-compose -f powerwall.yml config > /dev/null 2>&1; then
   echo "You have a current docker-compose"
else
   echo "You have an old docker-compose"
fi

It works. "You have an old docker-compose". I suggest we do something like this:

echo "Running Docker Compose..."
if docker compose version > /dev/null 2>&1; then
    # Build Docker (v2)
    docker compose -f powerwall.yml $pwextend $@
else
    if docker-compose version > /dev/null 2>&1; then
        # Build Docker (v1)
        if docker-compose -f powerwall.yml config > /dev/null 2>&1; then
            pwconfig="powerwall.yml"
        else
            echo "** WARNING **"
            echo "    You have an old version of docker-compose that will"
            echo "    be depreciated in a future release. Please upgrade or"
            echo "    report your use case to the project."
            echo ""
            echo "Applying workaround for old docker-compose..."
            pwconfig="powerwall-v1.yml"
            if get_profile "solar-only"; then
                pwconfig="powerwall-v1-solar.yml"
            fi
        fi
        docker-compose -f $pwconfig $pwextend $@
    else
        echo "ERROR: docker-compose/docker compose is not available or not running."
        echo "This script requires docker-compose or docker compose."
        echo "Please install and try again."
        exit 1
    fi
fi

This will still apply the workaround for now, but provide a user WARNING and hopefully prompt the community to report any edge use cases we need to consider before removing old docker-compose support.

What do you think?

Successful test output on my old system:

$ bash compose-dash.sh up -d
Running Docker Compose...
** WARNING **
    You have an old version of docker-compose that will
    be depreciated in a future release. Please upgrade or
    report your use case to the project.

Applying workaround for old docker-compose...
pypowerwall is up-to-date
influxdb is up-to-date
grafana is up-to-date
weather411 is up-to-date
telegraf is up-to-date

And success with docker compose V2 system:

$ bash compose-dash.sh up -d
Running Docker Compose...
[+] Running 5/0
 ✔ Container influxdb     Running                                                      0.0s 
 ✔ Container weather411   Running                                                      0.0s 
 ✔ Container grafana      Running                                                      0.0s 
 ✔ Container pypowerwall  Running                                                      0.0s 
 ✔ Container telegraf     Running                                                      0.0s
jasonacox commented 7 months ago

I added these changes to a v3.0.2 branch - https://github.com/jasonacox/Powerwall-Dashboard/commit/5ed0838d0259f4df30b01aeb06b48506659729cb

mcbirse commented 7 months ago

What do you think?

I added these changes to a v3.0.2 branch

Great solution, I like this!

I committed a small change... since the compose.env file is now defined in the globals as a variable, for consistency I replaced the hardcoded references with the variable. And also fixed a typo. 😉

Looks good to merge to me.

jasonacox commented 7 months ago

Thanks, @mcbirse ! Merged. v3.0.2 is live.