Tesla Solar-Only Dashboard

[jasonacox]

There has been a lot of interest by Tesla Solar owners who don't have a Powerwall, to get a similar dashboard for their system. I'm opening this as a feature request to explore creating a derivative dashboard using a similar stack plus the tesla-history script developed by @mcbirse as a potential service to feed a solar-only dashboard.

Reference: Reddit Thread

Proposal

Add a Tesla-Solar-Only dashboard option to in the tools folder. Basically, a InfluxDB+Grafana+Import-Tool (optional +Weather411) stack. The Import-Tool could be converted to be a python service, for instance, that polls the Tesla API every 5m or so and stores it in InfluxDB. We could use the dashboard.json and remove the Powerwall specific panels.

We need someone who has a Solar-Only setup to validate.

[apU823]

I have Solar-Only and can assist in validating

[mcbirse]

@jasonacox and @apU823 - I like this idea and it is certainly possible I believe. 👍

I have some ideas/thoughts about this that I'll throw out there to help determine what options we have and how we might best go about implementing this.

One thing I noted while developing the tesla-history tool was the ability to get "current" site data, i.e. solar generation etc. and the thought struck me that you could implement an alternative service to pypowerwall (or even make it configurable) that, instead of using the local gateway for the data source, could use the Tesla cloud service directly.

Will come back to this later with a more detailed post though, once I've had some more time to document and test a few ideas I have.

[apU823]

Hello,

following up if any way to support the non-PW dashboard :)

[jasonacox]

I created a placeholder for this effort: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/solar-only

It duplicates all of the core Dashboard project files with edits to remove pypowerwall and telegraf. Ultimately, I would like to de-dupe all of this if we can get it to work and have it as an option in the main setup.sh. But for now, and for testing, we can start here.

TO BE DONE: The setup creates the InfluxDB and Grafana services. The tesla-history script will need to be part of the setup.sh to get the Token, and then be converted to a service to poll the API every x minutes to pull in the updated data.

Also, @mcbirse if you have a better idea, I'm happy to change course.

[mcbirse]

Hello,

following up if any way to support the non-PW dashboard :)

Sorry @apU823 - I have not had time to test or investigate my ideas due to other projects taking all my time 😞

First step is trying to work out what changes we need to make to the tesla-history script.

Are you able to run the script in login mode with the debug option? It would be interesting to see what data is returned from your Tesla account about your site (if anything) without any changes to the script first.

You can do this as below.

# Run tesla-history login with debug output
python3 tesla-history.py --login --debug

If you could post your output that would be great.

Make sure to remove your personal information however, such as email, site id & name, address, geolocation, and serial numbers, etc.

Currently the tesla-history script uses the battery_list function of the TeslaPy module, so the above may not work at all. This may be something we need to change for solar-only accounts, as the solar_list function might be required instead.

@jasonacox - I think your approach is a great start to try to get something working.

I had some other ideas, but unfortunately I don't think I have time to work on this. My thoughts were to implement "live" monitoring rather than getting "historical" data from the Tesla cloud (like the tesla-history script does currently). Definitely possible, but more effort required.

In it's most basic / brute force form however, if the tesla-history script can retrieve historical data for solar-only accounts, it would even be possible to simply run it from a cron job every five minutes (e.g. run python3 tesla-history.py --today). Of course, converting it to a service that polls for data every 5 minutes is probably nicer.

[apU823]

Got the following error. forgot to run with debug on first so ran it a 2nd time

pi2@rp4:~/Powerwall-Dashboard/Powerwall-Dashboard/tools/tesla-history $ python3 tesla-history.py --login

Config file 'tesla-history.conf' not found

Do you want to create the config now? [Y/n] Y

Tesla Account Setup
-------------------
Email address: EMAILID@gmail.com
Save auth token to: [tesla-history.auth] yes

InfluxDB Setup
--------------
Host: [localhost]
Port: [8086]
User (leave blank if not used): [blank]
Pass (leave blank if not used): [blank]
Database: [powerwall]
Timezone (e.g. America/Los_Angeles): American/New_York
Invalid timezone

Timezone (e.g. America/Los_Angeles): America/New_York

Config saved to 'tesla-history.conf'

----------------------------------------
Tesla account: EMAILID@gmail.com
----------------------------------------
Open the below address in your browser to login.

https://auth.tesla.com/oauth2/v3/ -- full website removed

After login, paste the URL of the 'Page Not Found' webpage below.

Enter URL after login: LOGIN URL REMOVED
----------------------------------------
ERROR: No Tesla Energy sites found

pi2@rp4:~/Powerwall-Dashboard/Powerwall-Dashboard/tools/tesla-history $ python3 tesla-history.py --login --debug
----------------------------------------
Tesla account: EMAILID@gmail.com
----------------------------------------
ERROR: No Tesla Energy sites found
pi2@rp4:~/Powerwall-Dashboard/Powerwall-Dashboard/tools/tesla-history $

[mcbirse]

@apU823 - thanks for posting. This confirms we will need to change the script to use the solar_list function instead for solar-only accounts.

I'll work on some changes to the tesla-history script.

@jasonacox - I'll submit the updated script to the placeholder you have created, so we can keep these changes separate for now during testing and development.

I think first steps are some minimal changes to the script to see what historical data can be retrieved from solar-only accounts, unless you had any other suggestions?

[jasonacox]

Thanks @mcbirse ! I agree.

[mcbirse]

Hi @apU823 - I have updated the tesla-history script which should hopefully 🤞 now support Tesla accounts with solar only products.

I am unable to test the changes however, so it would be great to get some feedback from your testing.

Please note - the updated script is available in this directory only: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/solar-only/tesla-history

For testing this, you should be able to run git pull to get the latest changes, or may need to run upgrade.sh if you had already installed Powerwall-Dashboard.

Then when testing, make sure you are running the script from the "tools/solar-only/tesla-history" directory now.

If you hadn't installed Powerwall-Dashboard, you may need to run setup.sh from the "tools/solar-only" directory first as well - as the tesla-history script may not run fully without the InfluxDB database being available.

If you can try some of the below tests and post the output, that would terrific (please remove any personal info though).

# Run tesla-history login with debug output
python3 tesla-history.py --login --debug

Hopefully, this will now show some information about your site, rather than returning "ERROR: No Tesla Energy sites found".

We also need to find out what power history data is returned for solar only sites.

So, if the above worked, then you can try the below command. This should return output required to help us determine what further changes may be required based on the response data. 🙏

# Retrieve power history data for all of yesterday
python3 tesla-history.py --test --force --debug --yesterday

Looking forward to see how this goes! (and hoping it's not just a complete fail 😬 - very hard to know what will happen when unable to test this myself....)

[jasonacox]

Awesome @mcbirse ! Thanks!

@apU823 Were you able to run the setup (here)?

Thanks for helping test this. Let us know what you discover.

[apU823]

@mcbirse - did the git pull before i ran any of the scripts but still getting the same error:

pi2@rp4:~/Powerwall-Dashboard/tools/tesla-history` $ python3 tesla-history.py --login --debug

Config` file 'tesla-history.conf' not found

Do you want to create the config now? [Y/n] Y

Tesla Account Setup
-------------------
Email address: email@gmail.com
Save auth token to: [tesla-history.auth] yes

InfluxDB Setup
--------------
Host: [localhost]
Port: [8086]
User (leave blank if not used): [blank]
Pass (leave blank if not used): [blank]
Database: [powerwall]
Timezone (e.g. America/Los_Angeles): America/New_York

Config saved to 'tesla-history.conf'

----------------------------------------
Tesla account: email@gmail.com
----------------------------------------
ERROR: No Tesla Energy sites found

This is what my tesal-history.conf file looks like:

[Tesla]
USER = email@gmail.com
AUTH = yes

[InfluxDB]
HOST = localhost
PORT = 8086
USER = 
PASS = 
DB = powerwall
TZ = America/New_York

[apU823]

@jasonacox @mcbirse @jweier

I am also working with another developer here: https://github.com/jweier/SolarDataParser

I've been able to use his code to output data (on a daily basis). Not sure if this would be helpful to somehow merge his code into this project

[mcbirse]

@apU823 - It looks like you are running the original tesla-history script (which hasn't be changed).

Before trying to run the test commands I listed, please change your working directory.

cd ~/Powerwall-Dashboard/tools/solar-only/tesla-history

[apU823]

i swear i did the git pull before i ran the code but now it did pull it correctly.

the code ran

python3 tesla-history.py --test --force --debug --yesterday

{ "timestamp": "2023-03-05T04:55:00Z", "solar_power": 0, "battery_power": 0, "grid_power": 0, "grid_services_power": 0, "generator_power": 0 } ]

^^this was the last time slot it pulled the data for

saw this at the end of my output

Writing to InfluxDB (*** skipped - test mode enabled ***) Updating InfluxDB (*** skipped - test mode enabled ***) Done.

does this mean it did not write to the db?

whats next? would you like me to share the terminal output?

[mcbirse]

Hi @apU823 - that's great, looks like it is working!

Yes, I included --test option so it does not actually write to your InfluxDB. The --force option would also allow it to output data without needing the InfluxDB database present (hopefully), just in case things were not set up correctly. We can proceed with trying to write data later after I have reviewed and potentially made some more changes to the script.

Would be very helpful if you could post the full output from both the --login --debug (remove personal info) and the --test --force --debug --yesterday commands please (I know it will be long).

What I really need to see are example "power" values from when your solar was producing something. Thanks!

[apU823]

tesla-history.py - login - debug.txt tesla-history.py --test --force --debug --yesterday.txt

Please see attached.

a couple questions -

1. i am supposed to have two inventors installed on my system but it seems to only show one. is that how you read it as well?
2. the second file, it seems to end at 4:55AM (hence no solar generated) is that expected behavior?

[mcbirse]

Great, thanks. I need some time to review in detail to see what is going on here.

At first glance, there seems to be some issues though.

1. I'm not sure what Tesla is supposed to show for the inverters to be honest. I guess we would need to see examples from other people that have more than 1 Tesla inverter (I have 2x other brand inverters, so do not get any inverter data from my system).
2. Definitely not the expected behaviour.

In reference to 2. above. There seems to be an issue with timezones that is likely causing this. At the moment it is not getting a full day like it should.

Your output is showing "installation_time_zone": "America/New_York" for the site config, but when retrieving the historical data, this is showing "installation_time_zone": "UTC", which is weird and unexpected.

The script does not look at the timezone when retrieving the historical data and assumes it would be the same as the site config. This is likely causing the issue with getting a full day of data.

I'll need to make some further changes to the script to hopefully address this.

[mcbirse]

Hi @apU823 - I have updated the script - once this has been merged, please pull the latest changes and re-test the same commands as before and post the output. 🤞

[youzer-name]

@mcbirse I had a Tesla solar-only installation since Nov 2016 and Powerwalls + a Gateway 2 added in Feb 2022. I tried to use the latest history script to pull the historical data from when I only had solar, and I am getting an error:

Running: python3 tesla-history.py -t -d --start "2016-11-01 00:00:00" ---end "2023-02-27 23:59:59" returns "ERROR: Failed to retrieve SITE_CONFIG - 'site_name'"

I tried adding "--site "################" with the site ID that is listed for my solar and I still get the same response.

Running this in debug mode I can see that it is authenticating ok and is getting the data for two sites. First it has a long section for Get SITE_CONFIG for Site ID {my powerwall site ID}. Then it has a section starting with "Get SITE_DATA for Site ID {my powerwall site ID}". Then it shows a section starting with "Get SITE_CONFIG for Site ID {my solar site ID}". After all that config data it shows the error.

I also tried running it with the --site specifying my Powerwall site ID and get the same error.

I have successfully used an older version of the history tool in the past to fill in gaps in my data. I just ran that older version and it works for dates after my Powerwall was added, and doesn't return any data but does not throw any errors for dates before my Powerwall was added.

[youzer-name]

Tracing the code as best I can, it looks like there is no SITE_NAME attribute for my solar-only site which causes the error here:

 sitename = data['response']['site_name']

Here is the SITE_CONFIG with personal data removed:

Get SITE_CONFIG for Site ID 1234567890123456
{
    "response": {
        "id": "{siteUUID}",
        "site_number": "STE01234567-12345",
        "installation_date": "2016-11-07T19:00:00-05:00",
        "user_settings": {
            "storm_mode_enabled": null,
            "powerwall_onboarding_settings_set": null,
            "powerwall_tesla_electric_interested_in": null,
            "sync_grid_alert_enabled": false,
            "breaker_alert_enabled": false
        },
        "components": {
            "solar": true,
            "solar_type": "pv_panel",
            "battery": false,
            "grid": true,
            "backup": false,
            "gateway": "neo",
            "load_meter": false,
            "tou_capable": false,
            "storm_mode_capable": false,
            "flex_energy_request_capable": false,
            "car_charging_data_supported": false,
            "off_grid_vehicle_charging_reserve_supported": false,
            "vehicle_charging_performance_view_enabled": false,
            "vehicle_charging_solar_offset_view_enabled": false,
            "battery_solar_offset_view_enabled": false,
            "energy_service_self_scheduling_enabled": true,
            "rate_plan_manager_supported": true,
            "configurable": false,
            "grid_services_enabled": false
        },
        "time_zone_offset": -300,
        "geolocation": {
            "latitude": 39.9999999,
            "longitude": -77.9999999
        },
        "address": {
            "address_line1": "{My Address}",
            "city": "{MyTown}",
            "state": "MD",
            "zip": "12345",
            "country": "US"
        }
    }
}

[youzer-name]

So, after commenting out every reference to sitename and sitetimezone, since neither exist in my solar-only response data, I got it to list both sites.

Get SITE_DATA for Site ID {######}
{
    "response": {
        "solar_power": 3440,
        "grid_status": "Unknown",
        "grid_services_active": false,
        "timestamp": "2023-03-06T14:47:43-05:00",
        "wall_connectors": null
    }
}
      Site ID: {#######}
    Site type: Powerwall x2
    Installed: 2022-02-28 12:00:52-05:00
  System time: 2023-03-06 14:47:41-05:00
----------------------------------------
      Site ID: {#########}
    Site type: Solar
    Installed: 2016-11-07 19:00:00-05:00
  System time: 2023-03-06 14:47:43-05:00
----------------------------------------
ERROR: Multiple Tesla Energy sites found - select site with option --site "Site ID"

I then ran it with the solar site ID and it started throwing more errors, so I kept commenting out anything that was erroring on the missing site name and site timezone, and finally added "sitetz = influxtz" around line 897 just so there would be some value for sitetz, and seems to be working. I don't know how the timezones are lining up yet. I'm going to try to pull some old solar data and see if I can compare with the Tesla app.

[youzer-name]

I don't need to compare to the Tesla app, I can use the time of production vs the sun/moon graph to see if the data make sense.

I pulled 01 Jan 2022. Judging from the sun/moon curve and the solar curve, I think the timezones are lined up correctly:

I can't tell if it's matched up during DST. This is 01 Jul 2021. It looks like production starts after sunrise and ends after sunset, but I'm not sure:

Edit: Sometimes I'm a little slow... I compared to graphs from July 2022 and they are consistently centered on the sun/moon curve, so I think my hacked version is causing the data to be off by one hour during DST.

One issue: It seems the script (with my hacking) is duplicating the solar data in the home usage data, which is why my plots are green instead of yellow in the screenshots. Is that intentional? It seems the home usage should be null when pulling the solar-only site data. I hacked at it a little more and have it only writing the solar value to the database.

[youzer-name]

F###. Just deleted my entire http data while trying to delete one day of the imported data... and I think my backup wasn't working correctly, so I can't restore the database.

Looks like I'll need to use the import tool to restore my entire last year of Powerwall data. Have I mentioned that I HATE influxdb?

[mcbirse]

@youzer-name - thank you! This is really helpful information. 🙏

It is so difficult to make changes to the script without being able to test against your own system. And clearly, there are many variables with the data that the Tesla cloud can respond with or not include, that we need to try to account for.

Sorry to hear your data got deleted! Was this an issue with the tesla-history --remove option or just a silly typo with a date range? Just want to make sure the script didn't cause this unintentionally.

Are you able to give some debug output that includes the power history data please?

i.e. similar to what @apU823 provided below. I would really like to see if your data also includes the "installation_time_zone" field just before the "time_series" data.

* Loading daily history: [2023-03-04]
{
    "serial_number": "************************",
    "installation_time_zone": "UTC",
    "time_series": [
        {
            "timestamp": "2023-03-05T00:00:00Z",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },

I think your timezone data lined up mostly okay (although maybe not the DST changes), as I notice in your output for the site list both the "Installed" time and "System time" are the same timezone, so manually setting the timezone to influxtz would probably be fine.

    Installed: 2016-11-07 19:00:00-05:00
  System time: 2023-03-06 14:47:43-05:00

However, with the site list from @apU823 I noticed his "Installed" time is in UTC, but "System time" is in his timezone. I think this is why the power history timeseries data is returned in UTC - as his system may have been set to UTC initially I guess, and the Tesla cloud continues to store/return that data in UTC, despite the timezone being corrected later?

    Installed: 2023-01-15 21:18:03+00:00
  System time: 2023-03-05 22:33:17-05:00

InfluxDB - also not much of a fan. Timezones and dealing with datetimes and DST etc. - definitely not a fan!!

Given the info you have provided already, I can see some further improvements I can make to the script and will work on those. Thank you.

[youzer-name]

@youzer-name - thank you! This is really helpful information. 🙏

I'm glad my sacrifice helped the team 😦

Sorry to hear your data got deleted! Was this an issue with the tesla-history --remove option or just a silly typo with a date range? Just want to make sure the script didn't cause this unintentionally.

Not the script, but I didn't think I "typo'd" either. I think maybe I just issued a command that wasn't formatted correctly and Influx just ran with it.

I'm still suffering from PTSD (post-traumatic-stupid-database_user), but I think what I did first was I tried to delete the data I imported for 01 Jul 2021 by "delete from autogen.http where time < '2021-07-02'. InfluxDB responded with something about not being able to delete from specific retention policies... so I dropped the 'autogen' part. I'm pretty sure that "delete from http where time < '2021-07-02' wiped my entire http measurement. If I had typo'd the < into a >, I would still have had the data from 01 July, but everything was gone. So I think it just ignored the date. (I was running from "influx -precision rfc3339", so the date format should have been good).

Are you able to give some debug output that includes the power history data please?

i.e. similar to what @apU823 provided below. I would really like to see if your data also includes the "installation_time_zone" field just before the "time_series" data.

I've attached one day's data pulled with my modified version of the script. Mine has a "time_zone_offset" just before the time_series.

tesla-history.txt

I think your timezone data lined up mostly okay (although maybe not the DST changes), as I notice in your output for the site list both the "Installed" time and "System time" are the same timezone, so manually setting the timezone to influxtz would probably be fine.

The Powerwall gateway data that I just recreated using the old version of the tool seems to have everything lined up correctly for both DST and non-DST.

InfluxDB - also not much of a fan. Timezones and dealing with datetimes and DST etc. - definitely not a fan!!

Given the info you have provider already, I can see some further improvements I can make to the script and will work on those. Thank you.

The original script was a lifesaver for me today.

Lesson learned about verifying backups. My backup cron job was running every night, but not doing anything useful. Part of the script needed sudo rights, but not the part that made the archive and wrote it to my network folder, so I saw a new file showing up every day as expected and thought it was working. Unfortunately the part of the script where Influx was supposed to be taking the new snapshot wasn't working, so I've been backing up the database as of a day in March 2022 every night since then! It just kept backing up the last snapshot that I successfully made when originally setting the backup up.

I had to move the job to the sudo cron (sudo crontab -e) to get it to work. I haven't done an actual restore, so I suppose I'm still not being smart, but I opened the archive on the other end and checked the file dates to make sure it's working now.

I'm tempted to go full rogue and push everything to MySQL and reconfigure all my dashboards to use that. At least I know how not to hit the big red 'kill all my data' button in MySQL... (I think)

[apU823]

Hi @apU823 - I have updated the script - once this has been merged, please pull the latest changes and re-test the same commands as before and post the output. 🤞

waiting for merge to be approved :)

[mcbirse]

waiting for merge to be approved :)

All good @apU823 - I will be making more changes to the script and submitting updates anyway, but of course if you do have a chance to test that update it is still most welcome - the more feedback and example data I get the better!

I'm still suffering from PTSD (post-traumatic-stupid-database_user),

Lol - @youzer-name so sorry for your troubles and subsequent PTSD!

InfluxDB responded with something about not being able to delete from specific retention policies...

Yes, this is a frustrating limitation of InfluxDB where you cannot delete from a specific retention policy.

This is why in the tesla-history script, every data point added by that is tagged with a source=cloud tag (a tag that has never been used by anything else in this project).

The --remove option of the script uses this tag in the WHERE clause when deleting data, so no other data points or retention policies would ever be affected (even if you entered the entire date range of your data), only the data imported with the script. Highly recommend to use the script to remove imported data rather than trying to do it manually.

I'm pretty sure that "delete from http where time < '2021-07-02' wiped my entire http measurement. If I had typo'd the < into a >, I would still have had the data from 01 July, but everything was gone. So I think it just ignored the date.

I'm not sure what InfluxDB has done or why (it does weird unexpected things), but I have always made sure when I enter a datetime range to use a full date & time in the query, e.g. '2021-07-02 00:00:00' - I wonder if this was the cause of the misunderstood or ignored time period?

I've attached one day's data pulled with my modified version of the script. Mine has a "time_zone_offset" just before the time_series.

Thanks, this is really helpful. The "time_zone_offset" rather than a timezone name is interesting... I will need take this into account.

Also interesting is that it shows "time_zone_offset": -300 which is -5hrs, but I believe DST should be active for your timezone in that month, which is confirmed from the script output Retrieving data for gap: [2021-06-01 00:00:00-04:00] showing -4hrs, as the script operates in the configured influx timezone, and will recognise DST changes.

Also note from your data, because of that difference, the last hour of that day's data is missing.

Are you able to pull another day of data from a time period outside of DST please? I would like to see if the "time_zone_offset" value remains static at -300 or not... (trying to determine if we can rely on this to consistently configure the sitetz!??? -hope so)

The Powerwall gateway data that I just recreated using the old version of the tool seems to have everything lined up correctly for both DST and non-DST.

If the sitetz is determined correctly, everything should work perfectly. So as above for your "Solar" site, it appears sitetz should be set to an offset of -5hrs.

The original version of the script seemed to be working well, since the site timezone determined from the site config appears to be consistent with the timeseries data being returned. But these "Solar" only sites appear to be different.

Unfortunately all this having to deal with the timezones is due to how the calendar history data is returned. When we want to retrieve a full day of data, you need to send Tesla an end datetime for the day (e.g. 1s before midnight) but it has to be in the timezone that Tesla is storing it, or it won't send you the entire day.

There is a "timezone" parameter that can be sent with the request, but I have never been able to test this properly - I would need to test against someone else's site to determine the behaviour.

The original script was a lifesaver for me today.

Excellent. With some more changes I'm sure I'll get the updated script working with the "Solar" only sites as well. Dealing with timezone issues is a PITA though.

Silver lining from all this being your backups are working now? 😉

[youzer-name]

I'm not sure what InfluxDB has done or why (it does weird unexpected things), but I have always made sure when I enter a datetime range to use a full date & time in the query, e.g. '2021-07-02 00:00:00' - I wonder if this was the cause of the misunderstood or ignored time period?

That might be it. I haven't had the heart to test it (on a different InfluxDB container!) yet. I knew about the -remove command, I was just being stupid. At a minimum I should have done a "SELECT COUNT" query first with the same parameters to see how many records it was going to affect.

Also note from your data, because of that difference, the last hour of that day's data is missing.

Luckily with solar-only, the midnight hour doesn't usually contain a lot of data 😄

Are you able to pull another day of data from a time period outside of DST please? I would like to see if the "time_zone_offset" value remains static at -300 or not... (trying to determine if we can rely on this to consistently configure the sitetz!??? -hope so)

Here are the first few lines of a pull from standard time, solar-only, using my hacked script:

Running for period: [2021-01-01 00:00:00-05:00] - [2021-01-01 23:59:59-05:00] (23:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2021-01-01 00:00:00-05:00] - [2021-01-01 23:59:59-05:00] (23:59:59s)

Retrieving data for gap: [2021-01-01 00:00:00-05:00] - [2021-01-01 23:59:59-05:00] (23:59:59s)
* Loading daily history: [2021-01-01]
{
    "serial_number": "25957e56-56c1-448a-a1ce-c45a404691bc",
    "time_zone_offset": -300,
    "time_series": [
        {
            "timestamp": "2021-01-01T00:00:00-05:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2021-01-01T00:15:00-05:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2021-01-01T00:30:00-05:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },

If you need the whole day, let me know.

Silver lining from all this being your backups are working now? 😉

Hopefully. Better to have this happen now than after a few more years, I guess. Other than the data resolution, it looks like I lost the power frequency/voltage and the powerwall capacity data. Of those, the first year's reported capacity would have been the nicest to still have.

I'm going to try a restore from backup later this week.

[jasonacox]

F###. Just deleted my entire http data while trying to delete one day of the imported data... and I think my backup wasn't working correctly, so I can't restore the database.

Oh no!!! We should probably do more in the backup documentation space. TODO

For anyone helping with troubleshooting or development for the project, here is what I do: As a rule, I never use my main DB for testing. I always snapshot and deploy my DB to a separate instance (another Raspberry Pi) - using this:

# ideally stop the influxdb container first
docker stop influxdb

# snapshot
sudo tar -zcf snapshot.tgz Powerwall-Dashboard/influxdb/

I know that wouldn't make sense for most, but I needed a way to quickly test PRs or other changes w/o risking my data. At a minimum, doing a snapshot of the DB before testing means you can always roll back to that point in time if the test goes badly. I know that doesn't help you now (hopefully you still have a snapshot or can use the history script to pull in most).

[apU823]

@mcbirse

ran with the most current code. Please see files attached.

tesla-history.py --login --debug 3.7.23.txt tesla-history.py --test --force --debug --yesterday 3.7.23.txt

it seemed to have pulled all 24 hours (in 5min increments)

not sure if this was intended but I ran the script on: "timestamp": "2023-03-07T22:13:28-05:00",

but it pulled data for 3/8 3/7 and 3/6.

I was expecting it to only pull data for yesterday based on this command:

# Retrieve power history data for all of yesterday python3 tesla-history.py --test --force --debug --yesterday

[mcbirse]

Thanks @apU823 - I will look at this in more detail later, but it sounds promising.

Pulling different days of data to get your 1 day of "yesterday" would be correct.

This is because the timeseries data from Tesla is in UTC for your site. So your "yesterday" would encompass periods of those 2 days in UTC time.

When data is actually imported into InfluxDB it will convert to your defined local TZ and grab the correct time data from the daily data that was retrieved.

Example below: 2.4kW of solar generation at 9pm at night, which is unlikely. This would be converted to your local TZ when importing so the time is correct.

        {
            "timestamp": "2023-03-06T21:00:00Z",
            "solar_power": 2438.6666666666665,

I might update the line "* Loading daily history: [ date ]" to include the timezone perhaps.

Here's the 2 days in UTC it downloaded to get your "yesterday".

* Loading daily history: [2023-03-06]
{
    "serial_number": "**********************",
    "installation_time_zone": "UTC",
    "time_series": [
        {
            "timestamp": "2023-03-06T00:00:00Z",

* Loading daily history: [2023-03-07]
{
    "serial_number": "**********************",
    "installation_time_zone": "UTC",
    "time_series": [
        {
            "timestamp": "2023-03-07T00:00:00Z",

The other (1st section in the debug output) will always be a subset of the most recent daily data. This is just used to find out what timezone the timeseries data is in. You'll noticed there is no text output "Loading daily history" just before that output.

Thanks for testing. I think I'm on the right track to getting this working for all site types and the different timezone possibilities.

[apU823]

Glad to help!

I guess this also helps explain why in the app when I go to "weekly view" it calls yesterday activity as "Today"

Does Tesla need to fix my timezone on their backend?

[jweier]

Hey all - I know I'm late to the party but I'm happy to test on a Solar only setup and I'm comfortable in Python so I should be able to troubleshoot that aspect. Lmk if I can help at all.

[youzer-name]

@mcbirse @jasonacox I checked my backups and the cron job ran, but still no success. I found some discrepancies between my backup script and the latest version here, and after making some updates I was able to backup and restore successfully. I added a line at the end of the script to copy the backup to a Windows Storage Space on my network with RAID, so I'm good even if my Pi's SSD dies.

Since I had the restored database to play with, I decided to test a theory for how I managed to delete the data. Typing "delete from http where time <= '2021-07-02' works fine. However if you've been jumping between different databases all day and mistakenly type "delete from http where DATE <= '2021-07-02' then you better have a working backup 😏

I'm pretty sure MySQL and MSSQL would react to a statement like that by deleting nothing and saying "DATE? What are you talking about? There is no date column here. Have more coffee, check your backups, and try again." (ok, maybe not the last part)

Yup - tested and got:

[mcbirse]

@youzer-name - wow, great find and shockingly bad of InfluxDB to do that! 😮

[jasonacox]

Wow @mcbirse @youzer-name ! +1. Thanks for the warning on that.

@jweier We welcome your help! A few things you can try. Go here and follow instructions to get the dashboard set up: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/solar-only

# Download 
git clone https://github.com/jasonacox/Powerwall-Dashboard.git

# Select Solar Only
cd Powerwall-Dashboard/tools/solar-only

# Run setup
./setup.sh

# Test Tesla Owner API
cd ~/Powerwall-Dashboard/tools/solar-only/tesla-history

Then follow instructions by @mcbirse here: https://github.com/jasonacox/Powerwall-Dashboard/issues/183#issuecomment-1455010845

[mcbirse]

@jweier - I missed your post - thanks! If you can test that would be most welcome. Some more data from another solar only site would be great.

I haven't seen any solar only site with what I would call a "normal" or consistent timezone configuration as yet.... (at least compared to Powerwall sites).

e.g. @apU823's site timezone is different to the timezone returned in the calendar history data request (which is in UTC), and @youzer-name's timezone is showing an offset only (which therefore won't recognise DST changes per his actual timezone).

Regardless, I will have an update to script ready in a day or two, which should address all of the issues discovered so far and should work with all these possible timezone variations.

[jweier]

@mcbirse - I did some testing tonight and was able to get the import working and pulling my data into InfluxDB. The only thing I had to change in your code was removing the following three lines because my site does not have a site_name attribute in the JSON response:

sitename = data['response']['site_name'] sitelist[siteid]['name'] = sitename print(f" Site name: {sitelist[siteid]['name']}")

It doesn't look like you actually do anything with the sitename variable so it didn't seem to break anything. Below is the JSON object returned for my site config:

Get SITE_CONFIG for Site ID 225218026XXXXXX { "response": { "id": "cd5ccfa7-6702-4d46-8e16-f4b5", "site_number": "STE20230106-XXXXXX", "installation_date": "2023-01-06T22:30:00Z", "user_settings": { "storm_mode_enabled": null, "powerwall_onboarding_settings_set": null, "powerwall_tesla_electric_interested_in": null, "sync_grid_alert_enabled": false, "breaker_alert_enabled": false }, "components": { "solar": true, "solar_type": "pv_panel", "battery": false, "grid": true, "backup": false, "gateway": "gateway_type_none", "load_meter": true, "tou_capable": false, "storm_mode_capable": false, "flex_energy_request_capable": false, "car_charging_data_supported": false, "off_grid_vehicle_charging_reserve_supported": false, "vehicle_charging_performance_view_enabled": false, "vehicle_charging_solar_offset_view_enabled": false, "battery_solar_offset_view_enabled": false, "energy_service_self_scheduling_enabled": true, "configurable": false, "grid_services_enabled": false, "inverters": [ { "device_id": "13defb85-e4eb-403c-b856-a4f2", "din": "1538000-00-F--GF22233", "is_active": true, "site_id": "cd5ccfa7-6702-4d46-8e16-f4b5" } ] }, "installation_time_zone": "America/New_York", "geolocation": { "latitude": 4X.XXXXXX, "longitude": -7X.XXXXXX }, "address": { "address_line1": "Street Name", "city": "City", "state": "State", "zip": "Zip", "country": "US" } } }

So, I'd consider this a success. I will look through Grafana to see if the data lines up. Just like you, I do think there is an issue with Timezone messing things up.

[jasonacox]

Nice work @jweier ! Would you mind posting a screenshot of your Grafana panel when you are able to get enough data to render?

@mcbirse Would it make sense to add -d type daemon flag to make the script run as a service? E.g. from weather411:

nextupdate = time.time()

# Time Loop to fetch solar data
while(running):
    currentts = time.time()
    # Is it time for an update?
    if currentts >= nextupdate:
        nextupdate = currentts + (60 * WAITMIN)
        fetch()

    time.sleep(5)

We could then containerize it and run it as a service.

[jweier]

It definitely looks like there is some sort of calculation/timing issue with the metrics. My numbers seem to be ~10% lower then the app. Here is a screenshot of the dashboard. I started with the simple dashboard and have been working to remove the Powerwall pieces.

[mcbirse]

@jweier - Thanks, I already have some updates that will address those issues. A lot of the initial site data was really for information purposes only, so I have changed this plus better handling of timezones I think.

Just busy this weekend, currently at Fully Charged Live Sydney, so will submit a PR with the updates later.

@jasonacox - will think about a daemon option as well. Perhaps after testing of my latest changes by a few people however.

[jweier]

@mcbirse - That's great to hear man. No pressure at all. Just lmk if/when I can help at all. Enjoy the show!

[mcbirse]

Enjoy the show!

@jweier - Fully Charged Live show in Sydney was great! Actually spotted myself in the background of one of the videos Jack posted on Twitter when he was doing the rounds, haha.

Just lmk if/when I can help at all.

Would love to get some debug output of the power history data for your solar only site, if possible please.

i.e. this would be good:

# Retrieve power history data for all of yesterday
python3 tesla-history.py --test --force --debug --yesterday

I'm wondering with solar only sites, does the below calculation need to be modified? Does a solar only site still report grid_power values?

# Calculate power usage values
home = d['solar_power'] + d['battery_power'] + d['grid_power']

Your graph above seems to indicate it does?

However, data posted previously shows zero values for grid_power, so I'm a bit confused and I need to work out what should be done here to correctly calculate home power usage (if possible) given the different values that are output for solar only sites.

[jweier]

Everything I've seen shows that solar only setups report grid usage as long as the Neurio sensor is in place and setup properly. Here is one of my outputs:

{ "timestamp": "2023-03-12T13:10:00Z", "solar_power": 364, "battery_power": 0, "grid_power": 231.92599792480468, "grid_services_power": 0, "generator_power": 0 }

The previous data could've lacked a Neurio sensor (I believe older installations didn't have them) or it could be configured incorrectly (pretty common). I'm able to pull the grid power for both the "current" usage via https://owner-api.teslamotors.com/api/1/energy_sites//live_status and historical usage via https://owner-api.teslamotors.com/api/1/energy_sites//calendar_history. There are more fields available if you use the "historical" data as opposed to the live data. That's how I calculate different fields with other things I've written. Here is an example output of the "historical" data:

{ "response": { "serial_number": "cd5ccfa7-6702-4d46-8e16-f4b5", "period": "day", "installation_time_zone": "America/New_York", "time_series": [ { "timestamp": "2023-03-03T01:00:00-05:00", "solar_energy_exported": 24630, "generator_energy_exported": 0, "grid_energy_imported": 4576, "grid_services_energy_imported": 0, "grid_services_energy_exported": 0, "grid_energy_exported_from_solar": 19761, "grid_energy_exported_from_generator": 0, "grid_energy_exported_from_battery": 0, "battery_energy_exported": 0, "battery_energy_imported_from_grid": 0, "battery_energy_imported_from_solar": 0, "battery_energy_imported_from_generator": 0, "consumer_energy_imported_from_grid": 4576, "consumer_energy_imported_from_solar": 4869, "consumer_energy_imported_from_battery": 0, "consumer_energy_imported_from_generator": 0 } ] } }

The catch is that you only get the totals for the day...so you can't get the granular 5m increments like you can with live data.

[apU823]

@mcbirse - i have a feeling that my Neurio sensors are all messed up. I'm trying to get Tesla to come out to fix but as you can appriciate, its taking them a while and probably do not expect anyone before end of the month :(

[mcbirse]

Thanks @apU823 - I was suspecting the same thing, given which I believe the script does not need adjusting regarding the calculations. Hope you manage to get Tesla to fix it for you soon!

[mcbirse]

Hi all, I have just submitted a PR with some further changes to the script.

Again, once merged would appreciate some feedback from testing! 🙏

Changes as below:

• Better handling of site data/info retrieval to prevent script exiting due to missing data (now only the "Site ID" is required)
• Revised timezone retrieval from calendar history data to mitigate against instances when no data is returned for current time
• Added support for "time_zone_offset" when "installation_time_zone" is not found
• The timezone used by history data will be shown in the output: "Loading daily history: [YYYY-MM-DD] (TZ or UTC offset)"
• Added new option to also search for backup reserve percent data gaps and set missing values to a specific percentage
• Previous update also removed "--ignoretz" option (no longer required)

@youzer-name - this update should work with timezone offsets as opposed to a timezone name - let us know how it goes when you have chance and behaviour for DST (it should translate correctly to your configured influxtz).

The script output should show something similar to below if grabbing the offset from your history data correctly (UTC-5, instead of a timezone name):

* Loading daily history: [2023-03-13] (UTC-5)

Just a note on the new option I added for reserve percentage. Not really related this current testing, but it was something I wanted myself. Maybe it could be useful for others too(?) so I decided to include it.

The new --reserve x option can be used to fill missing data gaps for the backup reserve percentage that was added recently.

If could be useful if your system was down, and you then used the tesla history script (and maybe weather history too) to fill the missing data.

You can tell the script to set the "backup reserve percent" to a specific value, i.e. I know mine was set to 20% for this period, so below I ran the script with option --reserve 20 to fill the missing data from when my system was down.

e.g. below I had run the tesla history script without the option, and the backup reserve percent history is incomplete:

If running the script with the additional new option e.g. --reserve 20 then it will fill the missing reserve percent as well:

Searching InfluxDB for data gaps (backup reserve percent)
* Found data gap: [2023-03-13 00:00:00+11:00] - [2023-03-13 21:30:00+11:00] (21:30:00s)
* Found data gap: [2023-03-13 23:57:00+11:00] - [2023-03-13 23:59:59+11:00] (0:02:59s)

Setting missing backup reserve percent history to '20'
* Creating reserve pct data: [2023-03-13 00:00:00+11:00] - [2023-03-13 21:29:59+11:00] (21:29:59s)
* Creating reserve pct data: [2023-03-13 23:58:00+11:00] - [2023-03-13 23:59:59+11:00] (0:01:59s)

Then I ran the weather history tool to fill in my missing weather data:

Useful to create a fairly complete looking Energy Usage panel I believe on those occasions when data was lost for whatever reason (and you are pedantic about it... like me).

[youzer-name]

The new version seems to mostly work, but I'm still having timezone issues.

As I write this I am in daylight saving time, America/New_York, (EDT). That is UTC -0400. I just did an import for 1 December 2018 to 31 December 2019.

Each line is showing "Loading daily history: [yyyy-mm-dd] (UTC-4).

I've had my solar-only since 2016, but it seems I can't pull any data older than 1 December 2018 from the API. I don't know if that is a limitation of the API or just something going on with my account.

Here is the EST to EDT transition in March 2019:

And here is the transition back in November:

On the spring leap forward, the chart for Sunday is still off by one hour even though DST had kicked in before the sun came up. The graph doesn't line up with the sun/moon data until Monday's chart. Everything continues to line up correctly through the summer.

For the autumn fall back, the chart for Sunday is lined up correctly, and the first day with the misaligned solar is Monday, even though the clocks changed on Sunday before sunrise.

This may not be worth tracking down since the daily and monthly totals will be correct. For my installation I don't have any home use data so it's all zeros when the sun is down and the solar will be on the correct day even if it is off by an hour. The script is writing a home value, but it is identical to the solar value. so I'll probably modify it to ignore the home value.

[mcbirse]

Thanks for testing @youzer-name

Each line is showing "Loading daily history: [yyyy-mm-dd] (UTC-4).

Okay, this interesting. Previously your debug output showed UTC-5, and I think I recall asking for some output from both DST & non-DST history periods, but it remained static at UTC-5? (offset -300)

However, now this seems to indicate the offset or timezone in the calendar history data may not be static, and could change? In the script, I am now getting the timezone from the calendar history data (not site, as it could be different per @apU823's data), and it fetches this just once, as of "now" only.

I'm going to have a rethink on how to do this. The only reason, really, is to ensure we get a full day of history from the API call to get the calendar history data. Maybe I need to check the offset on every single response and adjust if it changes. Annoying.

It's also possible that your data stored in the Tesla cloud is just wrong and when stored it did not account for DST changes!? Your site timezone is an offset after all, not a timezone name. And an offset holds no information about DST.

Just curious, has the site timezone that the script reports now changed to UTC-4? (i.e. debug output for SITE_CONFIG - the data you posted before showed "time_zone_offset": -300)

Being in Aus, your DST change dates are weird to me. But I did just look this up and realise you changed to EDT as of 12th March. I guess that's somewhat convenient for this testing!!

The script is writing a home value, but it is identical to the solar value. so I'll probably modify it to ignore the home value.

Correct. Hence my questions about whether we need to adjust this calculation.

# Calculate power usage values
home = d['solar_power'] + d['battery_power'] + d['grid_power']

If the grid_power value is always returned as zero, then home will be equal to solar_power, which is wrong. Obviously this is just a consideration for solar only sites, not battery sites.

And per @jweier's data, some solar only sites DO return grid_power if the Neurio sensors are in place.

One solution, as you mentioned, is to simply modify this in Grafana to ignore home.

But, I think I can make some modification to the script for solar only sites that don't return grid_power values. I will look into this as well.

I have some more updates to work on then. Thanks for your post, we will get there eventually.

[youzer-name]

@mcbirse - All of the 'retrieving data for gap:' lines show -05:00, but the 'loading daily history' lines show 'UTC-4'

The site info looks like this now:

----------------------------------------
      Site ID: xxxxxxxxxxxxxxxxxx
    Site type: Solar
    Site name: None
     Timezone: UTC-4
    Installed: 2016-11-07 20:00:00-04:00
  System time: 2023-03-15 08:48:29-04:00
----------------------------------------

The data looks like this when retrieving a standard time date today (today is daylight saving time)

Running for period: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)

Retrieving data for gap: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)
* Loading daily history: [2020-01-01] (UTC-4)
* Loading daily history: [2020-01-02] (UTC-4)

and it looks like this when retrieving a date when DST is active:

Running for period: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)

Retrieving data for gap: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)
* Loading daily history: [2020-07-01] (UTC-4)
* Loading daily history: [2020-07-02] (UTC-4)

Also, I noticed it is looking for data gaps in power usage. If it might be pulling only solar production numbers, should the script be modified to look for gaps in solar instead?