search

jasonacox / Powerwall-Dashboard

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems
MIT License
300 stars 64 forks source link

Tesla Solar-Only Dashboard #183

Open jasonacox opened 1 year ago

jasonacox commented 1 year ago

There has been a lot of interest by Tesla Solar owners who don't have a Powerwall, to get a similar dashboard for their system. I'm opening this as a feature request to explore creating a derivative dashboard using a similar stack plus the tesla-history script developed by @mcbirse as a potential service to feed a solar-only dashboard.

Reference: Reddit Thread

Proposal

Add a Tesla-Solar-Only dashboard option to in the tools folder. Basically, a InfluxDB+Grafana+Import-Tool (optional +Weather411) stack. The Import-Tool could be converted to be a python service, for instance, that polls the Tesla API every 5m or so and stores it in InfluxDB. We could use the dashboard.json and remove the Powerwall specific panels.

We need someone who has a Solar-Only setup to validate.

mcbirse commented 1 year ago

@youzer-name - Thanks again for the info, I'll work on some more changes.

Also, I noticed it is looking for data gaps in power usage. If it might be pulling only solar production numbers, should the script be modified to look for gaps in solar instead?

No, it won't matter. Values for "home" are still going to be written to InfluxDB as zero's - so using this to identify the data gaps is okay.

jweier commented 1 year ago

@mcbirse - I'm still trying to dig into why the values don't quite line up. Before I dig too far into it, I was wondering if you have any initial thoughts? It looks like you're using get_calendar_history_data from TeslaPy to pull the historical data which is effectively the same way I'm calling the historical data when testing.

mcbirse commented 1 year ago

@jweier - This is usually a timezone mismatch somewhere. Maybe don't dig too far yet anyway, as I'll have another update to the script shortly that needs further testing, and I have changed the timezone handling again.

mcbirse commented 1 year ago

@jweier @youzer-name @apU823 - would appreciate some testing again once the latest changes have been merged.

For my installation I don't have any home use data so it's all zeros when the sun is down and the solar will be on the correct day even if it is off by an hour. The script is writing a home value, but it is identical to the solar value. so I'll probably modify it to ignore the home value.

@youzer-name - the above issue should be fixed. For solar only sites that don't return grid power values, home usage should now be set to zero instead of being identical to the solar value.

Also, @youzer-name & @jweier - not sure but maybe your timezone issues will be fixed. It depends on your history data though (may need some debug output), however the script should now identify if the timezone or offset changes in the history response and adjust accordingly.

You can see if it happens like below. I set my InfluxDB timezone to "America/Los_Angeles", however the script then detects the history data is in timezone "Australia/Sydney" and will switch to that and reload from the day it was up to.

Retrieving data for gap: [2023-03-19 06:13:00-07:00] - [2023-03-19 23:59:59-07:00] (17:46:59s)
* Loading daily history: [2023-03-19] (America/Los_Angeles)
* Loading daily history: [2023-03-20] (Australia/Sydney)

Effectively, this should mean any timezone/offset change while pulling the daily history data will be detected.

I can't test these changes easily on my system because all the timezones are in sync and I don't have a solar only site. Hopefully I have not made any breaking changes.

PS. You'll probably need to wipe old data imported before testing, or use a fresh test system with no existing data in InfluxDB.

apU823 commented 1 year ago

I did the git pull and it seems to have updated "tools/solar-only/tesla-history/tesla-history.py"

please see attached my outputs. Tesla seems have fixed my capturing my grid/solar production data but time zone appears to still be messed up. They did also fix that both my inventors were now showing up on my account (which they are now :D)

I have a new case open with Tesla to see how we can resolve it the time zone issue. I expect another week before resolution :(

more of a question for me but what is "grid_services_power" supposed to be/capture? is that supposed to be how much power is getting push to the grid? all those values show up as 0 for me but that could be because my monitors were not caputreing data correctly...

tesla-history.py --test --force --debug --yesterday 3.21.23.txt tesla-history.py --login --debug 3.21.23.txt

jweier commented 1 year ago

@mcbirse - I just wiped my data and imported yesterday's data to look at. The import worked great without issue and it appears that the time zone was picked up properly. Everything is showing the correct time in my Grafana dashboard.

The metrics themselves still seem to be a little off. My total solar for the day according to Tesla it was 42.9 but shows as 40.73 in Grafana. Same with Home usage, Tesla shows 19 and Grafana shows 16.41. I don't think it's time zone because solar generation should still be accurate even if it's off by a few hours since it's not generating anything overnight...It seems close enough to accurate that perhaps it's a rounding error somewhere?

@apU823 - I looked at my grid_services_power metric and it always seems to be 0. Not sure if it's used. Is yours anything but 0?

mcbirse commented 1 year ago

@apU823 & @jweier - thanks for the info again. Sounds more promising that it may be working better now. I will analyse this in detail later.

I believe "grid_services_power" will not be relevant for solar only sites, as I recall it is used for Powerwall sites and only during VPP events, to record the component of power used during the VPP.

mcbirse commented 1 year ago

Hi @youzer-name - just wondering if you'd had a chance to test the latest update? I'm hoping everything would be good now.

I'm going to add a daemon/background option to the script so we can run it as a service next.

youzer-name commented 1 year ago

@mcbirse Sorry I was busy with some other things and hadn't had time to test until today.

With the latest update, my data is off by an hour during standard time.

Running for January 1 (standard time)

python3 tesla-history-solar.py  --start "2020-01-01" --end "2020-01-02" --site xxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
Tesla account: xxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
      Site ID: xxxxxxxxxxxxxxxxx
    Site type: Solar
    Site name: None
     Timezone: UTC-4
    Installed: 2016-11-07 20:00:00-04:00
  System time: 2023-03-24 11:01:08-04:00
----------------------------------------
Running for period: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)

Retrieving data for gap: [2020-01-01 00:00:00-05:00] - [2020-01-02 00:00:00-05:00] (1 day, 0:00:00s)
* Loading daily history: [2020-01-01] (UTC-4)
* Loading daily history: [2020-01-02] (UTC-4)

Writing to InfluxDB
Updating InfluxDB

image

Running for July 1 (daylight saving time)

python3 tesla-history-solar.py  --start "2020-07-01" --end "2020-07-02" --site xxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
Tesla account: xxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
      Site ID: xxxxxxxxxxxxxxxxxxx
    Site type: Solar
    Site name: None
     Timezone: UTC-4
    Installed: 2016-11-07 20:00:00-04:00
  System time: 2023-03-24 11:03:37-04:00
----------------------------------------
Running for period: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)

Retrieving data for gap: [2020-07-01 00:00:00-04:00] - [2020-07-02 00:00:00-04:00] (1 day, 0:00:00s)
* Loading daily history: [2020-07-01] (UTC-4)
* Loading daily history: [2020-07-02] (UTC-4)

Writing to InfluxDB
Updating InfluxDB

image

mcbirse commented 1 year ago

Thanks @youzer-name

Okay, it appears the Tesla cloud history data always reports your timezone in the "current offset". And this is despite the fact you could be requesting history data for a different time period, where it should be a different offset due to DST.

When you ran this previously a few weeks ago (prior to 12th Mar), your offset was showing UTC-5 for all history data. Now it is UTC-4 because of your DST change.

image

Wow - this is pretty bad of Tesla. I wonder if we can make some assumptions for any sites that are reporting the timezone as an "offset" instead of a name, and I could shift the timezone based on the configured InfluxDB timezone instead. This would solve the problem.

Also wondering the following - are you able to check this?

What does the Tesla app show for your solar site? If you look at your current daily solar generation, then swipe back to before 12th Mar 2023 (before your DST change), does it look like the data is then off by an hour? I'm suspecting it might, which means even the app shows it incorrectly.

mcbirse commented 1 year ago

Hi @youzer-name - previously you uploaded some debug output from when you ran the script on 6th March for the period --start "2021-06-01 00:00:00" --end "2021-06-01 23:59:59"

Are you able to run the script with debug output, for the exact same period please? Below is a snippet from last time.

      Site ID: #################
    Site type: Solar
    Installed: 2016-11-07 19:00:00-05:00
  System time: 2023-03-06 18:12:26-05:00
----------------------------------------
Running for period: [2021-06-01 00:00:00-04:00] - [2021-06-01 23:59:59-04:00] (23:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2021-06-01 00:00:00-04:00] - [2021-06-01 23:59:59-04:00] (23:59:59s)

Retrieving data for gap: [2021-06-01 00:00:00-04:00] - [2021-06-01 23:59:59-04:00] (23:59:59s)
* Loading daily history: [2021-06-01]
{
    "serial_number": "####################",
    "time_zone_offset": -300,
    "time_series": [
        {
            "timestamp": "2021-06-01T00:00:00-05:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },

I need to see the output of this now since your switch to DST. Thanks.

youzer-name commented 1 year ago

Intersting.... downloading from the app, each line has the current UTC offset, but the time of day seems to be correct for the actual date.

Here is a part of the downloaded data showing the start and end of production for each day on 11, 12, and 13 March 2023:

03/11/23    
2023-03-11T06:00:00-04:00   0
2023-03-11T06:15:00-04:00   0
2023-03-11T06:30:00-04:00   0
2023-03-11T06:45:00-04:00   0
2023-03-11T07:00:00-04:00   0.1  <---
2023-03-11T07:15:00-04:00   0.2
2023-03-11T07:30:00-04:00   0.4
...
2023-03-11T17:15:00-04:00   0.3
2023-03-11T17:30:00-04:00   0.2
2023-03-11T17:45:00-04:00   0.2  <---
2023-03-11T18:00:00-04:00   0
2023-03-11T18:15:00-04:00   0

|
|
|

03/12/23    
2023-03-12T06:00:00-04:00   0
2023-03-12T06:15:00-04:00   0
2023-03-12T06:30:00-04:00   0.1  <---
2023-03-12T06:45:00-04:00   0.2
2023-03-12T07:00:00-04:00   0.2
2023-03-12T07:15:00-04:00   0.5
...

2023-03-12T17:15:00-04:00   0.2
2023-03-12T17:30:00-04:00   0.1
2023-03-12T17:45:00-04:00   0.1  <---
2023-03-12T18:00:00-04:00   0
2023-03-12T18:15:00-04:00   0
2023-03-12T20:00:00-04:00   0

|
|
|

03/13/23    
2023-03-13T06:00:00-04:00   0
2023-03-13T06:15:00-04:00   0
2023-03-13T06:30:00-04:00   0
2023-03-13T06:45:00-04:00   0
2023-03-13T07:00:00-04:00   0
2023-03-13T07:15:00-04:00   0
2023-03-13T07:30:00-04:00   0
2023-03-13T07:45:00-04:00   0
2023-03-13T08:00:00-04:00   0.1  <---
2023-03-13T08:15:00-04:00   0.2
2023-03-13T08:30:00-04:00   0.2
...
2023-03-13T18:00:00-04:00   0.4
2023-03-13T18:15:00-04:00   0.2
2023-03-13T18:30:00-04:00   0.1  <---
2023-03-13T18:45:00-04:00   0
2023-03-13T19:00:00-04:00   0
2023-03-13T19:15:00-04:00   0

I marked the start and end of production each day.

On Saturday production ran from 0700- 1745 (ignoring the TZ offset and treating it as local time). Sunrise was 06:27 EST, sunset 18:11 EST

On Sunday it ran from 0630 - 1745. Sunrise was 0726 EDT, sunset at 1912 EDT

On Monday it ran from 0800 - 1830. Sunrise was 07:24 EDT, sunset at 1913 EDT.

The times in the downloaded files match what the app displays on the solar-only graphs.

If I haven't confused myself, that means it was correct the day before the change (ignoring the reported offset and treating the times as local), it was incorrect on the day of the change, and it was correct on the day after the change. I think this lines up with the earlier screenshots I posted where the data didn't start to line up or mis-align until the day after the switch.

I attached the original files so you can see another anomaly in there... the missing hour on March 12th is at 1900 hours... not 0200 hours. It actually seems to be making the DST adjustment at midnight UTC on Monday, when it should have made it as 0200 local on Sunday.

I also attached the requested debug output.

Solar data Downloads.zip

debug-output.txt

mcbirse commented 1 year ago

Hi @youzer-name - thanks for the detailed information. I've made a small change to the script again.

Due to the incorrect behaviour of Tesla cloud history data when the timezone is reported as an offset, the offset will now be replaced with the configured InfluxDB timezone instead.

I think this may fix the issue with the data being misaligned due to DST changes (mostly), however cannot test this myself.

I expect there will still be a slight issue though, in that it will still be incorrect on the day of the change. This is due to the behaviour you noted where Tesla is not adjusting for DST at the correct time.

But before & after the day of a DST change, I am hoping the data will be aligned correctly now when imported. 🙏

We have no other examples of sites that are returning the timezone as an offset, so it is hard to know if Tesla behaves this way for all of them or not. I'm going to assume so at the moment.

Jot18 commented 1 year ago

I just tried to run Tesla-history under solar-only and had no luck. Getting error "No Tesla Energy sites found."

mcbirse commented 1 year ago

@Jot18 - Could you run the script with debug turned on please, and post the output? Run the below command.

# Run tesla-history login with debug output
python3 tesla-history.py --login --debug

You may want remove personal information from the output, such as email, site id & name, address, geolocation, and serial numbers, etc.

Jot18 commented 1 year ago

@Jot18 - Could you run the script with debug turned on please, and post the output? Run the below command.

# Run tesla-history login with debug output
python3 tesla-history.py --login --debug

You may want remove personal information from the output, such as email, site id & name, address, geolocation, and serial numbers, etc.

Success this time! I still attached the log. Time to play with it. Let me know if you want to run any further cmds. debug log.txt

Jot18 commented 1 year ago

Thank you guys for the awesome work!!! The data is little bit off when compared to tesla app. Screenshot (1)

jasonacox commented 1 year ago

This is awesome! Thanks @Jot18 ! How far off is the data?

Also, anyone with solar only, if you come up with an awesome dashboard we should host on the project, please submit as a PR or post here. :)

Jot18 commented 1 year ago

This is awesome! Thanks @Jot18 ! How far off is the data?

Also, anyone with solar only, if you come up with an awesome dashboard we should host on the project, please submit as a PR or post here. :)

image

jasonacox commented 1 year ago

I've seen differences monitoring local Powerwall system data vs Tesla data, but the local calculations are usually in line with what the Utility company computes so I haven't explored it more. I usually rationalize that by assuming we are sampling the data at a higher frequency (fidelity) locally than what Tesla pulls for their app. But in your case, that is essentially the same data the App would use (I assume).

I wonder what we would see if we pulled the raw data into a spreadsheet and ran the calculation rather than using Grafana's integral() calculation. In other words, is this an issue with Grafana/InfluxDB or with the data we are getting?

BuongiornoTexas commented 1 year ago

I touched on possible issues with grafana integral vs Tesla data in #87 - A whole lot of things have delayed my investigation, but I'm still hopeful of doing some more validation on tesla cumulative data vs utility reporting in the not too distant. The first step is to get ToU properly sorted out so that I can match my utilities billing methods with confidence (in progress now), and then I can test alternatives for usage calculations to see what matches utility reports best.

The one thing that did come out of the work I've done so far is that tesla cloud data can and will decide to do it's own thing with the underlying data. See long bullet point para at https://tesla-api.timdorr.com/energy-products/energy/history#get-api-1-energy_sites-site_id-calendar_history. The key point was this conclusion:

There is no clear alignment between the Tesla cloud API data and the data available from the local Powerwall API.

And because the Tesla app relies on the tesla cloud data, I would tend to treat it as the least reliable data source.

mcbirse commented 1 year ago

Comparing the values with the Tesla app is going to give you different results.

The tesla history script pulls per x minute daily history only (calendar history data call kind='power'). This returns either per 5 minute data, or I have noticed sometimes only per 15 minute data is available (noted in @youzer-name's debug output, his solar only site is returning per 15 minute data, whereas others are returning per 5 minute data). Per 15 minute data is going to be even less accurate when using integral calculations in Grafana for the Power Meters calculations.

By comparison, when viewing history in the Tesla app, the "daily", "weekly", "monthly" values etc. that are shown, are likely using the calendar history call kind='energy' with the selected period specified by e.g. period='day/week/month/etc'.

In my testing, calls using kind='energy' and period='day/week/month/etc' return a different value to if you had pulled the per 5/15 minute data and tried to calculate it from that alone. Tesla stores this data differently which is one reason why comparing with the app will show different values. It may be more accurate, although this is also questionable and as noted in the link provided by @BuongiornoTexas results are sometimes erratic, which has been my experience also.

Powerwall-Dashboard has been designed to store per x minute data so we can display power usage graphs with some reasonable level of granularity, hence why the script will only pull the per x minute daily history, and the cumulative Power Meters values are calculated from that. As touched on by Jason, the higher frequency (fidelity) of this data, the better the accuracy, however we are limited by the frequency of history data available from the Tesla cloud.

Jot18 commented 1 year ago

The final set up running Raspberry PI 3.

Thanks @jasonacox

SolarDB

jasonacox commented 1 year ago

That looks awesome @Jot18 ! Great job.

Have you tried this dashboard.json yet? 

https://raw.githubusercontent.com/jasonacox/Powerwall-Dashboard/main/tools/solar-only/dashboard.json

I bet it would look great in that orientation.

Also, how often are you running the tesla-history.py import script?

Jot18 commented 1 year ago

That looks awesome @Jot18 ! Great job.

Have you tried this dashboard.json yet?

https://raw.githubusercontent.com/jasonacox/Powerwall-Dashboard/main/tools/solar-only/dashboard.json

I bet it would look great in that orientation.

Also, how often are you running the tesla-history.py import script?

I been trying to configure auto data feed with crontab and haven't been successful yet. Any recommendations on how to set schedule for every 30 mins?

Here is what I tried. It runs fine in terminal but not crontab. I am mainly focused on --today. image

jasonacox commented 1 year ago

I suspect the issue is that it isn't able to load the Tesla token file when it runs. There are many ways to fix that. I would probably create a cron-tesla.sh file in the /home/tesla/Powerwall-Dashboard/ folder that would contain:

#!/bin/bash
echo "Fetching Tesla Solar Data..."

# move to right directory
cd /home/tesla/Powerwall-Dashboard/tools/solar-only/tesla-history

# pull data from tesla
python3 tesla-history.py --today

Make it executable:

chmod +x cront-tesla.sh

And then create a cron job to run every 5 minutes:

*/5 * * * * /home/tesla/Powerwall-Dashboard/cron-tesla.sh > /dev/null 2>&1
Jot18 commented 1 year ago

I suspect the issue is that it isn't able to load the Tesla token file when it runs. There are many ways to fix that. I would probably create a cron-tesla.sh file in the /home/tesla/Powerwall-Dashboard/ folder that would contain:

#!/bin/bash
echo "Fetching Tesla Solar Data..."

# move to right directory
cd /home/tesla/Powerwall-Dashboard/tools/solar-only/tesla-history

# pull data from tesla
python3 tesla-history.py --today

Make it executable:

chmod +x cront-tesla.sh

And then create a cron job to run every 5 minutes:

*/5 * * * * /home/tesla/Powerwall-Dashboard/cron-tesla.sh > /dev/null 2>&1

This worked. 😃😃

mcbirse commented 1 year ago

@Jot18 - great news. This should work running from cron for e.g. every 5 minutes, keeping your graphs up to date.

I still intend on adding a background option to the script, thereby eliminating the need for users to set up cron jobs / scripts. Just haven't had time to work on this yet.

For your cron script, you may want to add another cron entry to run once just after midnight, using the --yesterday option. This would ensure the last 5 mins of data for the previous day is filled in if you're worried about that.

Alternatively, and maybe an easier option, change your current script that executes every 5 minutes and add both options in the one command, e.g.:

# pull data from tesla
python3 tesla-history.py --today --yesterday

It won't hurt anything, it just means on every execution of the script it would search back for missing data for both the current day, and previous day.

Jot18 commented 1 year ago

@Jot18 - great news. This should work running from cron for e.g. every 5 minutes, keeping your graphs up to date.

I still intend on adding a background option to the script, thereby eliminating the need for users to set up cron jobs / scripts. Just haven't had time to work on this yet.

For your cron script, you may want to add another cron entry to run once just after midnight, using the --yesterday option. This would ensure the last 5 mins of data for the previous day is filled in if you're worried about that.

Alternatively, and maybe an easier option, change your current script that executes every 5 minutes and add both options in the one command, e.g.:

# pull data from tesla
python3 tesla-history.py --today --yesterday

It won't hurt anything, it just means on every execution of the script it would search back for missing data for both the current day, and previous day.

Please do include this in the next update. I just didn't do 5 mins and instead did 1 hr because I'm not looking at the data every hour and don't want to overload Tesla API by doing it every 5 mins...wouldn't too many requests result in a ban? And yes I will add that line. Didn't think thoroughly Abt data loss for yesterday. I had 0 knowledge about Linux/python. My friend helped me with initial setup now been running successfully :)

mcbirse commented 1 year ago

Sure, if you don't need to see the data as frequently it makes sense to have it run every hour or half hour instead.

don't want to overload Tesla API by doing it every 5 mins...wouldn't too many requests result in a ban?

It's possible, but probably unlikely.

Jot18 commented 1 year ago

I haven't been able to get today's data. Any idea what's going on? I got this error yesterday like once or twice but today nothing has worked. Data is there until midnight and not after.

Edit: It looks like it starts working after 5 pm and onwards for some odd reason....

MicrosoftTeams-image (63)

jasonacox commented 1 year ago

ERROR: No data returned for this date/time range

Were you able to see the data in the Tesla app? Perhaps the data was delayed (your system lost WiFi or connectivity to Tesla for a while) or Tesla had internal issues with processing that data?

Jot18 commented 1 year ago

ERROR: No data returned for this date/time range

Were you able to see the data in the Tesla app? Perhaps the data was delayed (your system lost WiFi or connectivity to Tesla for a while) or Tesla had internal issues with processing that data?

The app showed data throughout the day. It worked fine until 11:59 pm....Let's see tomm. I doubt system lost wifi because there are multiple other things in garage on wifi which were working fine.

mcbirse commented 1 year ago

Edit: It looks like it starts working after 5 pm and onwards for some odd reason....

This could be timezone issues again. Is it consistently not working until 5pm local time each day?

Can you add another argument to the script --debug so we can get some debug output please?

# pull data from tesla
python3 tesla-history.py --today --yesterday --debug
Jot18 commented 1 year ago

Edit: It looks like it starts working after 5 pm and onwards for some odd reason....

This could be timezone issues again. Is it consistently not working until 5pm local time each day?

Can you add another argument to the script --debug so we can get some debug output please?

# pull data from tesla
python3 tesla-history.py --today --yesterday --debug

yes, it does look like it doesn't work until 5 pm. Today is second day in a row. I ran file at 16:59 and it didn't work. It worked fine past 17:00. It works fine until12-1 am. Terminalv2

TodayDatauntil1am

TodayDatauntil5pm

DebugOutput4-6.txt

mcbirse commented 1 year ago

Thanks @Jot18 - very helpful info!

I can tell from the screenshot of the output, that your daily history data must be stored in UTC.

The debug output you uploaded unfortunately is only for when it didn't load the daily data, so I can't confirm for sure - but this is what it looks like from the script output, as you can see it switch timezone to UTC for the daily data.

So this is a bug in the script. I'll work on a fix for that.

Jot18 commented 1 year ago

Thanks @Jot18 - very helpful info!

I can tell from the screenshot of the output, that your daily history data must be stored in UTC.

The debug output you uploaded unfortunately is only for when it didn't load the daily data, so I can't confirm for sure - but this is what it looks like from the script output, as you can see it switch timezone to UTC for the daily data.

So this is a bug in the script. I'll work on a fix for that.

DebugV2.txt

Jot18 commented 1 year ago

I just tried to run Tesla-history under solar-only and had no luck. Getting error "No Tesla Energy sites found."

I suspect issue was same here as I was trying to run it just before 5 pm. @mcbirse

mcbirse commented 1 year ago

Thanks - confirmed UTC for the daily history. In the first data retrieval it attempts to get daily history, but for your local timezone, and only retrieves the data up to 7am UTC.

The script is supposed to detect & switch timezones to get the full day of data. This is not working correctly when no data is returned however, due to the different timezone.

* Loading daily history: [2023-04-06] (America/Los_Angeles)
{
    "installation_time_zone": "UTC",
    "time_series": [
        {
            "timestamp": "2023-04-07T00:00:00Z",
            "solar_power": 1702.6666666666667,
            "battery_power": 0,
            "grid_power": -951.360009765625,
            "grid_services_power": 0,
            "generator_power": 0
        },
.....
        {
            "timestamp": "2023-04-07T06:55:00Z",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        }
    ]
}
* Loading daily history: [2023-04-07] (UTC)
{
    "installation_time_zone": "UTC",
    "time_series": [
        {
            "timestamp": "2023-04-07T00:00:00Z",
            "solar_power": 1702.6666666666667,
            "battery_power": 0,
            "grid_power": -951.360009765625,
            "grid_services_power": 0,
            "generator_power": 0
        },
.....
        {
            "timestamp": "2023-04-07T23:55:00Z",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        }
mcbirse commented 1 year ago

Hi @Jot18 - I have made some more changes to the script which I'm hoping will solve the issue with getting history for "today".

Please pull the latest version when you can and give that a try. Fingers crossed it should be good now! 🙏

Jot18 commented 1 year ago

Hi @Jot18 - I have made some more changes to the script which I'm hoping will solve the issue with getting history for "today".

Please pull the latest version when you can and give that a try. Fingers crossed it should be good now! 🙏

It seems to working now. Thank you so much. @mcbirse I'll keep you updated if it stops working.

gpieroni1 commented 1 year ago

Hey all just wanted to say thanks for this great tool for us solar only users! I followed the instructions and got my dashboard up and running pretty quickly.

mcbirse commented 1 year ago

Awesome to hear @gpieroni1!. There will be some further improvements soon that I am currently working on, which should allow the script to be containerized (i.e. using docker) so that it will continually poll for updates instead of having to set it up manually to run periodically as a cron job.

mcbirse commented 1 year ago

Hi all - I have completed further changes to the tesla-history script to enable it to run as a docker container. This will be useful for solar-only setups, as you will no longer need to create your own cron jobs to keep your solar data updated.

Below is a summary / explanation of all the changes. Please see further below for the upgrade / install instructions.

Daemon mode changes

Setup changes

Docker container notes

Miscellaneous changes

TODO

Upgrading an existing install

If you have already installed the solar-only dashboard previously by running "setup.sh", you can upgrade and install the tesla-history docker container as follows. Make sure to stop/remove any cron jobs you had setup, as these will no longer be required.

Pull updates:

# Pull latest changes
git pull

Run solar-only setup:

# Setup or upgrade solar-only dashboard
cd tools/solar-only
./setup.sh

Example of upgrade process is below.

NOTE: A new section has been added to the setup script to configure the Tesla-History script & docker container. It is recommended for upgrades to answer "Y" to "Overwrite existing settings?" if prompted, to re-create the tesla-history.conf file.

$ cd tools/solar-only
$ ./setup.sh
Tesla Solar Dashboard (v2.9.8) - SETUP
-----------------------------------------
Timezone (leave blank for Australia/Sydney)
Enter Timezone:

Using Australia/Sydney timezone...
-----------------------------------------

Weather Data Setup
-----------------------------------------
Weather data from OpenWeatherMap can be added to your Powerwall Dashboard
graphs.  This requires that you set up a free acccount with OpenWeatherMap
and enter the API Key during this setup process.

Do you wish to setup Weather Data? [y/N]
No problem. If you change your mind, run weather.sh to setup later.

Tesla-History Setup
-----------------------------------------
Checking/installing python modules...

Existing config found 'tesla-history.conf'

Overwrite existing settings? [y/N] y

Creating config file 'tesla-history.conf'

Tesla Account Setup
-------------------
Email address: yourname@example.com

Config saved to 'tesla-history.conf'

If you have multiple Tesla Energy sites, the setup process will detect this and prompt you to enter the site id to use, and this will be saved to the config, e.g.:

----------------------------------------
Tesla account: yourname@example.com
----------------------------------------
      Site ID: 111111111111
    Site type: Powerwall x1
    Site name: Tesla Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2023-05-16 21:43:08+10:00
----------------------------------------
      Site ID: 222222222222
    Site type: Solar
    Site name: Tesla Solar
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2023-05-16 21:43:08+10:00
----------------------------------------
Multiple Tesla Energy sites found...
Enter Site ID: 222222222222
----------------------------------------
Running Docker Compose...
Pulling tesla-history (jasonacox/tesla-history:0.1.0)...
0.1.0: Pulling from jasonacox/tesla-history
.
.
Status: Downloaded newer image for jasonacox/tesla-history:0.1.0
influxdb is up-to-date
grafana is up-to-date
Creating tesla-history ...
Creating tesla-history ... done
-----------------------------------------
Waiting for InfluxDB to start...
 up!
Setup InfluxDB Data...
Executing single run query run-once-2.9.2.sql file...
Fetching local weather...
weather411
Restarting tesla-history...
tesla-history
------------------[ Final Setup Instructions ]-----------------

Open Grafana at http://localhost:9000/ ... use admin/admin for login.

Follow these instructions for *Grafana Setup*:

* From 'Configuration\Data Sources' add 'InfluxDB' database with:
  - Name: 'InfluxDB'
  - URL: 'http://influxdb:8086'
  - Database: 'powerwall'
  - Min time interval: '5s'
  - Click "Save & test" button

* From 'Configuration\Data Sources' add 'Sun and Moon' database with:
  - Name: 'Sun and Moon'
  - Enter your latitude and longitude (tool here: https://bit.ly/3wYNaI1 )
  - Click "Save & test" button

* From 'Dashboard\Browse', select 'New/Import', and upload 'dashboard.json' from
/home/admin/Powerwall-Dashboard/tools/solar-only

The setup process for a fresh install will be very similar to the above.

After install/upgrade, if you wish to check the tesla-history docker container is working, you can check the logs with command docker logs tesla-history

$ docker logs tesla-history
tesla-history Server [0.1.0]
* Configuration Loaded [/var/lib/tesla-history/tesla-history.conf]
 + Server - Wait: 5, Hist: 60, Log: no, Debug: no, Test: no
 + Tesla - User: yourname@example.com, Auth: [/var/lib/tesla-history/tesla-history.auth]
 + InfluxDB - Host: influxdb, Port: 8086, DB: powerwall, Timezone: Australia/Sydney
----------------------------------------
Tesla account: yourname@example.com
----------------------------------------
      Site ID: 222222222222
    Site type: Solar
    Site name: Tesla Solar
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2023-05-16 21:05:49+10:00
----------------------------------------
* Server Started
 + Retrieving 60 minutes of history every 5 minutes

If running okay, you should see solar data added to your dashboard (by default, past hour of data would be imported every 5 minutes).

If there are any errors, these will be logged. For example, if your Internet connection was down, you would see an error similar to below in the docker log.

ERROR: Failed to retrieve history data - HTTPSConnectionPool(host='owner-api.teslamotors.com', port=443): Max retries exceeded with url: /api/1/energy_sites/222222222222/calendar_history?kind=power&period=day&end_date=2023-05-17T23%3A59%3A59%2B10%3A00 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7fe0a43df760>: Failed to resolve 'owner-api.teslamotors.com' ([Errno -3] Try again)"))

The script has been designed to recover from transient faults, so will continue to retry on some errors at the next poll period, and once recovered fill in missed data up to the poll history period (60 mins by default).

If you wish to customise settings, edit the tesla-history.conf file and then restart the docker container with command docker restart tesla-history

An example of the config is shown below. A daemon options section has been added which is used by the docker container.

[Tesla]
# Tesla Account e-mail address and Auth token file
USER = yourname@example.com
AUTH = tesla-history.auth

[InfluxDB]
# InfluxDB server settings
HOST = localhost
PORT = 8086
# Auth (leave blank if not used)
USER =
PASS =
# Database name and timezone
DB = powerwall
TZ = Australia/Sydney

[daemon]
; Config options when running as a daemon (i.e. docker container)
# Minutes to wait between poll requests
WAIT = 5
# Minutes of history to retrieve for each poll request
HIST = 60
# Enable log output for each poll request
LOG = no
# Enable debug output (print raw responses from Tesla cloud)
DEBUG = no
# Enable test mode (disable writing to InfluxDB)
TEST = no
# If multiple Tesla Energy sites exist, uncomment below and enter Site ID
# SITE = 123456789

Don't forget to remove cron jobs that may be running the tesla-history script periodically, as these will no longer be necessary once the tesla-history docker container is installed and running.

Feedback welcome and please let us know if you find any issues!

Jot18 commented 1 year ago

@mcbirse That's great news! Can you send me the zip file I'll test it on another PC and let you know if I face any issues.

edit: I see your repository is public. I'll test it and update.

Jot18 commented 1 year ago

@mcbirse Do I need to do anything else after the install? It doesn't not seem to be updating...When I check docker logs tesla-history it shows error: "Error: No such container: tesla-history". When I run py teslahistory.py --today manually it works fine.

mcbirse commented 1 year ago

@Jot18 - the tesla-history docker container hasn't been pushed to Jason's Docker Hub as yet. Once that is done though, you will be able to test and can git pull the latest changes from Powerwall-Dashboard rather than my repository also.

jasonacox commented 1 year ago

The container has been pushed now. Let us know how it works!

Thanks for this great update @mcbirse !

mcbirse commented 1 year ago

The container has been pushed now.

Thanks @jasonacox !

I have set up another VM host myself for some longer term testing of this as well. It doesn't matter that I do not have a solar-only site really, and I can compare graphs with my main system that gets data from the gateway to check for any issues or anomalies.

One thing I have not done as yet is check for memory leaks, since the script now runs permanently in the docker container. Any suggestions for best way to measure/monitor this? At the moment I am just looking at the VM host memory usage itself, and seeing if it increases over time, however that is not limited to the specific container but shows the whole system memory usage.

@jasonacox - I just recalled you checked the pyPowerwall container for memory leaks before, and showed some Grafana graphs of the memory usage over time (found the post here) - how did you set this up?

Jot18 commented 1 year ago

@mcbirse I did fresh install with latest update. I still can't get it to pull 'docker logs tesla-history'. Any other way I can check the same? I don't see tesla-history image in docker.

edit: when running .\setup.sh in solar-only. I get an error "python3: command not found" then the setup closes.

image