Import Tesla app historical data

[cstout1]

Is it possible to import historical data that is exported as CSV from the Tesla iOS app?

[jasonacox]

You could, but you are better off using the Tesla-History tool that @mcbirse built (see https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/tesla-history#tesla-history-import-tool). It pulls this data from the same cloud API points that the Tesla App uses. Would that work or is there another reason you want to use the CSV?

[cstout1]

The only reason I mentioned the CSV was because I had no idea the data was available in the API. Wow, this project has everything. Thank you and I'll get working on using that excellent import tool.

[jasonacox]

Awesome! Big credit to @mcbirse for this brilliance.

[mcbirse]

@cstout1 - Enjoy!

If you're a weather nerd as well and like the weather411 service and the data it overlays on the dashboard, I also wrote another tool you can use to import historical weather data which could complement the Tesla power history data that you import: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/weather-history#weather-history-import-tool

[youzer-name]

@mcbirse I noticed an oddity on the Tesla History import when I was filling in a gap from a network outage yesterday. At about 2100 local time on 2 Jan (0200 3 Jan UTC ) I ran the import manually using the --today flag. It tried to pull data for 3 Jan. I'm not sure if it's practical to use local time for the --yesterday and --today flags, but aren't the dates/times treated as local when you specify a start and end time?

[mcbirse]

@youzer-name - this may be more of an oddity with the Tesla cloud data rather than the script itself... it depends what timezone your calendar history data is actually stored in.

If it is different or offset to your current local timezone, then when the script attempts to get "today" for your local timezone, this may span more than 1 day of calendar history data when the stored timezone in the cloud is different - so it would be retrieving more than 1 day of cloud data.

Can you run the script with the following options and provide output to confirm? (remove personal data)

python tesla-history.py --today --force --test --debug

Example snippet of what I would be looking for to confirm. The timezone in the CALENDAR_HISTORY_DATA and timestamps can actually be different to your local timezone or site timezone sometimes.

      Site ID: 1234567890
    Site type: Powerwall x1
    Site name: Home
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2024-01-04 13:45:08+11:00
----------------------------------------
Running for period: [2024-01-04 00:00:00+11:00] - [2024-01-04 23:59:59+11:00] (23:59:59s)

Get timezone from CALENDAR_HISTORY_DATA
{
    "serial_number": "XXXXXXXXXXXXXXXXXXXXXXX",
    "installation_time_zone": "Australia/Sydney",
    "time_series": [
        {
            "timestamp": "2024-01-04T00:00:00+11:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 782.1428571428571,
            "grid_services_power": 0,
            "generator_power": 0
        },

The standard output from the script where is says "Loading daily history" should actually show the timezone of your calendar history data at the end of each line (I think - it's been a while since working on the script, and I have tried to block out the painful experience of and headaches caused by dealing with timezones!)

Retrieving data for gap: [2024-01-04 00:00:00+11:00] - [2024-01-04 13:49:00+11:00] (13:49:00s)
* Loading daily history: [2024-01-04] (Australia/Sydney)

[youzer-name]

This is what I get on the end running for "--today" at 23:04 local

     Site ID: xxxxxxxxxxxxxxxx
    Site type: Solar
    Site name: My Solar
     Timezone: UTC-05:00
    Installed: 2016-11-08 03:00:00-05:00
  System time: 2024-01-04 23:04:37-05:00
----------------------------------------
Running for period: [2024-01-05 00:00:00-05:00] - [2024-01-05 23:59:59-05:00] (23:59:59s)

ERROR: No data available for this date/time range

When I run for "--yesterday"

      Site ID: xxxxxxxxxxxxxxxxxxxx
    Site type: Solar
    Site name: My Solar
     Timezone: UTC-05:00
    Installed: 2016-11-08 03:00:00-05:00
  System time: 2024-01-04 23:13:30-05:00
----------------------------------------
Running for period: [2024-01-04 00:00:00-05:00] - [2024-01-05 23:59:59-05:00] (1 day, 23:59:59s)

Get timezone from CALENDAR_HISTORY_DATA
{
    "serial_number": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "time_zone_offset": -300,
    "time_series": [
        {
            "timestamp": "2024-01-04T00:00:00-05:00",
            "solar_power": 0,
            "battery_power": 0,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0

[cstout1]

The import was flawless. What's interesting is it even pulled all the "missing" Powerwall data I've been unable to see since February. Kind of incredible that all the data is actually being stored but there must be some iOS app bug that's preventing it from displaying properly. Thank you @mcbirse and @jasonacox !

[mcbirse]

@cstout1 - That's great to hear, glad it worked!

@youzer-name - Very confusing, not sure why this is happening for you.

The --today / --yesterday options just get the local system time from the python datetime.today() function to determine the period for "today" or "yesterday".

This should return a timezone naive datetime object, which the program will then convert to UTC internally (based on your local timezone as set in the tesla-history.conf file).

What does the below command output when you run it at e.g. 23:04 local time?

python3 -c "import datetime; print(datetime.datetime.today())"

What is the TZ value set to in your tesla-history.conf file?

[youzer-name]

@mcbirse - Running the python command at 22:18 local on 11 January, I get:

2024-01-12 03:18:48.804908

so, UTC.

"TZ=America/New_York" in tesla-history.conf.

[jasonacox]

Interesting! All my server report local time with that command. Do you have a timezone set on your system?

pi@piserver:~ $ date
Thu 11 Jan 19:38:45 PST 2024

pi@piserver:~ $ python3 -c "import datetime; print(datetime.datetime.today())"
2024-01-11 19:38:52.047109

pi@piserver:~ $ ls -l /etc/localtime     
lrwxrwxrwx 1 root root 39 Jul 15 08:43 /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles

[mcbirse]

Running the python command at 22:18 local on 11 January, I get:

2024-01-12 03:18:48.804908

so, UTC.

Ahhh. Well that explains that then. The program expects today() is returning your local time, not UTC.

I wonder what the below returns please?

python3 -c "import datetime; print(datetime.datetime.now(datetime.timezone.utc))"

It could perhaps be safer to use in all cases rather than making assumptions? (i.e. does it consistently return an aware datetime in UTC, that is correct... that's what we would need to achieve).

[youzer-name]

From the host OS (Raspberry Pi)

alan@raspberrypi4b:~ $ python3 -c "import datetime; print(datetime.datetime.now(datetime.timezone.utc))"
2024-01-16 15:30:20.808481+00:00

alan@raspberrypi4b:~ $ date
Tue 16 Jan 2024 10:31:13 AM EST

alan@raspberrypi4b:~ $ ls -l /etc/localtime  
lrwxrwxrwx 1 root root 36 Dec 17 19:07 /etc/localtime -> /usr/share/zoneinfo/America/New_York

From inside the Tesla-History docker container:

2024-01-16 15:36:18.970628+00:00

/app $ date
Tue Jan 16 15:36:23 UTC 2024

/app $ ls -l /etc/localtime
ls: /etc/localtime: No such file or directory

From my Powerwall-Dashboard/tools/tesla-history/tesla-history.conf file:

[InfluxDB]
... 
# Database name and timezone
DB = solaronly
TZ = America/New_York

[mcbirse]

Thanks @youzer-name - I assume initially when you mentioned the issue you were running the tesla-history script from within the container (e.g. using docker exec)?

Either way, this has highlighted an issue/bug with the script, since when running from within the container the "local time" is in UTC and the script assumes datetime.today() is returning the time in your actual timezone.

I'll make an update to the script to fix this at some point. It's not a major issue and the specific condition for it to occur would be executing the script from within the container and using either the "--today" and/or "--yesterday" options.

In the meantime, I think you can fix this immediately for your setup by doing the following.

• Edit your powerwall.yml or powerwall.extend.yml with contains the tesla-history service definition
• Add a TZ variable to the "environment" section for your timezone
• Recreate and start the container

e.g.

services:
    tesla-history:
        environment:
            - TZ=America/New_York

# Recreate and start containers
./compose-dash.sh up -d

Before the change (date shows UTC):

$ docker exec -it tesla-history date
Thu Jan 18 10:28:19 UTC 2024

After (date should show your correct timezone, not UTC):

$ docker exec -it tesla-history date
Thu Jan 18 21:29:29 AEDT 2024

This should fix the problem without needing to update the tesla-history script itself. I'll update the script later though, so this won't be required.