Incomplete data files during daylight saving (summer)

[semiligneous]

Hi @zigam - this works really well, exactly what I needed, thank you!

I am experiencing something strange though, whereby all data from now (May 31 2023) back to April 2nd 2023 is good - there are 289 lines in each file, so data every 5 minutes for 24 hours per day.

However, from April 1st back to October 2nd, there are only 13 lines per CSV, corresponding to data in the hours of 00:xx only (midnight to 1am).

I think this is related somehow to daylight saving time - because April 2nd was when New South Wales (Australia) went out of daylight saving (to standard time), and October 2nd was when the region went into daylight saving time (DST).

https://www.timeanddate.com/time/change/australia/sydney?year=2022 https://www.timeanddate.com/time/change/australia/sydney?year=2023

wc -l 2*.csv
     289 2022-09-27.csv
     289 2022-09-28.csv
     289 2022-09-29.csv
     289 2022-09-30.csv
     289 2022-10-01.csv
      13 2022-10-02.csv
      13 2022-10-03.csv
      13 2022-10-04.csv
...

      13 2023-03-29.csv
      13 2023-03-30.csv
      13 2023-03-31.csv
      13 2023-04-01.csv
     301 2023-04-02.csv
     289 2023-04-03.csv
     289 2023-04-04.csv
     289 2023-04-05.csv

Example short file:

cat 2023-04-01.csv
timestamp,solar_power,battery_power,grid_power,grid_services_power,generator_power,load_power
2023-04-02 00:00:00,0,0,453.14285714285717,0,0,453.14285714285717
2023-04-02 00:05:00,0,0,472,0,0,472
2023-04-02 00:10:00,0,0,414.7142857142857,0,0,414.7142857142857
2023-04-02 00:15:00,0,0,413.8333333333333,0,0,413.8333333333333
2023-04-02 00:20:00,0,0,438.14285714285717,0,0,438.14285714285717
2023-04-02 00:25:00,0,0,468,0,0,468
2023-04-02 00:30:00,0,0,460.5,0,0,460.5
2023-04-02 00:35:00,0,0,411.57142857142856,0,0,411.57142857142856
2023-04-02 00:40:00,0,0,357.57142857142856,0,0,357.57142857142856
2023-04-02 00:45:00,0,0,397.8333333333333,0,0,397.8333333333333
2023-04-02 00:50:00,0,0,435.14285714285717,0,0,435.14285714285717
2023-04-02 00:55:00,0,0,484.42857142857144,0,0,484.42857142857144

I am not sure specifically why or how daylight saving causes issues with the script, but is this something you think could be easily addressed?

Cheers

[zigam]

Thanks for the report! Timezones are hard :)

I pushed a fix which should address this issue, please give it another try. Note that you'll have to move the corrupt files out of the way so they're downloaded again. Before re-running the script, either delete the corrupt files (which will download just those files again), or delete the entire download directory for a full re-rerun.

If you're still having issues, please re-run with --debug and copy the output here so I can debug further.

[zigam]

Btw I just fixed another issue with retries which required a new dependency, so you'll also have to rerun:

pip install -r requirements.txt

[semiligneous]

Thanks! I’ll test both and let you know later today when I get a chance!

Appreciate the quick response :-)

----- Original message ----- From: "Ziga Mahkovec - notifications at github.com" @.> To: "zigam/tesla-solar-download - tesla-solar-download at noreply.github.com" @.> Cc: semiligneous @.>, "Author - author at noreply.github.com" @. @.***>> Subject: Re: [zigam/tesla-solar-download] Incomplete data files during daylight saving (summer) (Issue #1) Date: Thursday, 1 June 2023 08:27

Btw I just fixed another issue with retries which required a new dependency, so you'll also have to rerun:

pip install -r requirements.txt

— Reply to this email directly, view it on GitHub https://github.com/zigam/tesla-solar-download/issues/1#issuecomment-1571045133, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOYQN4O3M3YD44F7TDHVRITXI7ATPANCNFSM6AAAAAAYVPQF6I. You are receiving this because you authored the thread.Message ID: @.***>

[semiligneous]

@zigam - this now works perfectly. I'm very grateful for this simple tool and the fast response. Thanks!