zabuldon / teslajsonpy

Apache License 2.0
56 stars 62 forks source link

Generalize API and remove home-assistant specific architecture #24

Closed alandtse closed 2 years ago

alandtse commented 5 years ago

Right now, this library creates entities specifically for HA to consume. This has two downsides that I can see.

  1. Many functions are not intuitive as they assume some knowledge of the HA architecture. It took me a while to figure out that controller.list_vehicles() returned a list of objects for HA and not just a list of vehicles. It also means people not using HA likely won't use this library which runs against the philosophy from HA requiring the library be put in pypi to allow others to reuse.
  2. Feature updates to HA for Tesla require changes to both teslajsonpy and HA. For example, if someone wants to make an additional sensor for charging rate, they will need to modify the sensor in this library and then make an equivalent edit to process the new data. This impacted my ability to make changes in HA and probably is impacting others who want to tweak it. If this library is just intended to allow access to the Tesla API, then teslajsonpy changes will only be needed if new endpoints/commands are created.

I think we should move the HA related architecture into the HA component. I'd be happy to take that project on.

From an initial review, controller, connection, and exception make sense to me in this library. Everything based on vehicle probably should go into HA.

@zabuldon, this is your project to start, do you have any objections to this rearchitecture? I'll plan to start on doing that unless you object.

zabuldon commented 5 years ago

@alandtse i'm not sure about moving into HA as an initial step. In my opinion is much better to just generalize API and keep backward capability while we are working for a refactoring/extending.

I would like to suggest next plan:

  1. Make "master" stable with current interfaces and merge only functionality not affecting cpabilities
  2. Re-work architecture
  3. Add tests/CI
  4. Release 2-3 version with backward capability layer (while we are still working for HA adoption)
  5. Replace master with re-architected version as soon as we get ready with all changes in HA.

What do you think?

alandtse commented 5 years ago

That seems fine. I would suggest that breaking changes (e.g., #23) all be saved for step 5.

I'll get started on it.

OverloadUT commented 5 years ago

Just tossing in a note here that I wholeheartedly agree with this plan. When I went to work on this component a few months back I ran in to the exact same confusion, where there was a bunch of HA-specific logic sitting in this library.

alandtse commented 5 years ago

Just to track my thinking, I think I'm going to adopt a lot of structure from simplisafe-python which seems quite mature (e.g., tests, documentation, linting) and also has an async based HA component that seems quite developed per the integration quality scale. For example, they were one of the first to rely on the config_flow mechanism.

llluis commented 4 years ago

Hi folks. I just wanted to chime in and let you know I'm available to help and contribute.

I took the standard Tesla component and loaded into my setup as custom with a few modifications to suit my needs, for instance, I implemented several services to use in automations (and disabled "should_poll" to let my car sleep). But as @alandtse mentioned, the HA code in here prevents me to further modify my setup as needed.

What's the current status on the generalization? How do you setup a dev lib for local testing?

alandtse commented 4 years ago

Not much has happened on generalizing the code. The only major change was the migration to async. The biggest issue with moving any code into HA out of this library is HA has a very long review process. The easiest path is probably to move the HA code into its own subdirectory.

We currently use pipenv to set up the dev environment (read the README). To test it with HA you just need to use pip to tell where the library is installed and copy files appropriately. For example, in hassio you can do the following command and then copy the modified files into your config directory.

pip3 install --target /config --no-dependencies teslajsonpy
alandtse commented 4 years ago

I took the standard Tesla component and loaded into my setup as custom with a few modifications to suit my needs, for instance, I implemented several services to use in automations (and disabled "should_poll" to let my car sleep). But as @alandtse mentioned, the HA code in here prevents me to further modify my setup as needed.

As a note, should_poll doesn't need to be modified as there is an update switch to disable polling of a vehicle. Also, the polling in the component will never wake up a vehicle. Only actual commands sent to the API will ever try to wake up a vehicle. In theory it's possible a vehicle won't go to sleep because of polling but in my experience on a standard 300 second polling interval my vehicles have been sleeping. Of course if someone can confirm that, that would be helpful.

llluis commented 4 years ago

Thanks for the update @alandtse! I saw the async stuff and the WIP for the config-flow. I like the idea to split this lib and move HA stuff to another subdirectory. It would allow to have the base lib clean and HA stuff grouped together.

I changed the should_poll to easy test the lib. Indeed, if the switch is off, the lib stops polling the vehicle. If the car is asleep it won't be woken. Even when restarting HA (and the switch ON).

My ultimate goal is to get rid of TeslaFi, so, my polling interval could be as short as 1min. So, some intelligence would be implemented to dynamically increase interval to 10-15 min to allow the car to sleep.

alandtse commented 4 years ago

One thing to note, there is a separate library that is started as a general purpose library. They originally were sync using requests but it appears they're now on aiohttp. It powers the tesla custom component.

Megabytemb commented 2 years ago

Ok, So we're coming up to a year later and still no progress.

I'm keen to implement some of the new features in Home Assistant like the Update entity, and I'm really in dev mood, So I'm actually pretty keen to tackle this.

The architecture of Home Assistant has changed a lot since this library was created, the the concept of devices and update coordinators. I've believe a pretty good job was done shoe-horning that functionality into the Tesla library, but Its a bit all over the place now, with the concept of a "device" existing in both libraries, and it kinda being hard to share updates and data between entities.

Would anyone be opposed against a 3.0 style branch on https://github.com/alandtse/tesla where we more-or-less start from scratch on the integration to clean everything up and use teslajsonpy purely as a API layer? This would allow us to really take advantage of all of Home Assistants new helpers, and get everything inline with their new development style and architecture.

alandtse commented 2 years ago

No opposition if you want to do it.

Megabytemb commented 2 years ago

righto, this is very much a WIP, but it gives you an idea of the style i'm going for. Have a look at the Update and Select entities

https://github.com/megabytemb/tesla/commit/cc7be783c28f54131e0251181e78f020ee1eeecb

purcell-lab commented 2 years ago

@megabytemb could I ask that you include not only vehicles but the other tesla products that are available via teslajsonpy; vehicles and solar are currently supported in this library, powerwall isn't but is available via the same API.

Some details are available here: https://github.com/alandtse/tesla/issues/79

Megabytemb commented 2 years ago

That's a bit hard, cause I don't own a powerwall, so it will be hard to debug.

But I'll do my best once I get the Car done.

That OK?

purcell-lab commented 2 years ago

That's a bit hard, cause I don't own a powerwall, so it will be hard to debug.

I'm happy to assist with debug.

Megabytemb commented 2 years ago

I've been thinking about the request to properly support Tesla Energy Sites in the component rewrite.

I just want to re-affirm, i do believe its worth while, and I think I'm writing it in a way that I could easily also support energy sites. But I'll be clear that Car support takes priority right now. Once i get Cars released, We'll work on Energy Sites.

Megabytemb commented 2 years ago

So I'm by no means feature complete, but I've made a Sizeable dent. Still have all the tests to write, and clean up a lot of code comments, But I'm interested in any feedback people have on the approach and code style.

I've also added in some new entities like Number and Update entities, and added some extra selects and buttons for config and diagnostics.

https://github.com/Megabytemb/tesla/tree/tesla-rewrite

alandtse commented 2 years ago

Thanks. Please submit a PR so the diff is easy to see and comments can be made about specific portions of the code. You can flag it DRAFT until you're ready. In terms of code style, if it's using black that's enough for me. I'll have more comments once I see both this and the changes in tesla. To be honest, I'd expect the changes here to be the removal of the /homeassistant/ folder and relevant tests which are they more or less moved/integrated into tesla.

crackers8199 commented 2 years ago

i've got both car and powerwall and am happy to help debug if needed as well. i just found this integration tonight while looking for a way to set operation mode on my powerwalls, will be installing it to check out the car side tonight.

Megabytemb commented 2 years ago

Thanks. Please submit a PR so the diff is easy to see and comments can be made about specific portions of the code. You can flag it DRAFT until you're ready. In terms of code style, if it's using black that's enough for me. I'll have more comments once I see both this and the changes in tesla. To be honest, I'd expect the changes here to be the removal of the /homeassistant/ folder and relevant tests which are they more or less moved/integrated into tesla.

Done - https://github.com/alandtse/tesla/pull/216

crackers8199 commented 2 years ago

just going to reference this here as well: as is, powerwall API commands with numeric values don't work with home assistant via teslajsonpy for some reason (and i haven't been able to figure out why). i'm able to set values via string params without an issue, but as soon as i try to use numeric params everything blows up...

see here for details: https://github.com/zabuldon/teslajsonpy/issues/316

@Megabytemb any ideas what could be causing this or how to fix it?

shred86 commented 2 years ago

i've got both car and powerwall and am happy to help debug if needed as well. i just found this integration tonight while looking for a way to set operation mode on my powerwalls, will be installing it to check out the car side tonight.

@crackers8199 I've been working on the rewrite which should support Powerwalls as well (just power sensors for now). If you have the ability to test it out, that would be great. Most of the discussion has been here but if you download the tesla-rewrite branch from repo, you should be able to drop in custom_components/tesla_custom to your HA install and it should run. I don't have Powerwalls so that part has not been tested yet (just solar w/ Tesla inverter, no poweralls).

crackers8199 commented 2 years ago

@shred86 i'll try to give it a shot this weekend!

purcell-lab commented 2 years ago

download the tesla-rewrite branch from repo, you should be able to drop in custom_components/tesla_custom to your HA install and it should run

So do I need both your updated tesla_custom & updated teslajsonpy to see powerwall?

I had updated the pointer to the later in manifest.json, but didn't see any new sensors.

shred86 commented 2 years ago

@purcell-lab I'm kind of in version control nightmare right now haha. I've been working on the rewrite so please use this version of the Tesla Custom Integration. The manifest.json is already updated to point to my other forked branch for teslajsonpy. The only thing I've noticed is you may have to do a pip uninstall teslajsonpy first in your HA container or host (depending on how you have HA installed)... for some reason it wasn't downloading my forked version. You should see solar, grid and load power sensors, as well as battery percentage... assuming I didn't screw it up. 😆

purcell-lab commented 2 years ago

@purcell-lab I'm kind of in version control nightmare right now haha.

Too many moving parts.

I tried to set up your link as a custom repo in HACS, but it still downloaded the same version.

I then downloaded your fork as a zip file and copied the files to tesla_custom/ but that wouldn't load. I'll try again and forward the error messages.

shred86 commented 2 years ago

Yeah unfortunately I don’t think you can setup this repo as a custom repository. I know this is obvious but just to make sure, when you download the repo, make sure you’re going into custom_components/tesla_custom and copying all those files over into HA’s config/custom_components/tesla_custom.

purcell-lab commented 2 years ago

Yeah unfortunately I don’t think you can setup this repo as a custom repository. I know this is obvious but just to make sure, when you download the repo, make sure you’re going into custom_components/tesla_custom and copying all those files over into HA’s config/custom_components/tesla_custom.

When I copy all your files over into custom_components/tesla_custom I get the following error message after I reboot:

This error originated from a custom integration.

Logger: homeassistant.config_entries
Source: custom_components/tesla_custom/__init__.py:149
Integration: Tesla Custom Integration (documentation, issues)
First occurred: 9:27:50 AM (1 occurrences)
Last logged: 9:27:50 AM

Error setting up entry mark@purcell.id.au for tesla_custom
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 357, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/config/custom_components/tesla_custom/__init__.py", line 149, in async_setup_entry
    result = await controller.connect(
TypeError: Controller.connect() got an unexpected keyword argument 'skip_add'

homeassistant.log:

2022-08-21 09:27:50.036 DEBUG (MainThread) [teslajsonpy.connection] Connecting with existing access token
2022-08-21 09:27:50.036 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry mark@purcell.id.au for tesla_custom
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 357, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/config/custom_components/tesla_custom/__init__.py", line 149, in async_setup_entry
    result = await controller.connect(
TypeError: Controller.connect() got an unexpected keyword argument 'skip_add'
2022-08-21 09:27:50.835 ERROR (MainThread) [homeassistant.components.sensor] Error adding entities for domain sensor with platform mobile_app
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 428, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 673, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 776, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 532, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 570, in _async_write_ha_state
    state = self._stringify_state(available)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 538, in _stringify_state
    if (state := self.state) is None:
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 388, in state
    value = self.native_value
  File "/usr/src/homeassistant/homeassistant/components/mobile_app/sensor.py", line 113, in native_value
    and (timestamp := dt_util.parse_datetime(state)) is not None
  File "/usr/src/homeassistant/homeassistant/util/dt.py", line 185, in parse_datetime
    return ciso8601.parse_datetime(dt_str)
TypeError: argument 1 must be str, not datetime.datetime
2022-08-21 09:27:50.849 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up mobile_app platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 289, in _async_setup_platform
    await asyncio.gather(*pending)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 428, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 673, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 776, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 532, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 570, in _async_write_ha_state
    state = self._stringify_state(available)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 538, in _stringify_state
    if (state := self.state) is None:
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 388, in state
    value = self.native_value
  File "/usr/src/homeassistant/homeassistant/components/mobile_app/sensor.py", line 113, in native_value
    and (timestamp := dt_util.parse_datetime(state)) is not None
  File "/usr/src/homeassistant/homeassistant/util/dt.py", line 185, in parse_datetime
    return ciso8601.parse_datetime(dt_str)
TypeError: argument 1 must be str, not datetime.datetime
shred86 commented 2 years ago

It looks like it didn't download the teslajsonpy from my repo, which is required for the changes. If you can get access to where HA is installed, you can do a pip uninstall teslajsonpy, then restart HA and it will download it. I need to figure out a way to force it to download from the git repo every time.

purcell-lab commented 2 years ago

It looks like it didn't download the teslajsonpy from my repo, which is required for the changes. If you can get access to where HA is installed, you can do a pip uninstall teslajsonpy, then restart HA and it will download it. I need to figure out a way to force it to download from the git repo every time.

Thanks, that has now loaded and I have additional controls over the car, like charging current numbers and the like:

image

A lot of sensors have been renamed, breaking change? sensor.duka_battery_sensor -> sensor.duka_battery

Number of entities has gone down from 41 to 37.

image

Nothing seen about the powerwall :-(

grep tesla home-assistant.log
2022-08-21 09:49:23.743 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration tesla_custom which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2022-08-21 09:49:23.749 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration tesla_gateway which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2022-08-21 09:50:17.792 DEBUG (MainThread) [teslajsonpy.connection] Connecting with existing access token
2022-08-21 09:50:17.795 DEBUG (MainThread) [teslajsonpy.controller] 399 endpoints loaded
2022-08-21 09:50:17.795 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 7:54:19
2022-08-21 09:50:17.795 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/vehicles {}
2022-08-21 09:50:19.429 DEBUG (MainThread) [teslajsonpy.connection] 200: {"response":[{"id":XXX,"vehicle_id":XXX,"vin":"XXX","display_name":"Duka","option_codes":"AD15,AF00,APFB,APH4,AU3P,BC3R,BT37,CDM0,CH06,COAU,DRRH,DV4W,FC3P,FG31,FM3P,GLFR,HL31,HM30,ID3W,IL31,LTPB,MDL3,MR30,PMNG,PC31,REAP,RF3G,RS3H,S3PB,SA3P,SC04,STCP,SU3C,T3M3,TM00,TW00,UT3P,W32D,WR00,ZINV,MI01,PL31,SLR1,ST30,BG32,I38M,OSSB,AUF1,RSF1,ILF1,FGF1,CPF1,HP30,PT00","color":null,"access_type":"DRIVER","tokens":["XXX","XXX"],"state":"online","in_service":false,"id_s":"XXX","calendar_enabled":true,"api_version":42,"backseat_token":null,"backseat_token_updated_at":null}],"count":1}
2022-08-21 09:50:19.429 DEBUG (MainThread) [teslajsonpy.controller] 98201: Changing car_online from {} to True
2022-08-21 09:50:19.429 DEBUG (MainThread) [teslajsonpy.controller] 98201: Resetting last_wake_up_time to: 1661039419
2022-08-21 09:50:19.429 DEBUG (MainThread) [teslajsonpy.controller] 98201: Resetting last_parked_timestamp to: 1661039419 shift_state None
2022-08-21 09:50:19.429 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 7:54:17
2022-08-21 09:50:19.430 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/products {}
2022-08-21 09:50:19.822 DEBUG (MainThread) [teslajsonpy.connection] 200: {"response":[{"id":XXX,"user_id":XXX,"vehicle_id":XXX,"vin":"XXX","display_name":"Duka","option_codes":"AD15,AF00,APFB,APH4,AU3P,BC3R,BT37,CDM0,CH06,COAU,DRRH,DV4W,FC3P,FG31,FM3P,GLFR,HL31,HM30,ID3W,IL31,LTPB,MDL3,MR30,PMNG,PC31,REAP,RF3G,RS3H,S3PB,SA3P,SC04,STCP,SU3C,T3M3,TM00,TW00,UT3P,W32D,WR00,ZINV,MI01,PL31,SLR1,ST30,BG32,I38M,OSSB,AUF1,RSF1,ILF1,FGF1,CPF1,HP30,PT00","color":null,"access_type":"DRIVER","tokens":["XXX","XXX"],"state":"online","in_service":false,"id_s":"XXX","calendar_enabled":true,"api_version":42,"backseat_token":null,"backseat_token_updated_at":null,"vehicle_config":{"aux_park_lamps":"Eu","badge_version":0,"can_accept_navigation_requests":true,"can_actuate_trunks":true,"car_special_type":"base","car_type":"model3","charge_port_type":"CCS","dashcam_clip_save_supported":true,"default_charge_to_max":false,"driver_assist":"TeslaAP3","ece_restrictions":false,"efficiency_package":"Default","eu_vehicle":true,"exterior_color":"MidnightSilver","exterior_trim":"Chrome","exterior_trim_override":"Chrome","has_air_suspension":false,"has_ludicrous_mode":false,"has_seat_cooling":false,"headlamp_type":"Premium","interior_trim_type":"Black","key_version":2,"motorized_charge_port":true,"paint_color_override":"19,20,22,0.8,0.04","performance_package":"Performance","plg":false,"pws":false,"rear_drive_unit":"PM216MOSFET","rear_seat_heaters":1,"rear_seat_type":0,"rhd":true,"roof_color":"RoofColorGlass","seat_type":null,"spoiler_type":"Passive","sun_roof_installed":null,"supports_qr_pairing":false,"third_row_seats":"None","timestamp":1661039419535,"trim_badging":"p74d","use_range_badging":true,"utc_offset":36000,"webcam_supported":false,"wheel_type":"Stiletto20DarkSquare"},"command_signing":"allowed"}],"count":1}
2022-08-21 09:50:19.823 DEBUG (MainThread) [teslajsonpy.controller] Get vehicles. Force: False Time: 1 Interval 60
2022-08-21 09:50:19.823 DEBUG (MainThread) [teslajsonpy.controller] 98201: online. Polling policy: normal. Update state: normal. Since last park: 1. Since last wake_up: 1. Idle interval: 600. shift_state: None sentry: None climate: None, charging: None 
2022-08-21 09:50:19.835 DEBUG (MainThread) [teslajsonpy.controller] 98201: Updating VEHICLE_DATA
2022-08-21 09:50:19.836 DEBUG (MainThread) [teslajsonpy.connection] Token expiration in 7:54:17
2022-08-21 09:50:19.836 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/vehicles/XXX/vehicle_data {}
2022-08-21 09:50:20.438 DEBUG (MainThread) [teslajsonpy.connection] 200: {"response":{"id":XXX,"user_id":XXX,"vehicle_id":XXX,"vin":"XXX","display_name":"Duka","option_codes":"AD15,AF00,APFB,APH4,AU3P,BC3R,BT37,CDM0,CH06,COAU,DRRH,DV4W,FC3P,FG31,FM3P,GLFR,HL31,HM30,ID3W,IL31,LTPB,MDL3,MR30,PMNG,PC31,REAP,RF3G,RS3H,S3PB,SA3P,SC04,STCP,SU3C,T3M3,TM00,TW00,UT3P,W32D,WR00,ZINV,MI01,PL31,SLR1,ST30,BG32,I38M,OSSB,AUF1,RSF1,ILF1,FGF1,CPF1,HP30,PT00","color":null,"access_type":"DRIVER","tokens":["XXX","XXX"],"state":"online","in_service":false,"id_s":"XXX","calendar_enabled":true,"api_version":42,"backseat_token":null,"backseat_token_updated_at":null,"charge_state":{"battery_heater_on":false,"battery_level":81,"battery_range":232.17,"charge_amps":11,"charge_current_request":11,"charge_current_request_max":16,"charge_enable_request":true,"charge_energy_added":8.17,"charge_limit_soc":90,"charge_limit_soc_max":100,"charge_limit_soc_min":50,"charge_limit_soc_std":90,"charge_miles_added_ideal":33.5,"charge_miles_added_rated":33.5,"charge_port_cold_weather_mode":false,"charge_port_color":"<invalid>","charge_port_door_open":true,"charge_port_latch":"Engaged","charge_rate":30.6,"charge_to_max_range":false,"charger_actual_current":11,"charger_phases":2,"charger_pilot_current":16,"charger_power":8,"charger_voltage":241,"charging_state":"Charging","conn_charge_cable":"IEC","est_battery_range":173.19,"fast_charger_brand":"<invalid>","fast_charger_present":false,"fast_charger_type":"ACSingleWireCAN","ideal_battery_range":232.17,"managed_charging_active":false,"managed_charging_start_time":null,"managed_charging_user_canceled":false,"max_range_charge_counter":0,"minutes_to_full_charge":50,"not_enough_power_to_heat":null,"off_peak_charging_enabled":false,"off_peak_charging_times":"weekdays","off_peak_hours_end_time":330,"preconditioning_enabled":false,"preconditioning_times":"all_week","scheduled_charging_mode":"Off","scheduled_charging_pending":false,"scheduled_charging_start_time":null,"scheduled_charging_start_time_app":543,"scheduled_departure_time":1656618300,"scheduled_departure_time_minutes":345,"supercharger_session_trip_planner":false,"time_to_full_charge":0.83,"timestamp":1661039420224,"trip_charging":false,"usable_battery_level":81,"user_charge_enable_request":null},"climate_state":{"allow_cabin_overheat_protection":true,"auto_seat_climate_left":true,"auto_seat_climate_right":true,"battery_heater":false,"battery_heater_no_power":null,"cabin_overheat_protection":"On","cabin_overheat_protection_actively_cooling":false,"climate_keeper_mode":"off","defrost_mode":0,"driver_temp_setting":20.0,"fan_status":0,"hvac_auto_request":"On","inside_temp":18.0,"is_auto_conditioning_on":false,"is_climate_on":false,"is_front_defroster_on":false,"is_preconditioning":false,"is_rear_defroster_on":false,"left_temp_direction":613,"max_avail_temp":28.0,"min_avail_temp":15.0,"outside_temp":20.0,"passenger_temp_setting":20.0,"remote_heater_control_enabled":false,"right_temp_direction":613,"seat_heater_left":0,"seat_heater_rear_center":0,"seat_heater_rear_left":0,"seat_heater_rear_right":0,"seat_heater_right":0,"side_mirror_heaters":false,"supports_fan_only_cabin_overheat_protection":true,"timestamp":1661039420224,"wiper_blade_heater":false},"drive_state":{"gps_as_of":1661033409,"heading":5,"latitude":-26.417307,"longitude":153.0768,"native_latitude":-26.417307,"native_location_supported":1,"native_longitude":153.0768,"native_type":"wgs","power":-8,"shift_state":null,"speed":null,"timestamp":1661039420225},"gui_settings":{"gui_24_hour_time":true,"gui_charge_rate_units":"kW","gui_distance_units":"km/hr","gui_range_display":"Rated","gui_temperature_units":"C","show_range_units":false,"timestamp":1661039420225},"vehicle_config":{"aux_park_lamps":"Eu","badge_version":0,"can_accept_navigation_requests":true,"can_actuate_trunks":true,"car_special_type":"base","car_type":"model3","charge_port_type":"CCS","dashcam_clip_save_supported":true,"default_charge_to_max":false,"driver_assist":"TeslaAP3","ece_restrictions":false,"efficiency_package":"Default","eu_vehicle":true,"exterior_color":"MidnightSilver","exterior_trim":"Chrome","exterior_trim_override":"Chrome","has_air_suspension":false,"has_ludicrous_mode":false,"has_seat_cooling":false,"headlamp_type":"Premium","interior_trim_type":"Black","key_version":2,"motorized_charge_port":true,"paint_color_override":"19,20,22,0.8,0.04","performance_package":"Performance","plg":false,"pws":false,"rear_drive_unit":"PM216MOSFET","rear_seat_heaters":1,"rear_seat_type":0,"rhd":true,"roof_color":"RoofColorGlass","seat_type":null,"spoiler_type":"Passive","sun_roof_installed":null,"supports_qr_pairing":false,"third_row_seats":"None","timestamp":1661039420225,"trim_badging":"p74d","use_range_badging":true,"utc_offset":36000,"webcam_supported":false,"wheel_type":"Stiletto20DarkSquare"},"vehicle_state":{"api_version":42,"autopark_state_v2":"standby","autopark_style":"standard","calendar_supported":true,"car_version":"2022.20.8 8f941dcd0ba7","center_display_state":0,"dashcam_clip_save_available":false,"dashcam_state":"Unavailable","df":0,"dr":0,"fd_window":0,"feature_bitmask":"9ff,0","fp_window":0,"ft":0,"is_user_present":false,"last_autopark_error":"no_error","locked":false,"media_state":{"remote_control_enabled":true},"notifications_supported":true,"odometer":27129.566503,"parsed_calendar_supported":true,"pf":0,"pr":0,"rd_window":0,"remote_start":false,"remote_start_enabled":true,"remote_start_supported":true,"rp_window":0,"rt":0,"santa_mode":0,"sentry_mode":false,"sentry_mode_available":true,"service_mode":false,"service_mode_plus":false,"smart_summon_available":true,"software_update":{"download_perc":0,"expected_duration_sec":2700,"install_perc":1,"status":"","version":" "},"speed_limit_mode":{"active":false,"current_limit_mph":50.0,"max_limit_mph":90,"min_limit_mph":50.0,"pin_code_set":true},"summon_standby_mode_enabled":true,"timestamp":1661039420224,"tpms_pressure_fl":null,"tpms_pressure_fr":null,"tpms_pressure_rl":null,"tpms_pressure_rr":null,"valet_mode":false,"vehicle_name":"Duka","vehicle_self_test_progress":0,"vehicle_self_test_requested":false,"webcam_available":false}}}
2022-08-21 09:50:20.445 DEBUG (MainThread) [custom_components.tesla_custom] Connected to the Tesla API
2022-08-21 09:50:20.446 DEBUG (MainThread) [custom_components.tesla_custom] Running controller.update()
2022-08-21 09:50:20.446 DEBUG (MainThread) [teslajsonpy.controller] Get vehicles. Force: False Time: 1 Interval 60
2022-08-21 09:50:20.446 DEBUG (MainThread) [teslajsonpy.controller] 98201: online. Polling policy: normal. Update state: normal. Since last park: 1. Since last wake_up: 1. Idle interval: 600. shift_state: None sentry: False climate: False, charging: Charging 
2022-08-21 09:50:20.446 DEBUG (MainThread) [teslajsonpy.controller] 98201: Skipping update with state online. Polling: True. Last update: 0 ago. Last parked: 1 ago. Last wake_up 1 ago. 
2022-08-21 09:50:20.446 DEBUG (MainThread) [custom_components.tesla_custom] Finished fetching tesla_custom data in 0.001 seconds (success: True)
2022-08-21 09:50:20.545 DEBUG (MainThread) [custom_components.tesla_custom] Running controller.update()
2022-08-21 09:50:20.545 DEBUG (MainThread) [teslajsonpy.controller] Get vehicles. Force: False Time: 2 Interval 60
2022-08-21 09:50:20.545 DEBUG (MainThread) [teslajsonpy.controller] 98201: online. Polling policy: normal. Update state: normal. Since last park: 2. Since last wake_up: 2. Idle interval: 600. shift_state: None sentry: False climate: False, charging: Charging 
2022-08-21 09:50:20.545 DEBUG (MainThread) [teslajsonpy.controller] 98201: Skipping update with state online. Polling: True. Last update: 1 ago. Last parked: 2 ago. Last wake_up 2 ago. 
2022-08-21 09:50:20.545 DEBUG (MainThread) [custom_components.tesla_custom] Finished fetching tesla_custom data in 0.001 seconds (success: True)
2022-08-21 09:50:31.182 DEBUG (MainThread) [custom_components.tesla_custom] Running controller.update()
shred86 commented 2 years ago

Ah I think I know why.. give me ten minutes and I'll push a new update.

@purcell-lab I just pushed a new update to the custom integration and teslajsonpy. If you can redownload the Tesla Custom Integration and follow the same steps (pip uninstall teslajsonpy again), let me know if that works!

Also to answer your question, there will be breaking changes unfortunately. As a part of the rewrite, I'm trying to conform to HA standards as much as I can. I want to get as much as we can aligned now to prevent future breaking changes.

purcell-lab commented 2 years ago

@purcell-lab I just pushed a new update to the custom integration and teslajsonpy. If you can redownload the Tesla Custom Integration and follow the same steps (pip uninstall teslajsonpy again), let me know if that works!

Back to skip_add issue. Your updated manifest.json no longer points to your teslajsonpy fork is that deliberate?

https://github.com/shred86/tesla/blob/ba648bb4239d1b70c2e56eb5c98fd05ee176e9e4/custom_components/tesla_custom/manifest.json#L8

shred86 commented 2 years ago

Ahhh, it was supposed to be for me only based on how I'm working on it but I forgot to change it back. Sorry about that, here's the line:

"requirements": ["git+https://github.com/shred86/teslajsonpy.git@rewrite-support#teslajsonpy==2.5.0"],

purcell-lab commented 2 years ago

Got it. Still no powerwall.

I'm a bit suspicious about this line from my logs above which doesn't have the powerwall product:

2022-08-21 09:50:19.430 DEBUG (MainThread) [teslajsonpy.connection] get: https://owner-api.teslamotors.com/api/1/products {}
2022-08-21 09:50:19.822 DEBUG (MainThread) [teslajsonpy.connection] 200: {"response":[{"id":XXX,"user_id":XXX,"vehicle_id":XXX,"vin":"XXX","display_name":"Duka","option_codes":"AD15,AF00,APFB,APH4,AU3P,BC3R,BT37,CDM0,CH06,COAU,DRRH,DV4W,FC3P,FG31,FM3P,GLFR,HL31,HM30,ID3W,IL31,LTPB,MDL3,MR30,PMNG,PC31,REAP,RF3G,RS3H,S3PB,SA3P,SC04,STCP,SU3C,T3M3,TM00,TW00,UT3P,W32D,WR00,ZINV,MI01,PL31,SLR1,ST30,BG32,I38M,OSSB,AUF1,RSF1,ILF1,FGF1,CPF1,HP30,PT00","color":null,"access_type":"DRIVER","tokens":["XXX","XXX"],"state":"online","in_service":false,"id_s":"XXX","calendar_enabled":true,"api_version":42,"backseat_token":null,"backseat_token_updated_at":null,"vehicle_config":{"aux_park_lamps":"Eu","badge_version":0,"can_accept_navigation_requests":true,"can_actuate_trunks":true,"car_special_type":"base","car_type":"model3","charge_port_type":"CCS","dashcam_clip_save_supported":true,"default_charge_to_max":false,"driver_assist":"TeslaAP3","ece_restrictions":false,"efficiency_package":"Default","eu_vehicle":true,"exterior_color":"MidnightSilver","exterior_trim":"Chrome","exterior_trim_override":"Chrome","has_air_suspension":false,"has_ludicrous_mode":false,"has_seat_cooling":false,"headlamp_type":"Premium","interior_trim_type":"Black","key_version":2,"motorized_charge_port":true,"paint_color_override":"19,20,22,0.8,0.04","performance_package":"Performance","plg":false,"pws":false,"rear_drive_unit":"PM216MOSFET","rear_seat_heaters":1,"rear_seat_type":0,"rhd":true,"roof_color":"RoofColorGlass","seat_type":null,"spoiler_type":"Passive","sun_roof_installed":null,"supports_qr_pairing":false,"third_row_seats":"None","timestamp":1661039419535,"trim_badging":"p74d","use_range_badging":true,"utc_offset":36000,"webcam_supported":false,"wheel_type":"Stiletto20DarkSquare"},"command_signing":"allowed"}],"count":1}

In contrast to

$ python3 ./cli.py.1 -e mark@purcell.id.au -l
2022-08-21 10:58:54,503 - root - INFO - 2 product(s), 2 selected
Product 0:
{
    "id": XXX,
    "vehicle_id": XXX,
    "vin": "XXX",
    "display_name": "Duka",
    "option_codes": "AD15,AF00,APFB,APH4,AU3P,BC3R,BT37,CDM0,CH06,COAU,DRRH,DV4W,FC3P,FG31,FM3P,GLFR,HL31,HM30,ID3W,IL31,LTPB,MDL3,MR30,PMNG,PC31,REAP,RF3G,RS3H,S3PB,SA3P,SC04,STCP,SU3C,T3M3,TM00,TW00,UT3P,W32D,WR00,ZINV,MI01,PL31,SLR1,ST30,BG32,I38M,OSSB,AUF1,RSF1,ILF1,FGF1,CPF1,HP30,PT00",
    "color": null,
    "access_type": "OWNER",
    "tokens": [
        "XXX",
        "XXX"
    ],
    "state": "online",
    "in_service": false,
    "id_s": "XXX",
    "calendar_enabled": true,
    "api_version": 42,
    "backseat_token": null,
    "backseat_token_updated_at": null
}
Product 1:
{
    "energy_site_id": XXX,
    "resource_type": "battery",
    "site_name": "Home Energy Gateway",
    "id": "XXX",
    "gateway_id": "XXX",
    "asset_site_id": "XXX",
    "energy_left": 2159.157894736842,
    "total_pack_energy": 14036,
    "percentage_charged": 15.383000104993174,
    "battery_type": "ac_powerwall",
    "backup_capable": true,
    "battery_power": 1780,
    "storm_mode_enabled": true,
    "powerwall_onboarding_settings_set": true,
    "sync_grid_alert_enabled": true,
    "breaker_alert_enabled": true,
    "components": {
        "battery": true,
        "battery_type": "ac_powerwall",
        "solar": true,
        "solar_type": "pv_panel",
        "grid": true,
        "load_meter": true,
        "market_type": "residential"
    }
}
shred86 commented 2 years ago

I'm looking into it now. I'm just filtering by "resource_type" when we get the product list. If it's either "solar" or "battery", it should work. I'll post an update here once I figure it out. The other thing I'm wondering is if it actually downloaded the updated version of teslajsonpy from my repo.

Are you able to execute this from the terminal of your HA install:

pip show teslajsonpy

You should see a "Location:" which should be /usr/local/python/lib/python3.10/site-packages

cd /usr/local/python/lib/python3.10/site-packages/teslajsonpy cat controller.py (or use nano)

Scroll down to line 561 and I just want to make sure it says:

if p.get(RESOURCE_TYPE) == RESOURCE_TYPE_SOLAR or p.get(RESOURCE_TYPE) == RESOURCE_TYPE_BATTERY

purcell-lab commented 2 years ago

Scroll down to line 561 and I just want to make sure it says:

if p.get(RESOURCE_TYPE) == RESOURCE_TYPE_SOLAR or p.get(RESOURCE_TYPE) == RESOURCE_TYPE_BATTERY

All good here, it has that line inside my docker.

I'm wondering rather than hijacking this thread further, should we switch to discord or something?

shred86 commented 2 years ago

Yep, I'm on the HA server, username just shred.

shred86 commented 2 years ago

At a bit of a loss on this one. @alandtse I'm curious if you have any ideas but basically when @purcell-lab is using teslajsonpy to hit the PRODUCT_LIST endpoint, he only gets a JSON response with a list and one dictionary, his car. When he uses TeslaPy, hitting the same endpoint PRODUCT_LIST and he gets a list with two dictionaries, his car and his solar powerwall setup. I've looked through the code and it looks like they're doing essentially the same thing and it's the same exact URI and endpoint. To make matters even more confusing, teslajsonpy works fine for me (I get a car and solar site in the response).... 🤷‍♂️

purcell-lab commented 2 years ago

Interesting my tesljsonpy logs from earlier in the year did show the battery product in the log.

https://github.com/alandtse/tesla/issues/79#issuecomment-1109224461

purcell-lab commented 2 years ago

Interesting my tesljsonpy logs from earlier in the year did show the battery product in the log.

alandtse/tesla#79 (comment)

I have upgraded to tesla_custom 2.4.2 and can confirm I have my battery being reported again via teslajsonpy=2.4.4 PRODUCT_LIST endpoint.

@shred86 I'll try your updated custom component again.

{
    "energy_site_id": XXX,
    "resource_type": "battery",
    "site_name": "Home Energy Gateway",
    "id": "YYYYYYYYYYY-XXXXX",
    "gateway_id": "XXX-XX-JX-XXX",
    "asset_site_id": "XXX",
    "energy_left": 6624.578947368423,
    "total_pack_energy": 14033,
    "percentage_charged": 47.207147063125646,
    "battery_type": "ac_powerwall",
    "backup_capable": true,
    "battery_power": 520,
    "storm_mode_enabled": true,
    "powerwall_onboarding_settings_set": true,
    "sync_grid_alert_enabled": true,
    "breaker_alert_enabled": true,
    "components": {
        "battery": true,
        "battery_type": "ac_powerwall",
        "solar": true,
        "solar_type": "pv_panel",
        "grid": true,
        "load_meter": true,
        "market_type": "residential"
    }
}

I end up with KeyError: YYYYYYYYYYY-XXX, where YYYYYYYYYYY-XXX is the "id:" element from the PRODUCTS_LIST endpoint above.


2022-08-29 21:55:29.375 DEBUG (MainThread) [teslajsonpy.controller] 98201: Changing car_online from {} to False
2022-08-29 21:55:29.376 DEBUG (MainThread) [teslajsonpy.controller] 98201: Resetting last_parked_timestamp to: 1661774129 shift_state None
2022-08-29 21:55:29.376 DEBUG (MainThread) [teslajsonpy.controller] Get vehicles. Force: False Time: 0 Interval 60
2022-08-29 21:55:29.376 DEBUG (MainThread) [teslajsonpy.controller] 98201: Skipping update with state asleep. Polling: True. Last update: 1661774129 ago. Last parked: 0 ago. Last wake_up 1661774129 ago. 
2022-08-29 21:55:29.377 DEBUG (MainThread) [custom_components.tesla_custom] Connected to the Tesla API
2022-08-29 21:55:29.377 DEBUG (MainThread) [custom_components.tesla_custom] Running controller.update()
2022-08-29 21:55:29.378 DEBUG (MainThread) [teslajsonpy.controller] Get vehicles. Force: False Time: 0 Interval 60
2022-08-29 21:55:29.378 DEBUG (MainThread) [teslajsonpy.controller] 98201: Skipping update with state asleep. Polling: True. Last update: 1661774129 ago. Last parked: 0 ago. Last wake_up 1661774129 ago. 
2022-08-29 21:55:29.382 ERROR (MainThread) [custom_components.tesla_custom] Unexpected error fetching tesla_custom data: 'YYYYYYYYYYY-XXXXX'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/config/custom_components/tesla_custom/__init__.py", line 280, in _async_update_data
    return await self.controller.update()
  File "/usr/local/lib/python3.10/site-packages/teslajsonpy/controller.py", line 918, in update
    return any(await asyncio.gather(*tasks))
  File "/usr/local/lib/python3.10/site-packages/teslajsonpy/controller.py", line 791, in _get_and_process_battery_data
    async with self.__lock[battery_id]:
KeyError: 'YYYYYYYYYYY-XXXXX'
shred86 commented 2 years ago

Well that’s good it’s showing up on products list now. So strange why it wasn’t earlier.

It looks like you are running my updated version of teslajsonpy with the current version of this integration. You’ll have to do a pip uninstall teslajsonpy again in the HA docker container.

That would be awesome if you could try the updated version again though as I’m looking for feedback. You will still have to do a pip uninstall teslajsonpy as I’ve made changes to both the integration and API wrapper. If you have any feedback or comments, feel free to post them here or in the rewrite pull request I have open. I’m planning on adding some more stuff for battery sites like setting the operation mode as well.

Edit: And I already found one error based on your comment. I just made another update to the rewrite version (my forked version) of teslajsonpy.

purcell-lab commented 2 years ago

Success, I have a second device showing up now, which is my battery.

image

This is what your tesla-rewrite fork is presenting, which has some errors, but it is showing some correct values so it's a great launchpad: image

Here is what my localAPI is presenting, which I believe are true sensor values: image

shred86 commented 2 years ago

Nice! Are those two screenshots taking at about the same time? How did they compare with your Tesla app? We're just reporting what's coming from the Tesla API at a 10 second interval, which obviously doesn't account for how often our inverter/gateways report data to the server and how quickly Tesla's pushes it back out. I'm assuming you'll always have some level of difference between the Tesla Powerwall integration (local API) versus this one, but I would also assume it should match the Tesla app.

Are you getting any errors in the logs? If so, could you please post them here when you get a chance. Thanks again for testing!

Edit: Just pushed a new version of the integration and teslajsonpy. Same thing, pip uninstall teslajsonpy and redownload the integration from my repo. Changes:

Some questions for you:

purcell-lab commented 2 years ago

Nice, here is your update.

I like that you are reporting W & Wh rather than kW.

One error is the Charging status, the battery is charging but the rewrite integration says not charging.

Also the battery reserve/ % charge has a 5% variance, but that is expected. When the battery is fully discharged the Tesla app and cloudAPI both report 0%, but the localAPI reports 5% (which I think is a reserve they don't display to users). When the battery is fully charged all APIs report 100% and it is a sliding scale of variance inbetween.

image

And some comparison screen shots: Screenshot_20220831-093203

and the localAPI: image

purcell-lab commented 2 years ago

Some questions for you:

  • Is the Powerwall backup reserve a setting you set yourself? How it is now with this integration is it's just reporting the backup_reserve_percent from BATTERY_DATA. I'm assuming this only changes when you change it.
  • Looking at this integration, it looks like when you set the operation mode you have to provide the default_real_mode and backup_reserve_percent. The backup real modes are self_consumption, backup and autonomous. My question is what does this look like in the app when you set it? Do you have to provide a backup reserve percent? Or can you just change the mode?

I'm very much looking forward to the control options you are talking about, as I'm currently setting them via a service call in a teslapy integration.

setting backup_reserve_percent determines how much battery to reserve for blackout as is normal operation the battery will not discharge below this level. It is very useful if you want to charge your battery from the grid on demand, as setting backup_reserve to a higher number (upto 100%) will start the battery charging (from the grid). This is shown as a % slider in the app.

setting real_mode allows control of the battery. The app has these as radio buttons, but you can only be in one real_mode so a three way option switch is probably the best option.

https://github.com/carboncoop/tesla-gateway-ha-component image

image

In the app these are a series of switches:

Screenshot_20220831-094151

Some bonus switches which have been recently added to the cloud API are 'Energy Exports' and 'Grid Charging' switches which limit grid behaviours and would also be welcome additions to the integration. These would be best implemented as switches in the integration.

Screenshot_20220831-094206 (2)

purcell-lab commented 2 years ago

Here is a side by side comparision of the tesla-rewrite cloudAPI (top) and localAPI (bottom): Screenshot 2022-08-31 10 16 40

purcell-lab commented 2 years ago

Another issue:

Battery % keeps dropping back to 0, maybe some sort of availability check is necessary.

image

shred86 commented 2 years ago

Ah, good to see it's mostly working. Just pushed an update for both the integration and teslajsonpy.

purcell-lab commented 2 years ago
  • The energy exports and grid charging one will be tough to do unless someone can figure out what the endpoints are for setting those modes.

Wow, you are powering through the energy end points, just uploading your changes now.

Here are the endpoinrs for exports & charging: https://github.com/tdorssers/TeslaPy/blob/61277b92a9ac2d6c59cdf6e4f8526d488f45f034/teslapy/__init__.py#L822

https://owner-api.teslamotors.com/api/1/energy_sites/{}/grid_import_export
Payload options:
"customer_preferred_export_rule": "battery_ok"
"customer_preferred_export_rule": "pv_only"
"disallow_charge_from_grid_with_solar_installed": false
"disallow_charge_from_grid_with_solar_installed": true
purcell-lab commented 2 years ago

It isn't loving the battery_reserve_percent ;-(

Also I wouldn't set the mode and the battery_reserve at the same time, leave them as separate controls.

2022-08-31 15:43:50.113 DEBUG (MainThread) [custom_components.tesla_custom] Finished fetching tesla_custom data in 1.111 seconds (success: True)
2022-08-31 15:43:50.114 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 151, in _handle_refresh_interval
    await self._async_refresh(log_failures=True, scheduled=True)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
    self.async_update_listeners()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 110, in async_update_listeners
    update_callback()
  File "/config/custom_components/tesla_custom/base.py", line 48, in refresh
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 532, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 570, in _async_write_ha_state
    state = self._stringify_state(available)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 538, in _stringify_state
    if (state := self.state) is None:
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 388, in state
    value = self.native_value
  File "/config/custom_components/tesla_custom/sensor.py", line 457, in native_value
    return round(self._energysite.battery_reserve_percent)
AttributeError: 'SolarPowerwallSite' object has no attribute 'battery_reserve_percent'