Updated envoy_reader with smarter fetching and fetching properties based on jsonpaths

[mnederlof]

And other updates:

• URL's are registered in a mapping, with some additional settings, like how often to fetch the url and if installer token is required
• The JWT token has a field for identifying the type of the token (owner or installer). This is now implemented in determining if we can fetch a url endpoint.
• Attributes can be referenced with jsonpath logic or be defined by @envoy_property wrappers. This way it is easier to add new measurements and also easier to differentiate between EnvoyStandard and Metered and Metered with CT.
• Because of the jsonpath logic, we can also use the non-None properties to get a list of endpoints we should update. Which urls are required are evaluated once, when one full getData call has completed.

I also included test data, so we can more easily test the EnvoyData parser, without waiting for output of the envoy, so it is easier to test the jsonpath values.

[mnederlof]

I have been running this version for some days now, works great (for me at least 😄 )

The way this is set up makes more sense to me, as the counter values can be defined separately for separate scenario's of a envoy. Also, all values are now placed in a EnvoyData class (e.g. EnvoyMetered or EnvoyStandard), while all fetching and token logic is stored in the EnvoyReader class.

This envoy_reader.py is fully backward compatible with the current way data is updated in __init__.async_update_data. Also adding more sensors should be easier in the future, as we only have to define the sensor for HA, and then define the jsonpath in the dataclass; as the __getattr__ methods in the EnvoyReader and EnvoyData class will resolve the value configured in the attribute holding the jsonpath.

[mnederlof]

(i just noticed that the escalation to installation account does not work properly yet, working on it) Fixed.

[vincentwolsink]

Wow, this is a huge improvement in code! 😃 Also love having the json fixtures included. We might want to use them to do unit testing later.

Also love the caching system. But the cache time per endpoint seems a bit arbitrary to me? At least for my Envoy most of the endpoints only update every 5 minutes. Why to choice to not cache some endpoints and others for 20 seconds?

Thanks for all your contributions. I will also set you as code owner in the manifest and add you as collaborator to the repo, if you like. 🙂

[vincentwolsink]

@mnederlof This change does throw quite a lot of warnings into my Home Assistant logs:

Logger: custom_components.enphase_envoy.envoy_reader
Source: custom_components/enphase_envoy/envoy_reader.py:1223
Integration: Enphase Envoy (Installer) ([documentation](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/), [issues](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/issues))
First occurred: 15:47:49 (84 occurrences)
Last logged: 15:50:52

Attribute daily_consumption_l2 unknown
Attribute lifetime_consumption_l2 unknown
Attribute consumption_l3 unknown
Attribute daily_consumption_l3 unknown
Attribute lifetime_consumption_l3 unknown

Logger: custom_components.enphase_envoy.envoy_reader
Source: custom_components/enphase_envoy/envoy_reader.py:617
Integration: Enphase Envoy (Installer) ([documentation](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/), [issues](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/issues))
First occurred: 15:47:49 (16 occurrences)
Last logged: 15:50:52

Attribute consumption unknown
Attribute daily_consumption unknown
Attribute lifetime_consumption unknown
Attribute voltage unknown

[mnederlof]

Maybe the detect_model comes up with EnvoyMetered, instead of EnvoyStandard. Could you post some json fixtures from you EnvoyStandard, then maybe i can find some other things to differentiate on

[mnederlof]

And we should make that _LOGGER.warning in __getattr__ in EnvoyReaderdo _LOGGER.debug maybe 😇

[mnederlof]

Wow, this is a huge improvement in code! 😃 Also love having the json fixtures included. We might want to use them to do unit testing later.

Also love the caching system. But the cache time per endpoint seems a bit arbitrary to me? At least for my Envoy most of the endpoints only update every 5 minutes. Why to choice to not cache some endpoints and others for 20 seconds?

Thanks for all your contributions. I will also set you as code owner in the manifest and add you as collaborator to the repo, if you like. 🙂

Well, yes, the inverters only update every 5 minutes, but they do not do this synchronously (so with a lot of inverters, it could yield a different output every X seconds. Since users can set the poll_interval now via configurable options, it would make sense to do this arbitrary value of 20 seconds

But, yes, i did choose that arbitrary value there.

Also, when using a CT, you could do polling a lot, which should reflect the realtime usage of that seconds (when not using the /stream/meter endpoint), so this 20 seconds should alleviate the pressure on the envoy a bit.

We could make that cache setting be configurable too though 😄

[mnederlof]

@mnederlof This change does throw quite a lot of warnings into my Home Assistant logs:

Logger: custom_components.enphase_envoy.envoy_reader
Source: custom_components/enphase_envoy/envoy_reader.py:1223
Integration: Enphase Envoy (Installer) ([documentation](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/), [issues](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/issues))
First occurred: 15:47:49 (84 occurrences)
Last logged: 15:50:52

Attribute daily_consumption_l2 unknown
Attribute lifetime_consumption_l2 unknown
Attribute consumption_l3 unknown
Attribute daily_consumption_l3 unknown
Attribute lifetime_consumption_l3 unknown

Logger: custom_components.enphase_envoy.envoy_reader
Source: custom_components/enphase_envoy/envoy_reader.py:617
Integration: Enphase Envoy (Installer) ([documentation](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/), [issues](https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/issues))
First occurred: 15:47:49 (16 occurrences)
Last logged: 15:50:52

Attribute consumption unknown
Attribute daily_consumption unknown
Attribute lifetime_consumption unknown
Attribute voltage unknown

see https://github.com/vincentwolsink/home_assistant_enphase_envoy_installer/pull/43