vk2him / Enphase-Envoy-mqtt-json

Takes real time stream from Enphase Envoy and publishes to mqtt broker
MIT License
55 stars 23 forks source link

HA 2023.7.2 split configuration help #20

Closed enkrypt3d closed 1 year ago

enkrypt3d commented 1 year ago

On V7 on the envoy and as of a few months ago you can't put any MQTT sensors in the configuration.yaml so I'm having a tough time getting this working. Here's my MQTT.yaml:

sensor:

And my configuration.yaml:

Configure a default setup of Home Assistant (frontend, api, etc)

default_config:

http: server_port: 8123 use_x_forwarded_for: true trusted_proxies:

Text to speech

tts:

group: !include groups.yaml automation: !include automations.yaml script: !include scripts.yaml scene: !include scenes.yaml template: !include templates.yaml sensor: !include sensors.yaml mqtt: !include mqtt.yaml

binary_sensor: !include binary_sensors.yaml

switch: !include switch.yaml

homeassistant: customize: !include customize.yaml auth_mfa_modules:

logger: default: info

binary_sensor:

I imagine I need to put some of your code in my templates.yaml file?

here is my templates.yaml:

vk2him commented 1 year ago

You can put mqtt sensors into configuration yaml, it just needs to be in the "new" format that starts with mqtt:

Here's mine from configuration.yaml - if you put them in mqtt.yaml they will work provided you have a reference to it in configuration.yaml


mqtt:
  sensor:
    - name: "mqtt_production"
      state_topic: "envoy/json"
      qos: 0
      unit_of_measurement: "W"
      value_template: '{% if is_state("sun.sun", "below_horizon")%}0{%else%}{{ value_json["production"]["ph-a"]["p"]  | float }}{%endif%}'
      state_class: measurement
      device_class: power

    - name: "mqtt_consumption"
      state_topic: "envoy/json"
      value_template: '{{ value_json["total-consumption"]["ph-a"]["p"] }}'
      qos: 0
      unit_of_measurement: "W"
      state_class: measurement
      device_class: power

    - name: "mqtt_power_factor"
      state_topic: "envoy/json"
      qos: 0
      unit_of_measurement: "%"
      value_template: '{{ value_json["total-consumption"]["ph-a"]["pf"] }}'
      state_class: measurement
      device_class: power_factor

    - name: "mqtt_voltage"
      state_topic: "envoy/json"
      qos: 0
      unit_of_measurement: "V"
      value_template: '{{ value_json["total-consumption"]["ph-a"]["v"] }}'
      state_class: measurement
      device_class: voltage

    - state_topic: "envoy/json"
      name: "mqtt_net_voltage"
      value_template: '{{ value_json["net-consumption"]["ph-a"]["v"] }}'
      qos: 0
      unit_of_measurement: "V"
      state_class: measurement
      device_class: voltage
vk2him commented 1 year ago

P.S. - I just noticed my Readme file doesn't show the "new" method for v5 - it still shows the old method. I'll update it with what I posted above as the new method is needed for all mqtt sensors

enkrypt3d commented 1 year ago

What about the split template config? Thanks!

On Sun, Jul 23, 2023, 8:45 PM Ian Mills @.***> wrote:

P.S. - I just noticed my Readme file doesn't show the "new" method for v5

  • it still shows the old method. I'll update it with what I posted above as the new method is needed for all mqtt sensors

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647033618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRVCIOAC262CBEMKSDLXRXASHANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

What about the split template config? Thanks!

That's what I meant in my comment above - currenty your mqtt.yaml code isn't being processed because you may not be referencing it.

I can see you have this:

customize: !include customize.yaml

however you don't seem to have this

customize: !include mqtt.yaml ?

enkrypt3d commented 1 year ago

group: !include groups.yaml automation: !include automations.yaml script: !include scripts.yaml scene: !include scenes.yaml template: !include templates.yaml sensor: !include sensors.yaml mqtt: !include mqtt.yaml

It's there.... but it's not using "customize". I have my gas meter running thru MQTT and metermon / RTL_TCP and it works fine.

I guess your code needs to be updated to the latest HA syntax?

I don't think my templates.yaml is correct yet.

image

vk2him commented 1 year ago

Hi - I just wanted to clarify some points as I may be misinterpreting what you're saying.

The addon simply sends a json string to the Mqtt broker - thats all it does.

If you want to use that json string, you need to split it into the discrete components, and in Home assistant you do that with templates. So when you say "I guess your code needs to be updated to the latest HA syntax?" , my addon doesn't need to be changed, the templates in Home assiatnt that you write need to be done correctly, and that's up you you.

My documentation in the readme file has some sample code home assistant templates and as I mentioned above, I just noticed that in the v5 section the templates still refer to the "old" method that HA used. The V7 ones are correct though.

Before you do anything like trying to get power wheel card etc working, I recommend you first confirm your mqtt sensors are working by using the States tab in Developer tools

image

You should probably also use mqttexplorer to see what the raw json string looks like (mine is v5 so is different to your v7)

image

enkrypt3d commented 1 year ago

Yes the MQTT broker works fine for my other entities, it's just confusing how you have your configurations layed out. You have some of the MQTT stuff split but not all of it as well as the template....

My attempt at getting template.yaml updated but it doesn't like it:

[image: image.png]

On Sun, Jul 23, 2023 at 9:33 PM Ian Mills @.***> wrote:

Hi - I just wanted to clarify some points as I may be misinterpreting what you're saying.

The addon simply sends a json string to the Mqtt broker - thats all it does.

If you want to use that json string, you need to split it into the discrete components, and in Home assistant you do that with templates. So when you say "I guess your code needs to be updated to the latest HA syntax?" , my addon doesn't need to be changed, the templates in Home assiatnt that you write need to be done correctly, and that's up you you.

My documentation in the readme file has some sample code home assistant templates and as I mentioned above, I just noticed that in the v5 section the templates still refer to the "old" method that HA used. The V7 ones are correct though.

Before you do anything like trying to get power wheel card etc working, I recommend you first confirm your mqtt sensors are working by using the States tab in Developer tools

[image: image] https://user-images.githubusercontent.com/22878629/255448829-1be2f107-b357-4c95-acf5-4f64ca9bf421.png

You should probably also use mqttexplorer to see what the raw json string looks like (mine is v5 so is different to your v7)

[image: image] https://user-images.githubusercontent.com/22878629/255449086-4fdec3b5-9fb9-4f65-a095-ed57fcf60de7.png

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647066131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRTJ3QJFIONCOB2T4ZDXRXGIJANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

Your templates are missing this in every case: state_topic: "envoy/json" Refer to the readme v7 templates - you're trying to overcomplicate it by splitting into template.yaml and dropping code along the way. Why not try deleting the template.yaml code and putting this into configuration.yaml instead cause it works 100%

configuration.yaml configuration examples For FW 7
mqtt:
  sensor:
    - name: envoy mqtt consumption
      state_topic: "envoy/json"
      value_template: '{{ value_json[1]["activePower"] | round(0) | int(0)}}'
      unique_id: envoy_mqtt_consumption
      qos: 0
      unit_of_measurement: "W"
      state_class: measurement
      device_class: power
    - name: envoy mqtt voltage
      state_topic: "envoy/json"
      value_template: '{{ value_json[1]["voltage"] | round(0) | int(0)}}'
      unique_id: envoy_mqtt_voltage
      qos: 0
      unit_of_measurement: "V"
      state_class: measurement
      device_class: voltage
    - name: envoy mqtt current
      state_topic: "envoy/json"
      value_template: '{{ value_json[1]["current"] | round(2)}}'
      unique_id: envoy_mqtt_current
      qos: 0
      unit_of_measurement: "A"
      state_class: measurement
      device_class: current
    - name: envoy mqtt power factor
      state_topic: "envoy/json"
      value_template: '{{ value_json[1]["pwrFactor"] | round(2)}}'
      unique_id: envoy_mqtt_power_factor
      qos: 0
      unit_of_measurement: "%"
      state_class: measurement
      device_class: power_factor
enkrypt3d commented 1 year ago

With having the state_topic, that it errors out as well. as I said you have a mixture of the legacy formatting along with the new.... I'm trying to translate it to the new format.... if it were only as easy as moving it into one file then I'd do that but the entire structure has to change....

On Sun, Jul 23, 2023 at 10:46 PM Ian Mills @.***> wrote:

Your templates are missing this in every case: state_topic: "envoy/json" Refer to the readme v7 templates - you're trying to overcomplicate it by splitting into template.yaml and dropping code along the way. Why not try deleting the template.yaml code and putting this into configuration.yaml instead cause it works 100%

configuration.yaml configuration examples For FW 7 mqtt: sensor:

  • name: envoy mqtt consumption state_topic: "envoy/json" value_template: '{{ value_json[1]["activePower"] | round(0) | int(0)}}' unique_id: envoy_mqtt_consumption qos: 0 unit_of_measurement: "W" state_class: measurement device_class: power
  • name: envoy mqtt voltage state_topic: "envoy/json" value_template: '{{ value_json[1]["voltage"] | round(0) | int(0)}}' unique_id: envoy_mqtt_voltage qos: 0 unit_of_measurement: "V" state_class: measurement device_class: voltage
  • name: envoy mqtt current state_topic: "envoy/json" value_template: '{{ value_json[1]["current"] | round(2)}}' unique_id: envoy_mqtt_current qos: 0 unit_of_measurement: "A" state_class: measurement device_class: current
  • name: envoy mqtt power factor state_topic: "envoy/json" value_template: '{{ value_json[1]["pwrFactor"] | round(2)}}' unique_id: envoy_mqtt_power_factor qos: 0 unit_of_measurement: "%" state_class: measurement device_class: power_factor

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647116446, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSRWKX4UZ6TK2PPBZTXRXOXJANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

As far as I've seen, the two can be used together - you can put some templates in configuration.yaml and others in templates.yaml

I'm not sure why you don't want to test with the snippet from the post above into configuration.yaml as it works 100% for losts of people

enkrypt3d commented 1 year ago

Just tried it and it has created the sensors but they're not getting data. Where do I specify the enlighten login? Maybe that's the issue..... also for some reason my gas meter is no longer working so I'm debugging that now too :(

On Sun, Jul 23, 2023 at 11:01 PM Ian Mills @.***> wrote:

As far as I've seen, the two can be used together - you can put some templates in configuration.yaml and others in templates.yaml

I'm not sure why you don't want to test with the snippet from the post above into configuration.yaml as it works 100%

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647125973, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRU3HRDG33AIP6FSOLTXRXQOZANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

First things first

1) fix your gas meter 2) Enlighten login details are in the addon Configuration - if you haven't set them then it wont work on V7 3) after entering them, restart the addon and look at the Log tab - you should see the token and a message saying it Subscribed to the MQTT_TOPIC: envoy/json 4) Use Developer Tools to look at the new mqtt sensors

enkrypt3d commented 1 year ago

Realized that if there are any MQTT configs in the configuration.yaml, it ignores everything in mqtt.yaml. So it'd be very helpful to just follow the new HA standard and keep everything in its own config for templates.yaml / mqtt.yaml etc. Any time I try to have them in both places, it only acknowledges one of them.

On Sun, Jul 23, 2023 at 11:38 PM Ian Mills @.***> wrote:

First things first

  1. fix your gas meter
  2. Enlighten login details are in the addon Configuration - if you haven't set them then it wont work on V7
  3. after entering them, restart the addon and look at the Log tab - you should see the token and a message saying it Subscribed to the MQTT_TOPIC: envoy/json
  4. Use Developer Tools to look at the new mqtt sensors

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647150046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRVP2WCBCK7NH7THB5LXRXU3HANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

Like I said earlier - writing the templates is your responsibility. I provided some template examples - use them or not - it's up to you.

enkrypt3d commented 1 year ago

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 24/07/2023 01:00:01 Serial number: 24/07/2023 01:00:01 Detected FW version 7 24/07/2023 01:00:01 FREEDS is inactive 24/07/2023 01:00:01 No token in file: /data/token.txt 24/07/2023 01:00:01 Generating new token 24/07/2023 01:00:01 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]> using this info {'user[email]': 'xxxxxxxxxxx@email.com', 'user[password]': 'xxxxxxxxxxxx'} 24/07/2023 01:00:01 Generating new token Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "//envoy_to_mqtt_json.py", line 355, in scrape_stream_meters headers = {"Authorization": "Bearer " + ENVOY_TOKEN}


TypeError: can only concatenate str (not "NoneType") to str
24/07/2023 01:00:01 Connected to 192.168.1.5:1883
24/07/2023 01:00:01 Subscribed to MQTT_TOPIC: /envoy/json
24/07/2023 01:00:01 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]>  using this info {'user[email]': xxxxxxxxx@email.com', 'user[password]': 'xxxxxxxxxxxxx'}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

######## 

your addon wont start even with the correct values.... 
enkrypt3d commented 1 year ago

24/07/2023 01:03:15 No token in file: /data/token.txt 24/07/2023 01:03:15 Generating new token 24/07/2023 01:03:15 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]> using this info {'user[email]': 'xxxxxxxx@email.com', 'user[password]': 'xxxxxxxxxxxxxxxx'} 24/07/2023 01:03:15 Generating new token Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "//envoy_to_mqtt_json.py", line 355, in scrape_stream_meters headers = {"Authorization": "Bearer " + ENVOY_TOKEN}


TypeError: can only concatenate str (not "NoneType") to str
24/07/2023 01:03:15 Connected to 192.168.1.5:1883
24/07/2023 01:03:15 Subscribed to MQTT_TOPIC: /envoy/json
24/07/2023 01:03:15 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]>  using this info {'user[email]': 'xxxxxxxx@email.com', 'user[password]': 'xxxxxxxxxxxx'}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
vk2him commented 1 year ago

Please confirm you have actually entered your enlighten username and password in the enphase_user and enphase_user_pass configuration tab? The 401 error you posted looks suspiciously like you're using the default values and not yours?

image

enkrypt3d commented 1 year ago

Yes all the data is entered correctly

On Mon, Jul 24, 2023, 1:32 AM Ian Mills @.***> wrote:

Please confirm you have actually entered your enlighten username and password in the ephase_user and enphase_user_pass configuration tab? The 401 error you posed looks suspiciously like you're using the default values and not yours?

[image: image] https://user-images.githubusercontent.com/22878629/255490239-0b9aa2c3-3f0a-4e78-9de0-8bf56844daf3.png

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647235913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRWLUYUB5TSQNTCQSJTXRYCHRANCNFSM6AAAAAA2U2DZNA . You are receiving this because you authored the thread.Message ID: @.***>

vk2him commented 1 year ago

I'll raise this in https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/5

helderfmf commented 1 year ago

@enkrypt3d Can you login here https://enlighten.enphaseenergy.com/login with your credentials? If yes, can you see data from you Envoy there, like readings, history, etc? Then go to https://entrez.enphaseenergy.com and login again and try to generate a token, just start typing 1000 in the "Select System *" it yours will apear.

enkrypt3d commented 1 year ago

Yes I'm using the same account with other enphase integrations and it's working with other integrations ...

On Mon, Jul 24, 2023 at 5:09 AM Helder Ferreira @.***> wrote:

@enkrypt3d https://github.com/enkrypt3d Can you login here https://enlighten.enphaseenergy.com/login with your credentials? If yes, can you see data from you Envoy there, like readings, history, etc? Then go to https://entrez.enphaseenergy.com and login again and try to generate a token, just start typing 1000 in the "Select System *" it yours will apear.

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647518310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRQAY27KFFZE6NRKT3DXRY3VDANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

enkrypt3d commented 1 year ago

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 24/07/2023 16:36:11 Serial number: 202117004813 24/07/2023 16:36:11 Detected FW version 7 24/07/2023 16:36:11 FREEDS is active, using topic: Inverter/GridWatts 24/07/2023 16:36:11 No token in file: /data/token.txt 24/07/2023 16:36:11 Generating new token 24/07/2023 16:36:11 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]> using this info {'user[email]': @.', 'user[password]': 'xxxxxxActualEnphasePassword'} 24/07/2023 16:36:11 Generating new token Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(self._args, self._kwargs) File "//envoy_to_mqtt_json.py", line 355, in scrape_stream_meters headers = {"Authorization": "Bearer " + ENVOY_TOKEN}


TypeError: can only concatenate str (not "NoneType") to str
24/07/2023 16:36:11 Connected to 192.168.1.5:1883
24/07/2023 16:36:11 Subscribed to MQTT_TOPIC: /envoy/json
24/07/2023 16:36:11 Failed connect to
https://enlighten.enphaseenergy.com/login/login.json? to generate
token part 1 got <Response [401]>  using this info {'user[email]':
***@***.***', 'user[password]': 'xxxxxxxxxxxxx'}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

I thought the token generation URL was
https://entrez.enphaseenergy.com/login not
https://enlighten.enphaseenergy.com? I can login just fine to both....

On Mon, Jul 24, 2023 at 10:43 AM Corey J ***@***.***> wrote:

> Yes I'm using the same account with other enphase integrations and it's
> working with other integrations ...
>
> On Mon, Jul 24, 2023 at 5:09 AM Helder Ferreira ***@***.***>
> wrote:
>
>> @enkrypt3d <https://github.com/enkrypt3d>
>> Can you login here https://enlighten.enphaseenergy.com/login with your
>> credentials?
>> If yes, can you see data from you Envoy there, like readings, history,
>> etc? Then go to https://entrez.enphaseenergy.com and login again and try
>> to generate a token, just start typing 1000 in the "Select System *" it
>> yours will apear.
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1647518310>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AWHLHRQAY27KFFZE6NRKT3DXRY3VDANCNFSM6AAAAAA2U2DZNA>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>
vk2him commented 1 year ago

@enkrypt3d

This is strange - it appears you haven't typed your password in correctly into the addon's configuration, or that the sequence of password characters you're using is causing an issue with the python script.

The addon code generates the token in two separate section - the first one scrapes some data when you log on to https://enlighten.enphaseenergy.com/login - this is the page that displays an overview of your system and is purely used to obtain the login details needed for the next section.

The next section takes the login details and plugs them into https://entrez.enphaseenergy.com/ which then generates your token.

From your error message, section 1 returns <Response [401]> which basically means https://enlighten.enphaseenergy.com/login is saying your username/password is incorrect.

So for testing, can I please ask you do two things? 1) Triple check you have entered your password/user email correctly into the addon configuration. 2) If it still fails with 401 after your confirm the password is correct, change your password on Enphase to be alpha numeric with no special letters like !@#$%^&*(){}[] etc ..... then put that new password into the addon configuration and report back here?

enkrypt3d commented 1 year ago

Instead of changing the password I just escaped the special characters in the password field. it'd be nice if this was added and also masked?

Now the add on is running. still trying to get the template & mqtt yaml code sorted out. Entities still in "unknown" state. it's a bit confusing to me....

On Mon, Jul 24, 2023 at 7:10 PM Ian Mills @.***> wrote:

@enkrypt3d https://github.com/enkrypt3d

This is strange - it appears you haven't typed your password in correctly into the addon's configuration, or that the sequence of password characters you're using is causing an issue with the python script.

The addon code generates the token in two separate section - the first one scrapes some data when you log on to https://enlighten.enphaseenergy.com/login - this is the page that displays an overview of your system and is purely used to obtain the login details needed for the next section.

The next section takes the login details and plugs them into https://entrez.enphaseenergy.com/ which then generates your token.

From your error message, section 1 returns <Response [401]> which basically means https://enlighten.enphaseenergy.com/login is saying your username/password is incorrect.

So for testing, can I please ask you do two things?

  1. Triple check you have entered your password/user email correctly into the addon configuration.
  2. If it still fails with 401 after your confirm the password is correct, change your password on Enphase to be alpha numeric with no special letters like !@#$%^&*(){}[] etc ..... then put that new password into the addon configuration and report back here?

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648742899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRQFV6GCM2I7KR266NDXR36FTANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

enkrypt3d commented 1 year ago

Well it starts and then has the same error again a few minutes later... changing the password would break all my other enphase stuff I have running.......

On Mon, Jul 24, 2023 at 7:17 PM Corey J @.***> wrote:

Instead of changing the password I just escaped the special characters in the password field. it'd be nice if this was added and also masked?

Now the add on is running. still trying to get the template & mqtt yaml code sorted out. Entities still in "unknown" state. it's a bit confusing to me....

On Mon, Jul 24, 2023 at 7:10 PM Ian Mills @.***> wrote:

@enkrypt3d https://github.com/enkrypt3d

This is strange - it appears you haven't typed your password in correctly into the addon's configuration, or that the sequence of password characters you're using is causing an issue with the python script.

The addon code generates the token in two separate section - the first one scrapes some data when you log on to https://enlighten.enphaseenergy.com/login - this is the page that displays an overview of your system and is purely used to obtain the login details needed for the next section.

The next section takes the login details and plugs them into https://entrez.enphaseenergy.com/ which then generates your token.

From your error message, section 1 returns <Response [401]> which basically means https://enlighten.enphaseenergy.com/login is saying your username/password is incorrect.

So for testing, can I please ask you do two things?

  1. Triple check you have entered your password/user email correctly into the addon configuration.
  2. If it still fails with 401 after your confirm the password is correct, change your password on Enphase to be alpha numeric with no special letters like !@#$%^&*(){}[] etc ..... then put that new password into the addon configuration and report back here?

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648742899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRQFV6GCM2I7KR266NDXR36FTANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

enkrypt3d commented 1 year ago

Tried changing my password and it eventually fails with the same error:

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 24/07/2023 19:42:00 Serial number: 202117004813 24/07/2023 19:42:00 Detected FW version 7 24/07/2023 19:42:00 FREEDS is active, using topic: Inverter/GridWatts 24/07/2023 19:42:00 No token in file: /data/token.txt 24/07/2023 19:42:00 Generating new token 24/07/2023 19:42:00 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]> using this info {'user[email]': @.', 'user[password]': 'xxxxxxxxxxx'} 24/07/2023 19:42:00 Generating new token 24/07/2023 19:42:00 Connected to 192.168.1.5:1883 24/07/2023 19:42:00 Subscribed to MQTT_TOPIC: /envoy/json Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(self._args, self._kwargs) File "//envoy_to_mqtt_json.py", line 355, in scrape_stream_meters headers = {"Authorization": "Bearer " + ENVOY_TOKEN}


TypeError: can only concatenate str (not "NoneType") to str
24/07/2023 19:42:00 Failed connect to
https://enlighten.enphaseenergy.com/login/login.json? to generate
token part 1 got <Response [401]>  using this info {'user[email]':
***@***.***', 'user[password]': 'xxxxxxxxxx'}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

On Mon, Jul 24, 2023 at 7:32 PM Corey J ***@***.***> wrote:

> Well it starts and then has the same error again a few minutes later...
> changing the password would break all my other enphase stuff I have
> running.......
>
> On Mon, Jul 24, 2023 at 7:17 PM Corey J ***@***.***> wrote:
>
>> Instead of changing the password I just escaped the special characters in
>> the password field. it'd be nice if this was added and also masked?
>>
>> Now the add on is running. still trying to get the template & mqtt yaml
>> code sorted out. Entities still in "unknown" state. it's a bit confusing to
>> me....
>>
>> On Mon, Jul 24, 2023 at 7:10 PM Ian Mills ***@***.***>
>> wrote:
>>
>>> @enkrypt3d <https://github.com/enkrypt3d>
>>>
>>> This is strange - it appears you haven't typed your password in
>>> correctly into the addon's configuration, or that the sequence of password
>>> characters you're using is causing an issue with the python script.
>>>
>>> The addon code generates the token in two separate section - the first
>>> one scrapes some data when you log on to
>>> https://enlighten.enphaseenergy.com/login - this is the page that
>>> displays an overview of your system and is purely used to obtain the login
>>> details needed for the next section.
>>>
>>> The next section takes the login details and plugs them into
>>> https://entrez.enphaseenergy.com/ which then generates your token.
>>>
>>> From your error message, section 1 returns <Response [401]> which
>>> basically means https://enlighten.enphaseenergy.com/login is saying
>>> your username/password is incorrect.
>>>
>>> So for testing, can I please ask you do two things?
>>>
>>>    1. Triple check you have entered your password/user email correctly
>>>    into the addon configuration.
>>>    2. If it still fails with 401 after your confirm the password is
>>>    correct, change your password on Enphase to be alpha numeric with no
>>>    special letters like !@#$%^&*(){}[] etc ..... then put that new password
>>>    into the addon configuration and report back here?
>>>
>>> —
>>> Reply to this email directly, view it on GitHub
>>> <https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648742899>,
>>> or unsubscribe
>>> <https://github.com/notifications/unsubscribe-auth/AWHLHRQFV6GCM2I7KR266NDXR36FTANCNFSM6AAAAAA2U2DZNA>
>>> .
>>> You are receiving this because you were mentioned.Message ID:
>>> ***@***.***>
>>>
>>
enkrypt3d commented 1 year ago

Where can I copy my token to? I already have a 1 year token so can you add a field for existing tokens?

On Mon, Jul 24, 2023 at 7:43 PM Corey J @.***> wrote:

Tried changing my password and it eventually fails with the same error:

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 24/07/2023 19:42:00 Serial number: 202117004813 24/07/2023 19:42:00 Detected FW version 7 24/07/2023 19:42:00 FREEDS is active, using topic: Inverter/GridWatts 24/07/2023 19:42:00 No token in file: /data/token.txt 24/07/2023 19:42:00 Generating new token 24/07/2023 19:42:00 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]> using this info {'user[email]': @.', 'user[password]': 'xxxxxxxxxxx'} 24/07/2023 19:42:00 Generating new token 24/07/2023 19:42:00 Connected to 192.168.1.5:1883 24/07/2023 19:42:00 Subscribed to MQTT_TOPIC: /envoy/json Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(self._args, self._kwargs) File "//envoy_to_mqtt_json.py", line 355, in scrape_stream_meters headers = {"Authorization": "Bearer " + ENVOY_TOKEN}


TypeError: can only concatenate str (not "NoneType") to str
24/07/2023 19:42:00 Failed connect to https://enlighten.enphaseenergy.com/login/login.json? to generate token part 1 got <Response [401]>  using this info {'user[email]': ***@***.***', 'user[password]': 'xxxxxxxxxx'}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

On Mon, Jul 24, 2023 at 7:32 PM Corey J ***@***.***> wrote:

> Well it starts and then has the same error again a few minutes later...
> changing the password would break all my other enphase stuff I have
> running.......
>
> On Mon, Jul 24, 2023 at 7:17 PM Corey J ***@***.***> wrote:
>
>> Instead of changing the password I just escaped the special
>> characters in the password field. it'd be nice if this was added and also
>> masked?
>>
>> Now the add on is running. still trying to get the template & mqtt yaml
>> code sorted out. Entities still in "unknown" state. it's a bit confusing to
>> me....
>>
>> On Mon, Jul 24, 2023 at 7:10 PM Ian Mills ***@***.***>
>> wrote:
>>
>>> @enkrypt3d <https://github.com/enkrypt3d>
>>>
>>> This is strange - it appears you haven't typed your password in
>>> correctly into the addon's configuration, or that the sequence of password
>>> characters you're using is causing an issue with the python script.
>>>
>>> The addon code generates the token in two separate section - the first
>>> one scrapes some data when you log on to
>>> https://enlighten.enphaseenergy.com/login - this is the page that
>>> displays an overview of your system and is purely used to obtain the login
>>> details needed for the next section.
>>>
>>> The next section takes the login details and plugs them into
>>> https://entrez.enphaseenergy.com/ which then generates your token.
>>>
>>> From your error message, section 1 returns <Response [401]> which
>>> basically means https://enlighten.enphaseenergy.com/login is saying
>>> your username/password is incorrect.
>>>
>>> So for testing, can I please ask you do two things?
>>>
>>>    1. Triple check you have entered your password/user email correctly
>>>    into the addon configuration.
>>>    2. If it still fails with 401 after your confirm the password is
>>>    correct, change your password on Enphase to be alpha numeric with no
>>>    special letters like !@#$%^&*(){}[] etc ..... then put that new password
>>>    into the addon configuration and report back here?
>>>
>>> —
>>> Reply to this email directly, view it on GitHub
>>> <https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648742899>,
>>> or unsubscribe
>>> <https://github.com/notifications/unsubscribe-auth/AWHLHRQFV6GCM2I7KR266NDXR36FTANCNFSM6AAAAAA2U2DZNA>
>>> .
>>> You are receiving this because you were mentioned.Message ID:
>>> ***@***.***>
>>>
>>
vk2him commented 1 year ago

The token is stored in the container that the addon creates, so you'll need Portainer or another method to access the container. The token is stored in /data/token.txt within the container (not /data/token in the HA OS) Let me know how you go, I "might" be able to let you enter the token into configuration and have it stored if you don't have access to the files in the container

enkrypt3d commented 1 year ago

Seems this doesn't work either - what format does it need to be in? I tried including the {"auth_token": "token"} and got the same errors....

s6-rc: info: service legacy-services successfully started 24/07/2023 21:59:28 Serial number: 202117004813 24/07/2023 21:59:28 Detected FW version 7 24/07/2023 21:59:28 FREEDS is active, using topic: Inverter/GridWatts 24/07/2023 21:59:28 Read token from file /data/token.txt : "eyJ**Olk75Sw"

Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(self._args, self._kwargs) File "//envoy_to_mqtt_json.py", line 356, in scrape_stream_meters stream = requests.get(url, timeout=5, verify=False, headers=headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, httplib_request_kw) File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/lib/python3.11/http/client.py", line 1286, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.11/http/client.py", line 1327, in _send_request self.putheader(hdr, value) File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 224, in putheader _HTTPConnection.putheader(self, header, values) File "/usr/lib/python3.11/http/client.py", line 1264, in putheader raise ValueError('Invalid header value %r' % (values[i],)) ValueError: Invalid header value b'Bearer "eyJr***kROlk75Sw"\n' 24/07/2023 21:59:28 Connected to 192.168.1.5:1883 24/07/2023 21:59:28 Subscribed to MQTT_TOPIC: /envoy/json s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped

On Mon, Jul 24, 2023 at 9:22 PM Ian Mills @.***> wrote:

The token is stored in the container that the addon creates, so you'll need Portainer or another method to access the container. The token is stored in /data/token.txt within the container (not /data/token in the HA OS)

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648827576, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRXCV6LJPJB2ABYHQW3XR4NTZANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

vk2him commented 1 year ago

It's the full token on a single line without a newline character at the end - so using your error and assuming your token starts with eyJr and ends with kROlk75Sw it should be a single line with exactly this:

eyJr*******************************************************************************************kROlk75Sw

vk2him commented 1 year ago

Can you also please advise the version number of my addon you're running, just in case you have an old variant somehow? Something weird is happening with your system ...

enkrypt3d commented 1 year ago

Using Current version: 1.0.12 - so no quotations just the token by itself correct?

On Mon, Jul 24, 2023 at 10:49 PM Ian Mills @.***> wrote:

Can you also please advise the version number of my addon you're running, just in case you have an old variant somehow? Something weird is happening with your system ...

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648936012, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSGG6DUM7BHPYSOVRTXR4X4RANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

vk2him commented 1 year ago

Correct - 1.0.12 is the current version as well - maybe try rebuilding it as a clutching straws effort

enkrypt3d commented 1 year ago

What does USE_FREEDS do?

Also note I'm on IQ Gateway Software Version 7.3.517 if that helps

On Mon, Jul 24, 2023 at 11:18 PM Ian Mills @.***> wrote:

Correct - 1.0.12 is the current version as well - maybe try rebuilding it as a clutching straws effort

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1648984686, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRXCGQYB4MZQHMY7JMLXR43GXANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

vk2him commented 1 year ago

I wrote the V5 part of the code as I have a v5 Envoy. Initially I didn't want to support V7 as I had no way of testing. Helder @helderfmf managed to get v7 working using my v5 code as a base. I merged his v7 code into mine last week.

He added some code specific for his use case which included Freeds - it's a low cost method to use excess solar generation to heat water or a floor slab. Just leave the Freed switch off if you don't need it.

You'll see I refer some questions to Helder if they are specific to v7 code that I can't test.

vk2him commented 1 year ago

What I can't understand is why you were able to get some data out of this and now it stopped with the error you showed above ... i can't test as I don't have v7 so hopefully Helder may give some insight

vk2him commented 1 year ago

@enkrypt3d - Helder has added some debug code that will help working out why it doesn't work for you. As this is testing, I don't want to merge the debug code into the Public released version. Can I ask you to assist please to test Helders code by adding his repo to HA, adding his test version, turn on debug in Configuration and then report back here with the log from the addon? https://github.com/helderfmf/Enphase-Envoy-mqtt-json Thanks

vk2him commented 1 year ago

@enkrypt3d - Hi, can you please try updating to my version (rather than Helders) as some debug code has been added today to hopefully identify where the issues may be. Turn on the Debug switch in Configuration and copy/paste the log here please after you remove your long token. Then turn off debug as it can fill the log quickly. Thanks

enkrypt3d commented 1 year ago

Ok I can take a look and let you know thanks!

On Thu, Jul 27, 2023 at 9:22 PM Ian Mills @.***> wrote:

@enkrypt3d https://github.com/enkrypt3d - Hi, can you please try updating to my version (rather than Helders) as some debug code has been added today to hopefully identify where the issues may be. Turn on the Debug switch in Configuration and copy/paste the log here please after you remove your long token. Then turn off debug as it can fill the log quickly. Thanks

— Reply to this email directly, view it on GitHub https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/20#issuecomment-1654837351, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRFR6AECHJZKZG74E3XSMH5ZANCNFSM6AAAAAA2U2DZNA . You are receiving this because you were mentioned.Message ID: @.***>

vk2him commented 1 year ago

@enkrypt3d - the error you're seeing was also reported in this issue https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/18 - it was resolved in the last few posts in the thread.

Take a read as the fix may help you - you may have a line feed character at the end of your token in your token.txt file.