[New Device] Smart Home Panel

[Ne0-Hack3r]

I there any interest in adding the Smart Home Panel to this integration? I have done so using a YAML package and Mosquito broker on Home Assistant but never got round to creating a proper integration. I would be willing to contribute information, testing, etc. if that is something others would like to see added and @tolwi is interested in doing.

[BeyondPixels]

Hello,

I'm definitely interested in getting a proper integration ! I'm planning on buying 2 SHP + Delta pros later this year and you can count me up for testing once I got them.

I've seen your and lwsrbrts great work but I'm not sure if I have the skills to configure all of that.

[dafmeuk]

Absolutely! An integration would be hugely valuable to me.

[Thomptronics]

+1

[bluefire100]

Is SHP already implemented?

[Ne0-Hack3r]

@tolwi

Any interest in collaborating to add SHP support to your integration?

[dafmeuk]

@towli … I’d just like to encourage you to take up the offer from @Ne0-Hack3r to collaborate on adding Smart Home Panel support to hassio-ecoflow-cloud

I’ve been able to do some truly useful automations with my Delta Pro’s, but the main thing holding back the potential of my automations is not being able to control when the Smart Home Panel ports are set to charge, and switching circuits to battery or mains. The SHP side of my automations can only ever be time based rules with Ecoflow’s native automations, which is really restrictive. I’m still using the original hassio-ecoflow integration, but would switch everything over to hassio-ecoflow-cloud in an instant it it would get me SHP support.

[Ne0-Hack3r]

@dafmeuk

There is a way to do this by building your own YAML package (or using mine as a base). But it is fairly involved compared to just loading a HACS integration. Given that I already have the key/value structure for all the SHP sensors and settings I'd just need some help understanding how to incorporate that into the python integration (or I can simply provide what I already have and let the author add it to this integration), I'd be happy to test and debug in that case as well.

[tolwi]

@Ne0-Hack3r attach your yaml and I'll try to provide you with a draft version for testing (I don't have SHP so I can not verify anything)

[Ne0-Hack3r]

@tolwi

I'm happy to share whatever you need and have invited you to my private repo.

The specific YAML package file is here: https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_integration.yaml

[Ne0-Hack3r]

@tolwi

There is a bunch more in that YAML that is needed to provide access to all the relevant sensors and settings and the package has evolved quite a bit around my own personal usage. Obviously your integration should just expose all relevant entities and let those that want to build something more elaborate create their own helpers, automations, etc.

[Ne0-Hack3r]

@dafmeuk

I might caution against trying to automate everything for the SHP within home assistant. While it can be done there are caveats to consider:

For example: if you create no automations other than setting up the built in outage automation and then automate daily usage/cycling by setting individual circuits to "use power station" you then also have to handle the fallback to grid (prior to discharge limit) or when the battery hits that limit your circuits will turn OFF. "use power station" means "only ever use power station" (never use grid) and vice versa. So you may want to go the route of creating an automation that runs 24x7x365 and then enabling or disabling it using an HA automation. Perhaps in adding SHP to this HACS integration we can expose an on/off switch for each active automation (or some such). Just trying to explain that SHP integration gets a bit complicated when it comes to automations and the internal handling of when/how things fall back to grid, etc.

[tolwi]

The specific YAML package file is here: https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_integration.yaml

Can you add a few raw json messages from mqtt to you repo, please.

[dafmeuk]

@dafmeuk

I might caution against trying to automate everything for the SHP within home assistant. While it can be done there are caveats to consider:

For example: if you create no automations other than setting up the built in outage automation and then automate daily usage/cycling by setting individual circuits to "use power station" you then also have to handle the fallback to grid (prior to discharge limit) or when the battery hits that limit your circuits will turn OFF. "use power station" means "only ever use power station" (never use grid) and vice versa. So you may want to go the route of creating an automation that runs 24x7x365 and then enabling or disabling it using an HA automation. Perhaps in adding SHP to this HACS integration we can expose an on/off switch for each active automation (or some such). Just trying to explain that SHP integration gets a bit complicated when it comes to automations and the internal handling of when/how things fall back to grid, etc.

Thanks for the advice. To be honest, creating a few Ecoflow automations that each run 24/7 and just activating/deactivating the one I want using HA would achieve what I need perfectly.

My situation is that I have both grid-tied solar (I.e. with an inverter supplying AC to the house) as well as some DC solar going directly into the Delta Pros. The AC coupled solar can export to the grid, and so I monitor that with a Shelly CT and use HA to dynamically raise or lower the Delta Pro charging level to match the grid export/import and make sure I’m charging just enough to balance the generation and self-consume all the grid-tied solar. This works well when I’m generating more than I’m using, but I also want the batteries to power the SHP circuits so that I’m also not drawing from the grid when my usage is higher than the generation. Unfortunately the Smart Home Panel is only ever using a Delta Pro to power the SHP circuits or to charge it, one Delta Pro can’t do both things at the same time. I get round this because I have two Delta Pro’s, so one can be soaking up excess solar while the other one is supplying the SHP circuits, and so I it keeps the house balanced, neither exporting nor importing. However, I also need to alternate the two Delta Pro’s through the day so I don’t end up with one full and one empty, and I do this through an EcoFlow automation on the SHP that periodically swaps Port 1 or Port 2 to be charging. The only way I can do this at the moment is to just have alternating time windows for each port in the Ecoflow rules and just hope that each battery get roughly the same amount of opportunity to charge versus discharge. I also realise this is not efficient to be both charging one Delta Pro and dischaging the other at the same time when the AC could have just been used directly.

Hence what I really need to be able to do is automate when the SHP is charging on Port 1 / Port 2 or both/neither, depending on whether I am generating more than I’m using or less and depending on which battery is lowest.

I also am on an electricity tariff where my overnight cheap rate sometimes starts earlier in the evening, not always a fixed time. My energy provider has a HA sensor that I can use to trigger an automation when this cheap rate kicks in, but I would like to automatically switch from all circuits on battery to both batteries charging when the cheap rate starts, but at the moment I can only achieve this with a fixed time automation in the SHP EcoFlow rules, not from HA, so that’s a second reason why a SHP integration would be really useful.

[Ne0-Hack3r]

The specific YAML package file is here: https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_integration.yaml

Can you add a few raw json messages from mqtt to you repo, please.

@tolwi

https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_raw.json

In addition to those samples there is one that rotates through on a less frequent basis unless a change is made to the charge/discharge limits. In my YAML package I use an automation that is triggered by this params.id to repost it to a separate topic on my local MQTT with a retained flag. You may have a better way to handle that within the integration. This was a bit of a work around to avoid long periods of the sensor sitting in "unknown" status when things got reloaded. a kluge at best.

{"params":{"forceChargeHigh":100,"cmdSet":11,"discLower":5,"id":30},"version":"1.0","cmdId":0,"cmdFunc":0,"id":1585896662957696715,"addr":0,"timestamp":1699358670545}

As with all the other devices a payload can be posted to ../get requesting "latestQuotas" which will return a large payload on ../get_reply containing the current state of all sensors. This is great for pre-populating sensor data but I did not think polling constantly for that was a particularly elegant way to handle it within a YAML package (basically using an automation to post the request every X seconds). Probably not too 'friendly' to the cloud mqtt server and possibly a way to get the account flagged as the app only posts that once when entering the screen for a given device.

[Ne0-Hack3r]

@dafmeuk

Sounds like you are running 2 DPs on SHP in single phase as am I. I use an automation along with a smart plug attached to a 1400W space heater plugged into a circuit supplied by SHP to "induce" parallel mode which has the effect of using both DPs for 1 minute after which the DP at lower SOC disengages...

alias: SHP - Load Balance
description: SHP - Load Balance
trigger:
  - platform: numeric_state
    entity_id: sensor.dp_all_soc_delta
    for:
      hours: 0
      minutes: 1
      seconds: 0
    above: 10
condition:
  - condition: numeric_state
    entity_id: sensor.shp_all_pro_output
    above: 300
  - condition: state
    entity_id: binary_sensor.shp_pro1_ac_charging
    state: "off"
  - condition: state
    entity_id: binary_sensor.shp_pro2_ac_charging
    state: "off"
  - condition: or
    alias: Only one DP discharging
    conditions:
      - condition: and
        alias: Only DP1
        conditions:
          - condition: state
            entity_id: switch.dp1_ac_output
            state: "on"
          - condition: state
            entity_id: switch.dp2_ac_output
            state: "off"
      - condition: and
        alias: Only DP2
        conditions:
          - condition: state
            entity_id: switch.dp1_ac_output
            state: "off"
          - condition: state
            entity_id: switch.dp2_ac_output
            state: "on"
action:
  - service: switch.turn_on
    data: {}
    target:
      entity_id: switch.basement_plug
  - wait_template: "{{ int(states('sensor.shp_all_pro_output')) > 1900 }}"
    alias: Wait up to 30 seconds for AC output >1900W
    timeout: "30"
    continue_on_timeout: true
  - service: switch.turn_off
    data: {}
    target:
      entity_id: switch.basement_plug
  - delay:
      hours: 0
      minutes: 9
      seconds: 0
      milliseconds: 0
mode: single

This uses a combination of my YAML package for SHP, the hassio (local) integration for DP, the integration for my smart plug, and a custom sensor I built called 'dp_all_soc_delta' that tracks the SOC differential between the two DPs...

- platform: template
  sensors:
   dp_all_soc_delta:
    friendly_name: "ALL DP SOC Delta"
    unit_of_measurement: "SOC%"
    value_template: >- 
     {{ (states('sensor.dp1_soc_avg') | float - states('sensor.dp2_soc_avg') | float) | round(1) | abs }}

[tolwi]

@Ne0-Hack3r can you make some pre-check with 140-new-device-smart-home-panel branch? I've implemented everything I could understand from your YAML. However, I need feedback because I can't really check if it works at all.

[Ne0-Hack3r]

@Ne0-Hack3r can you make some pre-check with 140-new-device-smart-home-panel branch? I've implemented everything I could understand from your YAML. However, I need feedback because I can't really check if it works at all.

@tolwi

I'll download the updated code and check it out...

[Ne0-Hack3r]

@tolwi

Out of the gate I've got a bit of an issue in that the add device dialog has no field names and does not allow selecting SHP...

I simply replaced the existing /ecoflow_cloud/ folder under /custom_components/ with the same folder from the the 140 branch of the repo... if there was something else to update, let me know.

[Ne0-Hack3r]

@tolwi

It appears this issue getting a new device setup may be unrelated to the code changes for SHP. I restored the original code from main branch and restarted HA and it does the same thing. I'm a bit behind on OS/Core updates for HA so I'm running a full backup (which is going to take a while) and then I'll get everything current and see if that resolves things. In the meantime if this is a known issue or you know what's going on, please let me know how to resolve it.

[Ne0-Hack3r]

@tolwi

I got the issue sorted out and was able to apply your updated code and add my SHP...

These sensors appear to be working...

The rest do not appear to be functional at this point.

I'm also noticing significant lag on my dashboards for DP and SHP that use my YAML packages with my local instance of mosquito (which is bridged to mqtt.ecoflow.com). Not sure if that is server related or perhaps related to running so many connections to the same server. Mosquito makes a single connection for all my devices (2xDP, D2, D2M, SHP). It appears your integration makes a connection per device, is that correct?

[Ne0-Hack3r]

@tolwi

After disabling the SHP device in the integration my existing dashboards returned to normal. Looking at both sides of the MQTT connection I suspect the logins and/or connection resets are stepping on each other. Due to issues with closing the native app causing existing mqtt connections to go stale I have an automation running that restarts mosquito if certain data updates get older than 1 minute... I suspect your integration does something similar? And yours and mine may be fighting each other...

[tolwi]

@Ne0-Hack3r Yes, there is a reconnect logic (in StatusSensorEntity), but it was not enabled for SHP. In my recent updates I've added a status sensor (with some metrics in attributes), fixed switches and selects, and generated docs

[Ne0-Hack3r]

@tolwi

The updated code triggers an exception when loading the integration:

2023-11-16 14:56:45.866 INFO (SyncWorker_1) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Login to EcoFlow API https://api.ecoflow.com/auth/login
2023-11-16 14:56:46.043 INFO (SyncWorker_1) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Successfully logged in: XXXXXX
2023-11-16 14:56:46.043 INFO (SyncWorker_1) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Requesting IoT MQTT credentials https://api.ecoflow.com/iot-auth/app/certification
2023-11-16 14:56:46.200 INFO (SyncWorker_1) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Successfully extracted account: app-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-11-16 14:56:46.224 INFO (MainThread) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Connecting to MQTT Broker mqtt.ecoflow.com:8883
2023-11-16 14:56:46.628 INFO (Thread-46 (_thread_main)) [custom_components.ecoflow_cloud.mqtt.ecoflow_mqtt] Subscribed to MQTT topic /app/device/property/SP0000000000000
2023-11-16 14:56:52.815 ERROR (Thread-46 (_thread_main)) [root] Uncaught thread exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 1164, in _loop
    rc = self.loop_read()
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 1556, in loop_read
    rc = self._packet_read()
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 2439, in _packet_read
    rc = self._packet_handle()
         ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 3033, in _packet_handle
    return self._handle_publish()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 3327, in _handle_publish
    self._handle_on_message(message)
  File "/usr/local/lib/python3.11/site-packages/paho/mqtt/client.py", line 3570, in _handle_on_message
    on_message(self, self._userdata, message)
  File "/config/custom_components/ecoflow_cloud/mqtt/ecoflow_mqtt.py", line 267, in on_json_message
    self.data.update_data(raw)
  File "/config/custom_components/ecoflow_cloud/mqtt/ecoflow_mqtt.py", line 160, in update_data
    self.__broadcast()
  File "/config/custom_components/ecoflow_cloud/mqtt/ecoflow_mqtt.py", line 164, in __broadcast
    self.__params_observable.on_next(self.params)
  File "/usr/local/lib/python3.11/site-packages/reactivex/subject/subject.py", line 59, in on_next
    super().on_next(value)
  File "/usr/local/lib/python3.11/site-packages/reactivex/observer/observer.py", line 39, in on_next
    self._on_next_core(value)
  File "/usr/local/lib/python3.11/site-packages/reactivex/subject/subject.py", line 66, in _on_next_core
    observer.on_next(value)
  File "/usr/local/lib/python3.11/site-packages/reactivex/observer/autodetachobserver.py", line 28, in on_next
    self._on_next(value)
  File "/config/custom_components/ecoflow_cloud/entities/__init__.py", line 101, in _updated
    if self._update_value(values[0].value):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/ecoflow_cloud/devices/smart_home_panel.py", line 28, in _update_value
    return super()._update_value({"sta": val['sta'], "ctrlMode": val['ctrlMode']})
                                         ~~~^^^^^^^
KeyError: 'sta'```

[Ne0-Hack3r]

@tolwi

I did not deep dive into your code so not sure what context the offending function was called in. Perhaps I can provide some sort of recording of live payloads or we can derive some secure way for you to tap into my live data (without giving away my EF creds or serial number). If you'd like to collaborate more directly/privately, let me know.

[Ne0-Hack3r]

@tolwi

I have setup an mqtt bridge to provide you with access to live sensor data from my SHP. It will not allow posting of settings changes (../set) for obvious reasons and topic paths use a generic UID/SN.

/app/device/property/SP99999999999999'
/app/9999999999999999999/SP99999999999999/thing/property/get'
/app/9999999999999999999/SP99999999999999/thing/property/get_reply'

Your code will need to be tweaked to allow using an alternate URL (other than mqtt.ecoflow.com) for the SHP test device but this setup should basically emulate Ecoflow's MQTT as it is simply bridging the messages from their cloud for the above topics and remapping to use '99999' to hide my private info.

If you are interested in utilizing this mechanism for your test/dev of SHP support please let me know how we can connect privately and I will provide you with URL and MQTT login info.

[benvanik]

I'm just getting started with HA and found this while trying to get my SHP integrated - I've got the last release installed via HACS but haven't figured out how to switch branches yet and before I embarked on that journey I wanted to check what the state of the branch is and if it's recommended for testing. I think I see how to add more sensors and would love to contribute those (watts/circuit, etc).

[mlamoure]

Also have a SHP and am willing to test or provide data. I'm hoping to specifically get a switch that performs the acton that the "recharge" toggle in the EcoFlow app.

For those with models where the app does not support the new Automation features (which is ridiculous), HA integration for recharge control could provide a replacement.

[Ne0-Hack3r]

I have heard nothing since my initial testing of the branch with partial SHP support.

[cirrusflyer]

Is there anything available to just get the sensor data?

[Thomptronics]

Really hoping support for the SHP happens.

[foxthefox]

The specific YAML package file is here: https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_integration.yaml

Can you add a few raw json messages from mqtt to you repo, please.

@tolwi

https://github.com/Ne0-Hack3r/ef-mqtt-hacks-ha/blob/main/shp/shp_raw.json

Would it be possible to get access to the raw.json? I have managed so far the iobroker adapter for several devices, but shp is missing, due to lack of information. I have started to port the basics of the api-handling into a node-red package using mqtt and autodiscovery for HA. It works quite well and as an intermediate solution I will make a repo for that. I have started to work on this repo, but I am not so familiar with python as well as HACS adapter development.

[foxthefox]

@Ne0-Hack3r If my question for getting access is to much, may it be possible to post the json data?

[Ne0-Hack3r]

@cirrusflyer @foxthefox

Invites to my private repo have been sent to you both. There are sample JSON payload as well as YAML packages and sample dashboards configs there for SHP. I have not developed a full grown HA integration but I am still willing to aid in augmenting this one to include SHP if the author is so inclined.

[foxthefox]

Thanks a lot, interstingly data structure, huge differences to the devices where it has clear parts (pd, inv, mppt...). If there is also a "latestQuotas" telegram available, it would be very favourable if you put that also into the folder.

[cirrusflyer]

I'm not experienced with manually doing this. Is there a quick couple steps on how to add this to an existing HACS install?

[Ne0-Hack3r]

I'm not experienced with manually doing this. Is there a quick couple steps on how to add this to an existing HACS install?

It is not a HACS integration. It is a custom YAML package. I'm not aware of a HACS integration for SHP (which is why I offered to help expand this one to include SHP).

To use my package you have to install and configure Mosquito Broker (for MQTT) drop the YAML into your config/packages folder (assuming you've defined that in your configuration.yaml) and restart Home Assistant. I've also provided YAML for a basic dashboard that can be cut and pasted into a new dash board (as RAW YAML). I've documented these steps in the repo but this is more of a template for a power user that requires some coding skills and depth of knowledge in Home Assistant to modify and customize beyond what I've provided.

[dafmeuk]

@Ne0-Hack3r , you previously granted me access to your repo to try your YAML but I never managed to grab it and the invite has now expired. Any chance you could re-share please?

[Ne0-Hack3r]

@dafmeuk - done!

[nickwinger]

+1

[nickwinger]

When will we see a release ? :)

[nickwinger]

will it be possible to switch the delta pros charging on/off ?

[nickwinger]

@Ne0-Hack3r ni, could you please also provide the yaml for me, maybe public for everyone? I also need to control the shp from home assistant. Would it be possible to make a new integration just for the shp as there is nothing going on no more here ? Or make a fork ? Best regards

[Ne0-Hack3r]

@Ne0-Hack3r ni, could you please also provide the yaml for me, maybe public for everyone? I also need to control the shp from home assistant. Would it be possible to make a new integration just for the shp as there is nothing going on no more here ? Or make a fork ? Best regards

I don't plan to fork this repo as there is now an official open API from Ecoflow. They are updating support for SHP at this point. I've invited you to my private repo which has code, examples, and instructions for mosquito/MQTT, same dashboards, etc. if you wish to continue to pursue this approach. I don't plan to make that repo public as it utilizes unactioned and unsupported access to public MQTT that could be changed by Ecoflow at any time without notice and my purpose was to share this collaboratively not to create and maintain an official HA or HACS integration (rather to provide a jumpstart for others to create their own packages or integrations). I submitted initial code change and testing to add SHP support to this repo (HACS integration) but the author never responded so I consider that effort dead at this point.

[cirrusflyer]

Are you saying that the hassio-ecoflow-cloud will likely support the SHP?

[Ne0-Hack3r]

Are you saying that the hassio-ecoflow-cloud will likely support the SHP?

I have no reason to anticipate SHP support by this project at this point as there has been no progress toward that for a very long time. I also have no plan to fork this repo and take on support for this integration. The more productive effort, at this point, may be to create a new integration based around the official API. The only drawback I see is that Ecoflow is painfully slow adding support for new devices and/or updating the API in general. The SHP1 support is still incomplete. SHP2 and DPU support were promised months ago and there is still no update. No telling how long it might be before DP3 support is added, etc.

So we are left with ongoing reverse engineering and using unsanctioned unsupported methods or waiting protracted periods of time for each device (and it's documentation) to be added to the API. The best way is a robust fully supported API but the other limiting factor is that Ecoflow will only grant a single API key to a single developer for access to the devices linked to their own account and stipulates that the API is for individual private use only. This means any integration would require the end user to sign up as a developer, acquire a key, and input that into the integration. Not near as user friendly as just asking them for their Ecoflow login. So pro's and con's and Ecoflow still not making it "easy" for those of us that want full scale integrations.