ksheumaker / homeassistant-apsystems_ecur

Home Assistant custom component for local querying of APSystems ECU-R Solar System
Apache License 2.0
183 stars 43 forks source link

ECU-C IP config doesn't work #40

Closed nielssteenvoorde closed 2 years ago

nielssteenvoorde commented 2 years ago

Cannot configure the IP address of my ECU-C. If I enter the correct WiFi address when integrations comes up, HA says it doesn't exist. Use version 1.2.5 by means of a HACS installation. Found the following entries in the log file:

This error was caused by a custom integration.

Logger: custom_components.apsystems_ecur.APSystemsECUR Source: custom_components/apsystems_ecur/APSystemsECUR.py:147 Integration: APSystems PV solar ECU-R (documentation) First occurred: 14:29:47 (1 occurrences) Last logged: 14:29:47

Unkonwn error from ECU after issuing cmd=APS1100280002XXXXXXXXXXXXXEND error=[Errno 104] Connection reset by peer. Closing socket and trying again try 1 of 3

For pictures take a look on: https://community.home-assistant.io/t/apsystems-sensor-for-home-assistant-not-visible/370041/33

If i check line 147 in the file APSystemsECUR.py I see: except Exception as err: _LOGGER.warning(f"Unkonwn error from ECU after issuing cmd={cmd.rstrip()} error={err}. Closing socket and trying again try {current_attempt} of {self.cmd_attempts}") if self.reopen_socket: await self.async_reopen_socket() pass

ksheumaker commented 2 years ago

This error means the ECU closed down the socket but didn't respond to the command the integration gave it. [Errno 104] Connection reset by peer. Can you try power cycling your ECU? Sometimes they get into a weird state and don't work correctly, and the only real remedy is to power cycle it.

nielssteenvoorde commented 2 years ago

Yes, I did a power cycle. Button now works 😉. This is a complete new ECU-C and first time using with this integration. I think it must be something else. Logfile now: Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur.config_flow Source: custom_components/apsystems_ecur/APSystemsECUR.py:128 Integration: APSystems PV solar ECU-R (documentation) First occurred: 16:11:54 (6 occurrences) Last logged: 20:13:17

Unhandled exception: Connection lost Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/config_flow.py", line 44, in async_step_user test_query = await ap_ecu.async_query_ecu() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 210, in async_query_ecu self.inverter_raw_signal = await self.async_send_read_from_socket(cmd) File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 128, in async_send_read_from_socket await self.writer.drain() File "/usr/local/lib/python3.9/asyncio/streams.py", line 387, in drain await self._protocol._drain_helper() File "/usr/local/lib/python3.9/asyncio/streams.py", line 190, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost

Maybe try again tommorrow when there is enough ☀️. The other version did work the first time i did only change the name in the APSystemsECUR.py from ….PRO to ECU-C. This version does have other code. I think it must be in this new code.

nielssteenvoorde commented 2 years ago

New try for getting data out of my ECU-C and still can't get this beta integration to work (v.1.2.5). Keeps telling me the IP-adres isn't correct and that isn't right at all. Checked it more than once, reconnected the wifi of my ECU-C twice with no effect, did some power cycles... I think this integration doesn't work with ECU-C anymore... Can't debug what is wrong. You have any suggestions I can try out?

nielssteenvoorde commented 2 years ago

Reinstalled the hacs integration https://github.com/ksheumaker/homeassistant-apsystems_ecur from repository and installed v1.1.2 and this one seems to work. Hope it keeps working the next day havent checked that out. In beta something seems wrong for using with the ECU-C. If ithe data links are broken tommorrow a report back to you. Love this integration when its working ;-)

ksheumaker commented 2 years ago

Thanks for the update, I haven't had much time this week to work on it, but will this weekend, and hopefully have a newer beta that fixes some issues.

nielssteenvoorde commented 2 years ago

All Apsystems integrations are down this next Day. Apsystems app gives information about generating energy so there is current . I'm now using the v1.1.2 integrations which worked yesterday. Version 1.2.5 had issues with ipadress. Logfile: Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur Source: custom_components/apsystems_ecur/init.py:53 Integration: APSystems PV solar ECU-R (documentation) First occurred: 05:32:08 (199 occurrences) Last logged: 08:50:46

Unexpected error fetching apsystems_ecur data: Error using cached data for more than 5 times. Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 228, in check_ecu_checksum checksum = int(data[5:9]) ValueError: invalid literal for int() with base 10: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/init.py", line 77, in update data = await self.ecu.async_query_ecu() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 112, in async_query_ecu self.process_ecu_data() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 254, in process_ecu_data self.check_ecu_checksum(data, "ECU Query") File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 231, in check_ecu_checksum raise APSystemsInvalidData(f"Error getting checksum int from '{cmd}' data={debugdata}") custom_components.apsystems_ecur.APSystemsECUR.APSystemsInvalidData: Error getting checksum int from 'ECU Query' data=b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data return await self.update_method() File "/config/custom_components/apsystems_ecur/init.py", line 96, in update data = self.use_cached_data(msg) File "/config/custom_components/apsystems_ecur/init.py", line 53, in use_cached_data raise Exception(f"Error using cached data for more than {self.cache_max} times.") Exception: Error using cached data for more than 5 times.

tv3 commented 2 years ago

I had same issue yesterday. TCP connection works, but repsonse is empty. It doesn't recover from that situation without a reboot. Investigating alternative way of gathering data form ECU_R_PRO. There is an internal website and some ajax calls embedded. I extracted the queries (replace IP address with your own).

curl -X POST http://192.168.4.8/index.php/realtimedata/old_power_graph -d "date=2022-02-10" | jq -r

weekly/monthly/yearly summaries

curl -X POST http://192.168.4.8/index.php/realtimedata/old_energy_graph -d "date=2022-02-10&period=yearly" curl -X POST http://192.168.4.8/index.php/realtimedata/old_energy_graph -d "date=2022-02-10&period=monthly" curl -X POST http://192.168.4.8/index.php/realtimedata/old_energy_graph -d "date=2022-01-10&period=weekly"

1=service mode. No data to ema site (max 30 minutes)

0=m=normal mode

curl -X POST http://192.168.4.8/index.php/management/set_polling_conf -d "polling_value=1"

nielssteenvoorde commented 2 years ago

Did copy some text out of my logfile. Maybe this will help fixing the problem:

warning logfile 1: Unexpected error fetching apsystems_ecur data: Somehow data doesn't contain a valid ecu_id 10:05:24 – (FOUT) APSystems PV solar ECU-R (aangepaste integratie)

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur Source: custom_components/apsystems_ecur/init.py:95 Integration: APSystems PV solar ECU-R (documentation) First occurred: 10:05:24 (1 occurrences) Last logged: 10:05:24

Using cached data from last successful communication from ECU. Error: Error getting checksum int from 'ECU Query' data=b''

warning logfile 2: Error setting up entry ECU: XXXXXXXXXXX for apsystems_ecur 10:05:24 – (FOUT) APSystems PV solar ECU-R (aangepaste integratie)

Deze fout is ontstaan door een aangepaste integratie.

Logger: homeassistant.config_entries Source: custom_components/apsystems_ecur/init.py:195 Integration: APSystems PV solar ECU-R (documentation) First occurred: 10:05:24 (1 occurrences) Last logged: 10:05:24

Error setting up entry ECU: XXXXXXXXXXX for apsystems_ecur Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 327, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/apsystems_ecur/init.py", line 195, in async_setup_entry inverters = coordinator.data.get("inverters", {}) AttributeError: 'NoneType' object has no attribute 'get'

warning logfile 3: Using cached data from last successful communication from ECU. Error: Error getting checksum int from 'ECU Query' data=b'' 10:05:24 – (WAARSCHUWING) APSystems PV solar ECU-R (aangepaste integratie)

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur Source: custom_components/apsystems_ecur/init.py:95 Integration: APSystems PV solar ECU-R (documentation) First occurred: 10:05:24 (1 occurrences) Last logged: 10:05:24

Using cached data from last successful communication from ECU. Error: Error getting checksum int from 'ECU Query' data=b''

It should be great if we (somebody else who is good in coding ;-)) can fix this ' problem' . It looks minor.

PS Cycling the power doesn't work with a ECU-C. Still nothing happens

ksheumaker commented 2 years ago

Can you try the latest beta version? 1.2.10. It's using different socket code and hopefully will fix the problem.

nielssteenvoorde commented 2 years ago

Did a reinstall of the HACS version op this integration and installed v1.2.10. This one doesn't find my EPC-U also. I'm sure it has the same IP adres. Checked the modem, checked with angry-IP and check if the local portal of my ECU-C was running and giving the same ip for the wifi. The normal version does work for one day. I post some pictures on: https://community.home-assistant.io/t/apsystems-sensor-for-home-assistant-not-visible/370041/33.

Logfile 1:

Unhandled exception: [Errno 113] Host is unreachable 15:51:10 – (FOUT) APSystems PV solar ECU-R (aangepaste integratie)

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur.config_flow Source: custom_components/apsystems_ecur/APSystemsECUR.py:154 Integration: APSystems PV solar ECU-R (documentation) First occurred: 15:51:10 (1 occurrences) Last logged: 15:51:10

Unhandled exception: [Errno 113] Host is unreachable Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/config_flow.py", line 45, in async_step_user test_query = await self.hass.async_add_executor_job(ap_ecu.query_ecu) File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 161, in query_ecu sock = self.open_socket() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 154, in open_socket self.sock.connect((self.ipaddr, self.port)) OSError: [Errno 113] Host is unreachable

Logfile 2: Unhandled exception: [Errno 107] Socket not connected 15:50:19 – (FOUT) APSystems PV solar ECU-R (aangepaste integratie)

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur.config_flow Source: custom_components/apsystems_ecur/APSystemsECUR.py:142 Integration: APSystems PV solar ECU-R (documentation) First occurred: 15:50:19 (1 occurrences) Last logged: 15:50:19

Unhandled exception: [Errno 107] Socket not connected Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/config_flow.py", line 45, in async_step_user test_query = await self.hass.async_add_executor_job(ap_ecu.query_ecu) File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 199, in query_ecu self.close_socket() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 142, in close_socket self.sock.shutdown(socket.SHUT_RDWR) OSError: [Errno 107] Socket not connected

ksheumaker commented 2 years ago

This line in the error is the key message:

"/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 154, in open_socket
self.sock.connect((self.ipaddr, self.port))
OSError: [Errno 113] Host is unreachable

Host is unreachable means your HA system can't connect to your ECU. Either you have something wrong in the IP address or networking. Or the ECU stopped responding. Which unfortunately has happened to other people.

Have you tried verifying you can ping that address from your HA system or even use something more advanced like netcat to make sure the ECU is responding on that port?

nielssteenvoorde commented 2 years ago

Have deleted al maps using file editor plugin in customs map en did a new install. Still same problem. When scanned with ip scanner it looks like ther are using the same mac adres. Is that possible? More pictures on HA forum ;-)

nielssteenvoorde commented 2 years ago

V1.1.2 now is working the next day with ECU-C. Yesterday I pushed a new wifi ip-adress and replaced the former ip-adress in the configuration.yaml. I still think it’s strange behaviour but it works for now💪👍.

ksheumaker commented 2 years ago

I'm not surprised by this. The ECUs are not very good devices. This kind of stuff seems to happen where they just stop responding correctly.

I think you should be able to use the latest version 1.2.10 without problem if it works on 1.1.2.

nielssteenvoorde commented 2 years ago

There expanding by APSystems when I'm reading there newsfacts... Everythings crashed around 12 o clock. More on forum HA -> https://community.home-assistant.io/t/apsystems-sensor-for-home-assistant-not-visible/370041/46 . Logfiles see below: warning logile1: Using cached data from last successful communication from ECU. Error: index out of range 11:41:02 – (WAARSCHUWING) APSystems PV solar ECU-R (aangepaste integratie) Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur Source: custom_components/apsystems_ecur/init.py:101 Integration: APSystems PV solar ECU-R (documentation) First occurred: 11:41:02 (1 occurrences) Last logged: 11:41:02

Using cached data from last successful communication from ECU. Error: index out of range

Fault logfile 2: Unexpected error fetching apsystems_ecur data: Cached data doesn't contain a valid ecu_id 11:45:02 – (FOUT) APSystems PV solar ECU-R (aangepaste integratie) - bericht kwam voor het eerst om 11:41:02 en verschijnt 5 keer

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.apsystems_ecur Source: custom_components/apsystems_ecur/init.py:56 Integration: APSystems PV solar ECU-R (documentation) First occurred: 11:41:02 (5 occurrences) Last logged: 11:45:02

Unexpected error fetching apsystems_ecur data: Cached data doesn't contain a valid ecu_id Traceback (most recent call last): File "/config/custom_components/apsystems_ecur/init.py", line 77, in update data = await self.ecu.async_query_ecu() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 151, in async_query_ecu data = self.process_inverter_data() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 310, in process_inverter_data signal = self.process_signal_data() File "/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 283, in process_signal_data strength = data[location] IndexError: index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data return await self.update_method() File "/config/custom_components/apsystems_ecur/init.py", line 102, in update data = self.use_cached_data(msg) File "/config/custom_components/apsystems_ecur/init.py", line 56, in use_cached_data raise Exception(f"Cached data doesn't contain a valid ecu_id") Exception: Cached data doesn't contain a valid ecu_id

Couldn't you do some programming for them ;-)

nielssteenvoorde commented 2 years ago

I'm not surprised by this. The ECUs are not very good devices. This kind of stuff seems to happen where they just stop responding correctly.

I think you should be able to use the latest version 1.2.10 without problem if it works on 1.1.2.

Version 1.2.10 (beta) doen't response on wifi-IP adres and v1.1.2 does. Should i try again?

ksheumaker commented 2 years ago

I put this on the forum, but including it here for reference. I'm not saying it isn't possible, but if the device is in the same state, there shouldn't be a condition where v1.1.2 and v1.2.10 would connect differently. If v1.1.2 connects then v1.2.10 should too. They are both doing basically the same thing at that point. Try the debugging techniques I pasted below to make sure your ECU is behaving correctly outside of the integration. If you don't get a response from the command in netcat, then the ECU is in bad state, and it may recover after some time, or you may need to power cycle it.


The 1.2.10 has changed the socket code in a hope for more stability with the C and ECU_R_PRO devices. But I can tell you weird behavior from the ECUs seems to be the norm more than anything.

We’ve reverse engineered this protocol, and this protocol was intended only for their ECU phone app to connect and initially provision the ECU by your installer. We’ve seen all kinds of issues where the ECU stops responding on the wifi address, it shuts down the service running on port 8899 we are connecting to, or it just spits out bogus data. We’ve not been able to figure out reasons behind any of this - other than bad firmware or quirks in the device we just don’t understand. I’m pretty sure they ever intended it to be queried non-stop like the integration is doing. The devices’s webportal and ability send data to their EMA app usually are not impacted even when the integration fails.

I only have an ECU-R device, so obviously it’s a little hard for me debug issues with the C. To fully make sure your device is working, you need to use something like netcat to test connection to the device.

Here’s an example (nc is in hassio so you should be able to bring up a terminal on your HA instance)

Run this command (replace the with your ECU’s ip)

nc -v <IP address> 8899

Once it connects (which will just be a blank line) type the following:

APS1100160001END

You should then get a response with something like this

APS1100940001<ECUID>01:+10012ECU_R_1.2.19009Etc/GMT-8END

Where will have your ECU’s ID and some additional data. If this fails to work the integration is not going to work in the state your ECU is in. It could be an invalid IP address, or the ECU has shut down the port and a power cycle is necessary.

Hope this helps a little.

nielssteenvoorde commented 2 years ago

APS1100160001END

In terminal (HACS plugin) of HA this commando doesn't work. When I put this commando in putty-client (SSH-session) it also doesn't seem to work. I'll try to get this done before the night comes in ;-)

ksheumaker commented 2 years ago

That command needs to be after issuing the

nc <IP> 8899

If that immediately returns you to a command prompt i.e. (starts with a $) then it couldn't connect. You can try running it like this instead

nc -v <IP> 8899

This is verbose mode. If you get a message about it not being able to connect then the IP address is wrong, or the ECU shutdown the port we connect to (for whatever reason), and a power cycle is probably required.

nielssteenvoorde commented 2 years ago

I understand. It doen't respond when put in nc wifi-ip-adress 8899 (only with spacebar between)

ksheumaker commented 2 years ago

Here's an example of it working, and not working with the -v verbose flag (you will need to press CTRL-C if it works to end the connection)

Screen Shot 2022-02-14 at 10 29 12 AM

nielssteenvoorde commented 2 years ago

in commando prompt nothings shows up.... I do a lot of ssh sessions and this one doesn't response at all...

ksheumaker commented 2 years ago

Did you add the -v option? you should at least get an "open" or "unreachable" kind of message. If not then the ECU could be in a crazy weird state where the socket doesn't really open correctly.

nielssteenvoorde commented 2 years ago

Yes i did. Nowit's saying its open. When I past the command " APS1100160001END" after that nothing happens .... I quit for now. Picture: https://community.home-assistant.io/t/apsystems-sensor-for-home-assistant-not-visible/370041/55

ksheumaker commented 2 years ago

Ok, connecting is half way there, then if you type in the command and press enter, you get nothing? You should at least see it echo what you are typing on the screen.

nielssteenvoorde commented 2 years ago

No response at all. To many rentries it's saying

ksheumaker commented 2 years ago

Too many retries from nc -v XXX.XXX.XXX.XXX 8899 command? Or from the HA integration? At this point we need netcat to work before trying the integration. If netcat is not working the integration never will.

ksheumaker commented 2 years ago

Oh I see your screenshot now. Your ECU is in a bad state. My suggestion is to power cycle and see if that makes a difference.

FYI: CTRL-C is the correct quit command not CTRL-Q.

nielssteenvoorde commented 2 years ago

Oh I see your screenshot now. Your ECU is in a bad state. My suggestion is to power cycle and see if that makes a difference.

FYI: CTRL-C is the correct quit command not CTRL-Q.

CTRL+Q I ment but typed it wrong. My english isn't that good ;-). I'm gonna powercycle the ECU-C again

ksheumaker commented 2 years ago

No worries, your English is way better than my non existent Dutch.

Once you've restarted try the nc commands again. We need that to respond correctly before enabling the integration. It may require sunlight, it doesn't on my ECU-R but I'm not sure on the C.

nielssteenvoorde commented 2 years ago

I did a restart and now the devices comes up but no entitities in HA. No errors so I am gonna try the netcat coding ;-)

nielssteenvoorde commented 2 years ago

now host is unreachable. I think i've to disconnect HA integration, than try netcat coding and if everything is working than do a new install in HA right. But now wait for sunpowerrrrr ;-)

ksheumaker commented 2 years ago

Wow, there is something weird there for sure. I've never seen it go down so quickly. But yes my suggestions are:

A couple of things, do not query this at rate faster than 300 (every 5 minutes).

Do you have other things attempting to query the ECU? Like another home assistant instance, or some other program you are running to try and query it? Too much activity certainly appears to kill the device.

nielssteenvoorde commented 2 years ago

I am gonna try all of the above tommorrow when there is sun on my panels. And no, no other devices connecting to this ECU-C, no other installation of HA, refreshrate always default 300, I'll be back

nielssteenvoorde commented 2 years ago

Hello @ksheumaker @HAEdwin send me a message on the HA forum (also speaks dutch). I'll try to debug this ECU-C by using packet-sender on his recommendation. I'll get back to you and share what is coming out.

HAEdwin commented 2 years ago

Seems that the ECU-C responses are well formed when using packet-sender. Maybe over time the ECU-C might also respond with unexpected data, we discussed this earlier having to anticipate on these unexpected messages. I've seen similar error messages as descibed earlier in this thread.

ksheumaker commented 2 years ago

It seems most of the time his ECU-C is offline though, and not responding at all. Stuff like this:

"/config/custom_components/apsystems_ecur/APSystemsECUR.py", line 154, in open_socket
self.sock.connect((self.ipaddr, self.port))
OSError: [Errno 113] Host is unreachable
HAEdwin commented 2 years ago

@nielssteenvoorde you should give it a try leaving packet sender run for a whole night to see in what state the ECU-C will be next morning. This might rule out difficulties with the 8899 method.

ksheumaker commented 2 years ago

Is the packet sender just sending the first command to the ECU, or is doing all 3? I wonder if one of these is more problematic than the other?

HAEdwin commented 2 years ago

Yes, it sure will be - focus on the signal query, Niels has seven microinverters that makes that the responses from the ECU must be well filtered to ignore unexpected responses inbetween. It indeed sends only the ECU query, but it's a way to show that if the software is capable of handling other responses the ECU and software will continue.

nielssteenvoorde commented 2 years ago

I'm now monitoring with packetsender from 17:15u and check tommorrow. Is there an easy way to send you this logfile? Very happy that you guys are so determined to crack this unit ;-)

nielssteenvoorde commented 2 years ago

See HA forum for more details (in Dutch sorry): https://community.home-assistant.io/t/ecu-c-logfile-op-vaste-ip-adres/393011/12 And I have 6 inverters (3xQS1, 3xYC-600 and 1xECU-C)

HAEdwin commented 2 years ago

@nielssteenvoorde Is it allright if you close this issue and continue here: https://github.com/ksheumaker/homeassistant-apsystems_ecur/issues/67 we also have our dutch e-mail conversation on the Home Assistant website.

HAEdwin commented 2 years ago

This issue can be closed @nielssteenvoorde, great to hear the integration also works on the ECU-C!!

nielssteenvoorde commented 2 years ago

Yes it finally works for more than one day. Great work @ksheumaker and @HAEdwin 💪. Integration now works for owners of a ECU-C. Really happy with this integration and it was a pleasure to do all the testing 👍👌