Integration RS485 to USB

[mattwilson1066]

Just making this issue to continue the USB discussion

[LucasTor]

Just dropping the link to our last thread for context https://github.com/nathanmarlor/foxess_modbus/discussions/439

[LucasTor]

I found the same adapter you have on Aliexpress, I'll get one for myself so I can pull off some tests as well Maybe you could leave your computer running during the day when you are not at home (not sure how practical that would be tho) to test that small script and make sure the connection with the inverter is established properly Also let me know if you have any questions along the way, so I can help you out

[mattwilson1066]

I'm hoping to have a bit of time Friday to have a go with it as I'll be working from home and see if I can get anywhere with it.

[LucasTor]

Hey @mattwilson1066 Good news on the project, my RS485 adapter just arrived yesterday, I tested it and I was able to pull data from the inverter using a code very similar to yours, so I guess we are in the right direction Were you able to make any progress with it?

[mattwilson1066]

That's great news, keep us updated with progress.

Slightly annoyingly my inverter occasionally came up with a fault, so I contacted Fox to get a firmware update and it hasn't worked since they updated the firmware, now I get an SCI Commfault. Support have been super helpful so far and are raising a replacement request and just waiting to hear if it's approved.

So until that is replaced, all solar is going through my hybrid inverter which is annoying and I'm not sure how long a replacement might take.

[mattwilson1066]

Hi @LucasTor my inverter is back up and running again and I have enlisted one of the Dev's from work to help me with the USB side of things as its out of my skillset.

The script we are currently using is this, which is just putting the data its read into the log file. You can see there are a couple of commented out versions to get it work, I haven't cleaned the file up until its all fully working.

import serial
import codecs

@service
def read():
    # Define the serial port and baud rate
    serial_port = '/dev/ttyACM1'  # Change this to your serial port
    baud_rate = 9600  # Change this to match your device's baud rate

    try:
        # Open the serial connection
        ser = serial.Serial(serial_port, baud_rate)

        # Continuously read data from the serial port
        #while True:
        #data = task.executor(ser.readline)  # Read a line of data
        data = task.executor(ser.read_until, b'\xE7')
        #print(data)  # Print the data without a newline
        # decoded_data =codecs.decode(data, "hex_codec")   # Decode bytes to a string
        # decoded_data = (codecs.decode, data, "hex_codec")
        log.info(data.hex())  # Print the data without a newline

    except serial.SerialException as e:
        log.error(f"An error occurred: {e}")

    finally:
        if ser.is_open:
            ser.close()  # Close the serial connection when done

These are the current outputs, the problem we currently have is that its stopping on e7, so depending on the string its cutting off too early I would imagine. Hopefully He will help sort that next week, unless you have any suggestions which I can try over the weekend?

7e7e0265cf1e16009600000074000009aa0007137d00740000000000000000000000000000000007350002000005dd00040000000000000000000000000000001c001c002c00a00000010f00000000000000000000000000000000000000000000000000001a650033000000000000000000000020000000000461000700000000000000000000000000000000000000000000000000000000000000000002484ae7

d800960000002e0000099300071382002e00000000000000000000000000000000060c0000000005c600020000000000000000000000000000001c001c002b00024fabe7

But certainly making some progress in the right direction. So any hints, tips or pointers would be greatly appreciated!

[mattwilson1066]

@LucasTor My developer buddy totally nailed it and the output I got was... the below, so I think we're ready to progress??

7e7e0265d43af50096000000fe000009c1000b138f00fe000000000000000000000000000000000649000600000790000700000000000000000000000000000016001600250015000001b90000000000000000000000000000000000000000000000000000214e001d0000000000000000000000200000000002f3000400000000000000000000000000000000000000000000000000000000000000000002714de7e7

import serial

@service
def read():
    # Open the serial port
    ser = serial.Serial('/dev/ttyACM1', 9600)  # Adjust port and baud rate as per your requirement

    start_marker = b'\x7e\x7e'  # Start marker
    end_marker = b'\xe7\xe7'    # End marker

    data_buffer = b''  # Buffer to store received data
    while True:
        # Read data from the serial port
        data = task.executor(ser.read,1)

        # Append the read data to the buffer
        data_buffer += data

        # Check if the start marker is found
        if start_marker in data_buffer:
            # Find the index of the start marker
            start_index = data_buffer.index(start_marker)

        # Check if the end marker is found
        if end_marker in data_buffer:

            # Extract the complete message
            complete_message = data_buffer[start_index:]

            log.info(complete_message.hex())
            event.fire('inverter_data_read', data= complete_message.hex())
            break

    # Close the serial port when done
    ser.close()

[LucasTor]

Hey @mattwilson1066, that's great news, that's exactly the payload we need from the inverter, the only problem I see with those is that we can't really use a while loop because it freezes Home Assistant (don't ask me how I know), but I think we can work around that using a different thread to read the serial port, I also have my RS485 adapter to USB on hand now, so I can also help out on some tests

I will mock something up in a separate branch here in this repo so we can test that out and I'll let you know, thanks for all the help!

[mattwilson1066]

@LucasTor not a problem, very glad I was able to help. I did try using thread and asyncio but didn't fully know what I was doing. Let me know when you've had the chance to knock something up to test. I can certainly call on my Dev buddy to help where needed.

[LucasTor]

Hey @mattwilson1066, sorry for the huge delay, my university classes just started back and things are really busy over here, I just finished implementing what I think would work for you and me, I did some basic testing with mock data, tomorrow I plan on testing with my USB - 485 adapter and I will let you know if it works, in case it does, I will commit it here into Github so you can git it a shot as well

Are you able to run it in a sandboxed environment for the first tests? It could be a docker container, a VM on your PC, whatever you feel like, as things can fail pretty catastrophically if it ends up in an infinite loop somewhere (tho it should not)...

[LucasTor]

Lol, I just hit the dumbest roadblock possible, when using MacOS, you cannot share devices between the host and the container, which means I can not properly test the integration on my Mac I am now downloading Ubuntu to run on my old PC and test it, but sadly I won't be able to complete it today, as I was gonna test in my lunch time (in which the inverter is on)

[LucasTor]

Hey @mattwilson1066, I just did some tests and it seems to be working as expected with the USB dongle, I guess at this point you could try and use it as well (I would first test in a VM or something to make sure everything looks good) I still did not give much attention to the config flow, so when setting the integration up, just set a 0 to the host and port (otherwise they get the defaults for some reason), and set the correct USB port in the 3rd field (ex: /dev/serial/by-id/USB_Serial_Adapter-123456) After that, if it was able to connected, 32 sensor should appear in the integration, and in my case they update every ~1 minute

[mattwilson1066]

Hi @LucasTor this is great progress and exciting times!! I've just updated the integration, gave it the USB port and can now see the 32 sensors appear. I don't have any easy way of running it in a container and am also on a Mac, so I thought i'd just throw caution to the wind, especially as you did some testing already. I stuck a couple of the entities on a dashboard so I could monitor their values. After a few minutes, some still showing as unknown, but some of them have updated but I am getting 0W as the values, and I don't think mine are pooling every minute. Looking at the logs its throwing some errors for me, thats probably why mine isn't updating. any ideas?

[LucasTor]

That's great, we definitely have progress

It seems like it's failing to capture the message between the 7E7E and E7E7, also I added some protection to prevent an infinite while loop (which is the one you see it saying the port is flooding and it's skipping) which would cause homeassistant to turn unresponsive (not even the frontend loads anymore)

You could try to run your script you posted a couple days ago to validate the communication is working properly between the PC and the inverter, I wonder if there's infact more garbage going through that port than only the 7E7E messages, which might cause this to happen, how does the "RX" led on the adapter blink? Mine blinks every ~20s (but 2 messages are discarded as they are not what we want)

Do you mind capturing the debug logs for a couple minutes and sending me that? You can do that by opening the integration and clicking "Enable debug logs", once you disable it a file should be downloaded, just make sure there's no sensitive info in there

Also the one sensor I usually look at to validate if we have data coming in is the Gen Power, a lot of them report as 0W for me as I don't have the mains measuring stuff attached to my inverter, but not sure it's your case

[LucasTor]

Also I tried decoding the payload you posted above manually and I got the correct data, so once we figure out this communication problem we should be golden

timestamp: "2024-02-20T02:39:01",
grid_power: 0,
gen_power: 254,
load_power: 0,
grid_voltage_R: 249.7,
grid_current_R: 1.1,
grid_frequency_R: 50.07,
grid_power_R: 27467,
grid_voltage_S: 0.0,
grid_current_S: 0.0,
grid_frequency_S: 0.0,
grid_power_S: 0,
grid_voltage_T: 0.0,
grid_current_T: 0.0,
grid_frequency_T: 0.0,
grid_power_T: 0,
PV1_voltage: 160.9,
PV1_current: 0.6,
PV1_power: 0,
PV2_voltage: 193.6,
PV2_current: 0.7,
PV2_power: 0,
PV3_voltage: 0.0,
PV3_current: 0.0,
PV3_power: 0,
PV4_voltage: 0.0,
PV4_current: 0.0,
PV4_power: 0,
boost_temperature: 22,
inverter_temperature: 22,
ambient_temperature: 37,
todays_yield: 2.1,
total_yield: 44.1,

[mattwilson1066]

These are the only things logged which refer to the integration which is repeated quite a few times. There is a mention of it starting up, then just these repeated.

2024-03-14 14:58:07.514 WARNING (Thread-124) [custom_components.foxess_tseries.sensor] Serial port flooding, skipping...
2024-03-14 14:58:08.521 ERROR (Thread-125) [custom_components.foxess_tseries.sensor] Unknow error
2024-03-14 14:58:08.521 ERROR (Thread-125) [custom_components.foxess_tseries.sensor] cannot access local variable 'start_index' where it is not associated with a value

[LucasTor]

@mattwilson1066 I think I figured out what must be happening to the integration, I just pushed a possible fix for it, do you mind giving that a shot?

Also, these are only warnings and errors, but you can activate like a "deeper" level of logging using the debug feature on the integration, which by default is not shown (no need to do that now tho, as I think the new commit might fix your issue)

[mattwilson1066]

Thanks for looking at that so quickly, its certainly getting better, I do not get the warnings now and its getting some figures and values, but I'm not 100% sure how accurate they are or if they are associated with the wrong things. I have attached a screenshot of all the entities that seem to have values which I will keep an eye on to see if they update. Things like 'Todays Yield' is not correct as I have done 18kWh of solar today and I have certainly not done the total yield. But I have values, which is a massive steph forward.

[mattwilson1066]

@LucasTor what I did just think was that my inverter was updated to the latest firmware since we ran the initial USB tests and that payload you decoded. I wonder if thats had any changes to the data structure some how.

[LucasTor]

That could make sense, would you be able to get another raw payload from the inverter? If that's the case, we'd have to contact FoxESS and ask for the updated documentation, then we can add a "payload version" field or something to the integration config so it works with all versions

I know this was the case for the guys who implemented this with ESPHome as well see this thread comment

[mattwilson1066]

Yeah I remember reading something before about them changing the payload. I can contact the guy who did the firmware update, although I ended up having to do the firmware update manually as them doing it remotely bricked the new inverter they sent me. So it had been offline for a week or so for a second time, but got it back up and running over the weekend.

7e7e0165f32cd9006c312e37330000312e30360000322e34360000302e3030000000014600000000000000463436303000000000000000000000000000000000000000000011f800030000312e37330000312e30360000322e34360000302e303000000000000000000000000000000000000000008ff1e7e7

[mattwilson1066]

My F4600 is now running the following firmware versions. Master 1.73 Slave 1.06 Manager 2.46

Before the FW update, I was running Master 1.31 Slave 1.05 Manager 2.18

[LucasTor]

I was just reading through the forum, and apparently, 6 bytes were added to the start of the string, maybe it's just a matter of offsetting the string by this amount, also the payload you just captured is not what we are looking for, it should start with 7E7E02, not 7E7E01, which is the inverter reporting generation data to the cloud, it would be great if you could capture another one with the correct header Sorry for the confusion

[mattwilson1066]

Thats interesting, as I think the code was looking for any value starting 7e7e, but if it needs to start 7e7e02 we should specify that in the start marker? I just ran it again and this time it gave me a 7E7E02, so hopefully this is better.

7e7e0265f3323d00ee00000000000000510000000009930007138d005100000000000000000000000000000000057a0001000006be00020000000000000000000000000000001e001e002d00ba000006b000000000000000000000000000000000000000000000000000000000000000000000002901af01980000000000000542001f0000000000000000000000000000000000000000000000000000000000000000000200020000000000000000000000000fea000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a087e7e7

[LucasTor]

I guess that was it, the 6 new bytes I still have to figure out about the Generation Power that is somewhere else according to that thread on the forum About the function code (02), I have a check in place for it in the function I use to validate the payload here

Here's the somewhat adjusted decoded payload, I think we are going in the right direction as you said you generated 18kWh today Later today I will add a config for payload version and do the necessary adjustments to the decode code, and I will let you know so you can test it again

timestamp: "2024-03-14T14:22:05",
grid_power: 81,
gen_power: 0,
load_power: 0,
grid_voltage_R: 245.1,
grid_current_R: 0.7,
grid_frequency_R: 50.05,
grid_power_R: 17157,
grid_voltage_S: 0.0,
grid_current_S: 0.0,
grid_frequency_S: 0.0,
grid_power_S: 0,
grid_voltage_T: 0.0,
grid_current_T: 0.0,
grid_frequency_T: 0.0,
grid_power_T: 0,
PV1_voltage: 140.2,
PV1_current: 0.1,
PV1_power: 0,
PV2_voltage: 172.6,
PV2_current: 0.2,
PV2_power: 0,
PV3_voltage: 0.0,
PV3_current: 0.0,
PV3_power: 0,
PV4_voltage: 0.0,
PV4_current: 0.0,
PV4_power: 0,
boost_temperature: 30,
inverter_temperature: 30,
ambient_temperature: 45,
todays_yield: 18.6,
total_yield: 171.2,

[mattwilson1066]

Looking at the Fox Cloud data on their app, the adjusted values look like your on the right track. It's starting to get dark here so I'm not getting all the PV values now, but I can compare them tomorrow when it's sunny. If you are able to provide a payload version selector on setup that would be amazing and certainly give some future proofing if there are other variations which needs accommodating. This is such good progress and very close to getting it nailed!!

[LucasTor]

Great, it's looking good already, hopefully, I'll be able to add this feature tonight, so you can test it tomorrow, as I will be off tomorrow because of some family health issues, and prob on the weekend as well, so hopefully this fixes it for you so you can start using the integration with the correct data

I appreciate all the fast responses and the help you gave me today in figuring out the issues

[LucasTor]

Hey @mattwilson1066, just implemented the payload version option, you'll likely have to remove the integration and add it back (it should not be a problem tho as up until this point we only had garbage data), please still fill IP/Port with 0s, fill in your serial port, and use payload version 1, and if everything goes right, the payload should be decoded in the newer firmware format

I was not able to test it, but it was a simple fix and it shouldn't cause any issues, fingers crossed I should really write some tests to this integration lol

Good luck and please let me know once you test it :)

[mattwilson1066]

That is great work @LucasTor here is what i'm seeing this morning, the payload selector worked nicely. Not sure why I am not seeing Gen Power, currently I think it should be around the same as Grid Power and Grid Power R, also not seeing the PV power. Having a look at the Fox app, I don't have anything similar to Gen Power listed, only the PV1 and PV2 power. I think my combined total would be the RPower. Also sorry to hear about the family health issues, I hope things improve!!

[LucasTor]

Hey @mattwilson1066, sorry for the delay in the response, how is the integration behaving these last few days you have been using it? I just checked and apparently gen power is being read as grid power for some reason, I'll add that to the list of things to fix About the PVs powers, I'll calculate them in code based on the voltages and currents, so we can have that data as well Also, I think I can calculate the grid power based on grid power R + S + T

[mattwilson1066]

Hi @LucasTor no need to apologise at all. The integration has been stable the whole time thanks. I wonder if its because the payload from the F series is slightly different to the T series that things are being read slightly differently? It's certainly not a blocker as I can see the gen power and comparing it to the other devices in the system I know its correct which is great, and daily and total yield are great to have. It would be interesting to graph the PV powers, but I don't plan on doing any automation based on the individual PV powers so certainly not a priority. Have you got your inverter up and running now or are you still using test data?

[LucasTor]

Great to know it is working for you! Yes! I did get my inverter installed a couple months ago actually, and I already had to hack it because WEG (the company who rebrands FoxESS here in Brazil removed the RS485 port from the aux connector and left it only for the wifi dongle, so I had to open the wifi dongle and tap in to the 485 port) Here is how the setup looks:

Also, I think the most important data for this integration is the Today's yield, which is the one you add to the energy dashboard, I have mine paired with an Eastron SDM120CT with ESPHome to get my home energy usage data as well, so I can generate these graphs in the energy dashboard

[KickingAnimal]

Hey, great stuff! Am I reading correctly that RS-485 - USB is implemented and working?

I have a T3-G3 and i'm so infuriated by their cloud app and not being able to see what my meter reads compared to the inverter. if this works it would be awesome, and I'll be willing to test/help as far as i have the time and knowledge.

I havent even messed about with HA itself yet just exploring what it possible with this inverter

I have managed to set up HA Core on my debian machine and it seems like this is all working.

how does one add everything in the energy view? - I do not see the entities listed when configuring

my apologies if this is basic stuff that I am asking

[mattwilson1066]

Hi @KickingAnimal so I have an F series and Lucas has a T series. To access via USB, you need take a feed from the wifi dongle, and that then feeds into USB. I'll find a couple of photos and share them for what you would need to do. Obviously doing this is not covered under the manufactures warranty. Definitely get HASC setup in Home Assistant, that way you can load in the integration. If you setup or edit the Energy view, it asks which entities it takes the values from, so you can put in the battery status, the amount of solar. You can do this from the cloud version, its just not real time and it can be quite hit and miss how often it updates. Where as the RS-485 to USB is so good.

[mattwilson1066]

You need to open up the WiFi dongle and solder as below to access the RS485 Comms. This route also leaves the cloud portal working. Then mine is wired down into my Pi using the Waveshare USB to RS485

There is also a bit more info in the original thread we started before this integration came along which can be found here https://github.com/nathanmarlor/foxess_modbus/discussions/439

[KickingAnimal]

Hi!

Thanks for the responses. In the meantime I did some stumbling around with HA and manuals. My T series has a com port so I did not need to open anything up. So far everything seems to be working.

I think I got confused with the readme.md and this thread with if the usb support worked or not. Seems like it does!

[LucasTor]

Hey guys! Sorry for the delayed response, I am currently using the integration on WiFi, so thanks @mattwilson1066 for letting us know everything is working as expected with USB, I had to push an update a couple of days ago because the integration broke on the new version of home assistant once I updated, but everything should be back to normal now

Considering the USB integration is working and validated by you both @mattwilson1066 and @KickingAnimal, should we close this thread and update the readme?

[KickingAnimal]

Hi,

The Home Assistant update did not break the intigration for USB, however I just installed the update you pushed and things broke? I have attached a screenshot

I tried to add the intigration again but that resulted with no entities shown.

[mattwilson1066]

Thats interesting, I have installed the latest update of this integration, but I have not yet done the HA updates. It looks like there is a HA core update to 2024.5.2 and a Supervisor update to 2024.05.1 which I have not done yet. @KickingAnimal have you done a full restart of your HA setup, to see if the comms start working?

[KickingAnimal]

Yes a restart is required after the update.

There may be another issue.

I had updated HA a few days ago without any issues and today I updated the intigration and this was the result. It might be a combination of things?

[KickingAnimal]

I see now that it seems to have lost comms to the serial TTY. Let me investigate further

Edit, seems like my system decided that the adapter wasn't on /dev/ttyACM0 anymore but ACM1?.

Very strange have never seen this behaviour before. I did not touch anything. All seems to be working with it set to the right serial port.

[mattwilson1066]

Thanks for updating us on what you found. I guess when I do the next HA updates I may run into the same issue and I will report back when I do and if I have the same issue. But maybe something strange in the latest update as you wouldn't expect it to change the adapter location. You would only expect the change if you had physically plugged the USB into the other port?

[hnkvhf]

"Edit, seems like my system decided that the adapter wasn't on /dev/ttyACM0 anymore but ACM1?."

I have seen this before, only on devices running some form of Linux. (Sat-decoder, router) And if it's repeating, from my old memory I think you have to add the port to 'fstab' edit: If you add it to fstab you can switch the usb to another port and it will keep the same portname, if it's not in use

• How fstab works – introduction to the /etc/fstab file on Linux 29 May 2020 by Egidio Docile The /etc/fstab file is one of the most important files in a Linux-based system, since it stores static information about filesystems, their mountpoints and mount options. In this tutorial we will learn to know its structure in details, and the syntax we can use to specify each entry in the file.

https://linuxconfig.org/how-fstab-works-introduction-to-the-etc-fstab-file-on-linux

[mattwilson1066]

Hey guys! Sorry for the delayed response, I am currently using the integration on WiFi, so thanks @mattwilson1066 for letting us know everything is working as expected with USB, I had to push an update a couple of days ago because the integration broke on the new version of home assistant once I updated, but everything should be back to normal now

Considering the USB integration is working and validated by you both @mattwilson1066 and @KickingAnimal, should we close this thread and update the readme?

Yeah the USB integration appears to be working smoothly, and it sounds like the issue KickingAnimal had wasn't specifically related to the integration, more the OS deciding to change the ports around. Althought I will do the HA updates myself and see if it still works for me afterwards or if I have the same issue.

Also really helpful info @HvHoof thank you

[mattwilson1066]

I did all the updates required and it is still working as expected. So I think this can now be closed.

[LucasTor]

Edit, seems like my system decided that the adapter wasn't on /dev/ttyACM0 anymore but ACM1?.

@KickingAnimal you can use /dev/serial/by-id/yourdevice if available, this will never change as well

[LucasTor]

I did all the updates required and it is still working as expected. So I think this can now be closed.

That's great @mattwilson1066, thanks for all the help in the development and debugging of the integration, I will close this issue and you guys can open new ones in case you stumble across anything new