search

tbnobody / OpenDTU

Software for ESP32 to talk to Hoymiles/TSUN/Solenso Inverters
GNU General Public License v2.0
1.69k stars 471 forks source link

HMS made for American Market: default of 915MHz instead of 865MHz! #1540

Closed Fribur closed 5 months ago

Fribur commented 6 months ago

What happened?

FCC permitted the American DTU-S to communicate on 80 Channel between 905MHz and 925 MHz. Accordingly, HMS-350-1T-NA (=US version of HMS350) is not responding to any signals send by OpenDTU between 860 and 870 MHz (European Channels).

To Reproduce Bug

Get an American HMS (e.g. HMS-350-1T-NA) and try to receive anything between 860 and 870 MHz.

Expected Behavior

I debugged the issue with the following changes:

  1. HoymilesRadio_CMT.cpp: HOY_BOOT_FREQ 915000000, HOY_BASE_FREQ 905000000 (note MIN_FREQ_KHZ needs channel 0 not 1 in case there CMT_BASE_CH_OFFSET860 <=1!)
  2. HoymilesRadio_CMT.h: HOYMILES_CMT_WORK_FREQ 915000
  3. cmt2300wrapper.h: CMT_BASE_FREQ 905000000
  4. cmt2300_params.h re-generated using RFPDK mimicking the githib version but changing base frequency from 860 to 905 MHz. Gladly the sync word and all other parameters appear to be the same for US.
  5. configure CMT GPIO2 and GPIO3 to "-1". This one is weird, as Blinkyparts OpenDTU has them connected to ESP32 GPIO 16, 19, so the interrupts should work, but they do not trigger. Maybe an issue with my CMT, I will order a new one.

After all those changes (and some changes to the GUI to adjust error messages), everything works as expected.

cmt2300a_params.h.txt

Install Method

Self-Compiled

What git-hash/version of OpenDTU?

N/A

Relevant log/trace output

No response

Anything else?

It would be nice to be able to switch between country specific configs: EU: 860-870MHz, default 865MHz US: 905-925MHz, default 915MHz Brazil: 915-928 MHz, default ??? ... If you like I can give it stab to implement all that and convert this into a pull request. Would require some re-factoring as it would need to be possible to re-initialize the CMT2300A while it is running, e.g. could also set it up that OpenDTU does nothing until country is selected.

stefan123t commented 6 months ago

As discussed on Discord, could you supply the HW / FW version and other details OpenDTU queries from the inverter ? Interestingly the Modelnumber sticker on your Model HMS-350-1T-NA: 350SNRSOS.US.HM also mentions US, we should probably document that together with HW / FW versions.

Probably EU would still be the default, unless we make this a compile time choice in order to truly comply with radio/frequency legislation in most countries. Alternatively we could require the user to make a choice after installation, before the OpenDTU would power-up the CMT2300A radio chip in either mode. I do think setting the Country on a DTU level would be sufficient, though.

Adding your changes as a pull-request should give the maintainers more options to review and comment on your proposed / necessary changes. I recall that you were still unsure why you had to comment out some code sections ?

Fribur commented 6 months ago

Together with homeautomation22 (Discord handle) we figured out that the American version of the Hoymiles I have (HMS-350-1T-NA) does not respond to Channel Change Commands (regardless if it is sent on 868 or 915 MHz). All other commands work just fine. The only things that does work, (and it also works with the trunk version here on GitHub) is to move the communication to 915MHz. Can be done via WebInterface. The inverter starts responding to them without prior Channel Change command...apparently that is also different to the Europeans version which (as I understand) requires a Channel Change Commend before responding to any other command. So many of my suggestions and changes above are a bit pointless until someone can trace how an original DTU (American settings) and an American Inverter agree on a new channel. The same way it was done for the Europeans DTU + inverter. I do not have access to an official DTU right now..so I guess resolving this issue for good is stuck right now :-(

Couple of additional inverter data that may or may not be useful:

rj45jack commented 5 months ago

I have 8 HMS inverters in North America. I haven't been able to get OpenDTU to connect and gather data. Is there some sort of trick? I set to 915mhz and, nothing.

I have 7 HMS 2000 and 1 HMS 1000.

stefan123t commented 5 months ago

@rj45jack according to Fribur and homeautomation22 this requires some modification to the code, i.e. skip the Channel Change command(s) which either do not work with the NA versions of the HMS.

I do not know if @Fribur already created a PR for his changes to be integrated here ?

Did you check that you have a common HMS and not one of the newer HMS-xxxxW WiFi Series Model which has the so called DTU-BI (built in DTU) and can only be reached via WLAN from the S-Miles Installer App or

rj45jack commented 5 months ago

Oh interesting. Thank you. I do have the non W I believe. I have the dtu pro on my desk.

stefan123t commented 5 months ago

@rj45jack you should not use more than one DTUs with an inverter. The inverter will get confused by cross-talk and tends to talk to Hoymiles DTU Pro S preferably as they probably save the current DTU for about 15 minutes and use a heartbeat mechanism to prevent complications.

tbnobody commented 5 months ago

@rj45jack according to Fribur and homeautomation22 this requires some modification to the code, i.e. skip the Channel Change command(s) which either do not work with the NA versions of the HMS.

As I read the comment from @Fribur I thought that it's enough to change the communication channel to 915MHz. If the inverter is reachable at 915 then the channel change command gets not sent anyway. (And even if it's send, it will be sent to a frequency the inverter isn't listening)

But @stefan123t is right. It's not possible to use the DTU pro and OpenDTU at the same time. (If you used the "pro" before, you have to wait at least 15min until the inverter falls back to it's default frequency and OpenDTU can communicate with it). And make sure your inverter is powered with DC. AC only is not enough.

rj45jack commented 5 months ago

OK I will try again tomorrow when the sun comes out. I'll only have the OpenDTU running with 915Mhz. Good to know I can only run one at a time.

stefan123t commented 5 months ago

@rj45jack if you do have a Hoymiles DTU Pro S on your desk, you may also be able to trace the necessary communication for the North American (NA) HMS models.

There are some pictures of the bare circuits of a DTU W Lite in our sister project AhoyDTU and a guide on How To setup up a serial monitor

Using that information it should be possible to spot the two RX/TX testpoints on the PCB between the GD32 cpu and CMT2300A RF chip in your Hoymiles DTU Pro S. When you have the testpoints you can connect to it using one/two USB2Serial Adapters in order to trace the communication data sent to the CMT2300A using a terminal monitor program.

Let us know if you need support in setting this up ?

rj45jack commented 5 months ago

@rj45jack if you do have a Hoymiles DTU Pro S on your desk, you may also be able to trace the necessary communication for the North American (NA) HMS models.

There are some pictures of the bare circuits of a DTU W Lite in our sister project AhoyDTU and a guide on How To setup up a serial monitor

Using that information it should be possible to spot the two RX/TX testpoints on the PCB between the GD32 cpu and CMT2300A RF chip in your Hoymiles DTU Pro S. When you have the testpoints you can connect to it using one/two USB2Serial Adapters in order to trace the communication data sent to the CMT2300A using a terminal monitor program.

Let us know if you need support in setting this up ?

Oh thats very interesting. I can surely do that. I originally was going to see about the RS485 on the back for my own data collection, but then I found OpenDTU and built that. I'm eager to get off the S-miles cloud.

stefan123t commented 5 months ago

You may want to join us in the Discord channel #hms-hmt-series for further discussion.

image

The complete pin mapping can be found in the Hoymiles HMS101 manual:

https://manuals.plus/hoymiles/hms101-sub-1g-transmitter-module-manual

NO. Symbol I/O Type Function
1 GND P Power supply reference ground pin
2 NC I Null pin, no internal connection
3 P1.5 I/O RESERVED, P1.5, which is connected to P1.5 on the IC
4 P1.6 I/O RESERVED, P1.6, which is connected to P1.6 on the IC
5 P0.1 I/O RESERVED,P0.1, which is connected to P0.1 on the IC
6 P0.6 I/O RESERVED, P0.6, which is connected to P0.6 on the IC
7 NC I/O Serial peripheral interface clock pin
8 GND P Power supply reference ground pin
9 RXD I/O UART_RX, which is connected to P0.4 on the IC
10 TXD Output UART_TX, which is connected to P0.3 on the IC
11 GND P Power supply reference ground pin
12 VCC P Power supply pin
13 PRG I/O Set high to enter flash programming mode
14 SCK I/O Serial peripheral interface clock pin
15 RST I/O Hardware reset pin (active at a low level)
16 GPIO I/O Reserved
17 MI I/O Reserved
18 SCN I/O Reserved
19 GND P Power supply reference ground pin
20 GND P Power supply reference ground pin

Table 1 HM2401 interface pins

Here are the test pins FT-RX3 / FT-TX3 which probably can be used: image

some close ups of the HMS101H-v1.0.1 module:

image

as well as pictures of the HMS101 module without the lid:

TX and RX are obviously connected to FT-TX3 and FT-RX3 as hinted above.

10 9 8 7 6 5 4 3 2 1
TX RX GND NC P0.6 P0.1 P1.6 P1.5 NC GND

image

11 12 13 14 15 16 17 18 19 20
GND 3.3V DIO/PRG CLK/SCK RST GPIO MI SCN GND GND

While GND, 3.3V DIO and CLK are all available on the CN202_CN1 or the PN201_NC connectors.

image image

NOTE: the BAUD rate on the UART between the CPU and the RF Chips (NRF / CMT) is usually 125000, 8N1.

Fribur commented 5 months ago

@tbnobody: what your discord handle? To clarify also here: the current trunk version should work with the NA version of the HMS out of the box. No code change needed except for moving frequency to 915MHz. The change I propose above would be required to "localize" the configuration of the CMT2300A to North America (needed to reach the upper limit of 925 MHz)....but is a bit pointless right now for 2 reasons

  1. we do not know what the channel change command is (the current command is ignored by the NA version of the HMS despite sending it on the initial HMS listening frequency of 915MHz. Also tried 865, 868..useless, it just does not work). Together with @rj45jack and @stefan123t, homeautomation22 (discord handle) we might be able to figure that out
  2. we do not know what the base frequency is, which we need to know as the channels are calculated in reference to a base. Above I speculate it is 905MHz...but we do not know for sure. Once we successfully trigger a channel change, I can figure out the base frequency by scanning to which frequency we moved the inverter (using an RTL-SDR).
  3. Once all this is done, I am happy to contribute a pull request that makes the region configurable via Web Interface / config
stefan123t commented 5 months ago

@Fribur you solved the initial issue that @rj45jack had with homeautomation22 yesterday in the discord. It was most likely caused by the CMT2300A module not connecting the GPIO3 which is used for interrupt. Once GPIO2/3 were changed to -1 in the pin_mapping.json for CMT module the answer were polled and data arrived.

Regarding the Channel Change Command I understood from homeautomation22 that he assumes the command is ignored by the DTU and it happily continues on 915 MHz ever-here-after.

Maybe it would be sufficient for your purposes to have an RTL-SDR trace of the communication between the Hoymiles DTU Pro S and the HMS inverter from @rj45jack ?

Regarding the other requests for tracing commands e.g. Search Inverter, Grid Profile update and Firmware update we will most likely require the Terminal traces using one/two Serial2USB for the TX/RX Testpoints on the DTU Pro S board to really get meaningful data. Extracting the exact commands sent by the DTU Pro S to the HMS inverter using RTL-SDR proved to be too difficult and brittle in the past. As the Terminal will happily return all the necessary data in binary form it is probably the easiest way to trace the actual commands used by Hoymiles.

Maybe @rj45jack can remove all but one inverter from his S-Miles Cloud account in order to record communication with one inverter only or having several "unknown" inverters for the Search / Gongfa command ?

Fribur commented 5 months ago

Documenting also here the resolution developed on discord (channel change command for America): 

00   01 02 03 04   05 06 07 08   09   10   11   12   13   14   15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
-----------------------------------------------------------------------------------------------------------------
56   71 60 35 46   80 12 23 04   02   15   21   00   14   00   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- (Europe)
56   71 60 35 46   80 12 23 04   03   17   3c   00   14   00   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- (America)
^^   ^^^^^^^^^^^   ^^^^^^^^^^^   ^^   ^^   ^^   ^^   ^^   ^^
ID   Target Addr   Source Addr   ?    ?    ?    CH   ?    CRC8

Full patch to configure OpenDTU for America: OpenDTU_NA.patch

stefan123t commented 5 months ago

I peeked into the patch, the param table is probably generated using RFPDK. Has this been documented somewhere in the source ? Switching frequency precision from Hz to kHz is good, it is uint_32 internally. Only change request would be "North America" instead of "America" for the NA option in the notes/text.

Fribur commented 5 months ago

cmt2300a_params.h has already this comment: Generated by CMOSTEK RFPDK 1.46. I just played around with the parameters until I got exactly the output as it was (the 860MHz base in trunk), and then just changed the frequency to 900 MHz and updated cmt2300a_params.h accordingly. Here an updated patch that prohibits channel 0 (900MHz) and channel 255 (965,75) as those channel do not work with the American inverter.

Addressing also your change request "North America". To be precise, we would need a different warning for each country (e.g. Brazil permitted 915-925). However "country" is not configured anywhere right now. I would propose to just make region configurable, and initialize CMT2300A either with 860 base or 900 base. Warning could be permanently displayed (listing some of the currently known possibilities: EU: 860-870MHz, default 865MHz US: 905-925MHz, default 915MHz Brazil: 915-928 MHz, default ??? OpenDTU_NA.patch

tbnobody commented 5 months ago

I've already implemented a lot of changes to make the whole configuration more flexible. Will maybe post a new firmware here today or tomorrow.

tbnobody commented 5 months ago

The code is not yet pushed. But if you give me positive feedback regarding this firmware I will publish it. So please test it if it's working with your inverter. I am only able to test it with an european one. (Compiled firmware is attached)

image

I added the Country/Region in the DTU settings. The settings are applied without restart of the ESP. The total range as well las the legal ranges are adjusted dynamically.

generic_esp32.zip

Fribur commented 5 months ago

The bin works.

  1. Selecting initially 915MHz (NA region), I got immediately connection
  2. Selecting then a lower frequency, a channel change command is issued after 40s (RX timeout), and right after I got connection
  3. Selecting then a higher frequency, the channel change command sent on 915Mhz after 40s is obviously ignored by inverter, and it takes exactly 900seconds for inverter to fall back from the lower frequency to default of 915MHz to receive then Channel Change command, to then switch to the requested higher frequency, and right after I got connection again.

If you like a serial monitor log of all that I can provide it.

So I suppose this concludes this journey of making the NA inverter work with OpenDTU. Nicely implemented (reconfiguring CMT2300A without re-boot)!

Just out of curiosity: what is default/what does OpenDTU do when no region has been selected yet? I noticed on initial setup the frequency is something like 200MHz..I suppose there is no TX/RX until configured?

tbnobody commented 5 months ago

what is default/what does OpenDTU do when no region has been selected yet?

constexpr frozen::map<CountryModeId_t, CountryFrequencyDefinition_t, 3> countryDefinition = {
    { CountryModeId_t::MODE_EU, make_value(FrequencyBand_t::BAND_860, 863e6, 870e6, 865e6, 868e6) },
    { CountryModeId_t::MODE_US, make_value(FrequencyBand_t::BAND_900, 905e6, 925e6, 915e6, 915e6) },
    { CountryModeId_t::MODE_BR, make_value(FrequencyBand_t::BAND_900, 915e6, 928e6, 915e6, 915e6) },
};
constexpr CountryFrequencyDefinition_t make_value(FrequencyBand_t Band, uint32_t Freq_Legal_Min, uint32_t Freq_Legal_Max, uint32_t Freq_Default, uint32_t Freq_StartUp)

So the second last value is the default value when changing to a new frequency. First and second value is the legal range and the last value the default frequency of the inverter after startup.

And the default value should be something like 865 for Europe if nothing else is configured

tbnobody commented 5 months ago

Thank you all for your research! Will release a new version today.

homeautomation2022 commented 5 months ago

And the default value should be something like 865 for Europe if nothing else is configured

In America I would also pre-configure something other than the usual 915MHz. Here in Europe it has something to do with the unclear transmission power/duty cycle as to why we chose 865 as the default. In America there is a lot more choice, you can also just use 920 MHz or something like that. Don't know what the most standard stuff is there.

carbogit commented 5 months ago

coming from v24.1.4 with unchanged standard setting of 865 MHz, v24.1.14 shows this:

20240114_230150

trying to set 865 is rejected with the following message:

20240114_230859

weitheng commented 5 months ago

3. ency, the channel change command sent on 915Mhz after 40s is obviously ignored by inverter, and it takes exactly 900seconds for inverter to fall back from the lower frequ

I am curious how do you trigger the Channel Command Change? Is it triggered automatically by OpenDTU when it's unable to connect to the inverter, sending the Channel Command Change to the frequency that was set previously? I am in Singapore, and for some reasons the inverters is also communicable only on 915MHz, but I am hoping that if it's possible to switch to a lower frequency say 865MHz (which is legal here), I might then have a better connection with the inverters

Fribur commented 5 months ago

coming from v24.1.4 with unchanged standard setting of 865 MHz, v24.1.14 shows this:

Delete the Chrome/Edge cache. You are missing the region setting, so what you see is not the updated web interface, its the old one. For whatever reason Chrome/Edge does not recognize that the WebApp has changed, so it loads from the cache instead of fetching the new one from OpenDTU (which is fixable but I did not yet dig deep enough into the vue framework to understand how)

Fribur commented 5 months ago

I am curious how do you trigger the Channel Command Change? ...

In case your inverter are per default 915MHz (could you post exact model number?), then they are only able to communicate between 900.25MHz and 963.50MHz. Fundamentally not possible to make them communicate below 900.25MHz, so your idea to set them to 865MHz will not work. The channel change commend is send as soon as the OpenDTU has no connection for ~40 seconds. Repeatedly until connection is established. It is always sent on 915MHz or 865MHz (depending on the new region setting), and will move the communication to the frequency (/channel) that you selected.

weitheng commented 5 months ago

I am curious how do you trigger the Channel Command Change? ...

In case your inverter are per default 915MHz (could you post exact model number?), then they are only able to communicate between 900.25MHz and 963.50MHz. Fundamentally not possible to make them communicate below 900.25MHz, so your idea to set them to 865MHz will not work. The channel change commend is send as soon as the OpenDTU has no connection for ~40 seconds. Repeatedly until connection is established. It is always sent on 915MHz or 865MHz (depending on the new region setting), and will move the communication to the frequency (/channel) that you selected.

I see. If that's the case, what if the frequency/channel has already been set to another channel/frequency say 916MHz, will OpenDTU still be able to send the Channel Change Command to set the inverters to communicate at another frequency? Is the command only sent at 915MHz/ 865MHz, if so, does that mean the channel/frequency can only be changed once?

Fribur commented 5 months ago

The inverter falls back after 15 minutes (900sec) to the default frequency of 915MHz or 865MHz...so the channel change command should always work..eventually.

tbnobody commented 5 months ago

I would assume we can close this issue as the country switch is is implemented. Please reopen if necessary

github-actions[bot] commented 3 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns.