Abstraction for byte-based raw UART/RS485 communication

[loongyh]

Describe the problem you have/What new integration you would like

Devices with a serial interface such as UART, RS485 from different manufacturers usually have frame structures defined by start_code, device_address, identifier, payload, checksum:

Instead of creating yet another integration for a specific manufacturer, why not have a generic abstraction such that the user can define in .yaml the start_code, device_address, payload (byte representation for the state of a trait), etc., this will enable users to easily integrate numerous RS485 devices into ESPHome potentially without having to dive into C++.

Please describe your use case for this integration and alternatives you've tried:

This integration should span the core abstractions (light, cover, fan, switch, etc.). For any UART-interfacable device that follows the frame structure, and uses the same UART settings (baud rate, start bits, parity, etc.) the user can define in .yaml the start_code, device_address, identifier, payload, checksum type for each device and can then string them together in a RS485 bus to be controlled using one ESPHome master device.

Additional context

Upon receiving a valid command packet, RS485 devices usually transmit a response packet with the result (and payload if any) for the command, usually with the same frame structure. The user defines in .yaml the bytes representing the command codes for each core abstraction trait (such as 0x01 for COVER_OPEN).

# Example config.yaml
uart:
  tx_pin: D0
  rx_pin: D1
  baud_rate: 9600

cover:
  generic_uart:
    name: Bedroom Window
    start_code: 0x55
    address: 0xFEFE
    checksum_type: CRC16
    command:
      open: 0x01   #command code for COVER_OPEN
      stop: 0x00   #command code for COVER_STOP
      close: 0x02  #command code for COVER_CLOSE
    response:
      octet: 5            #vector index of the byte in a frame representing the position of the window
      position_min: 0x00  #byte representation of fully closed
      position_max: 0x64  #byte representation of fully open

Considerations: Although the frame structure may appear to be consistent across manufacturers, there are still other variables to account for such as differences in:

• No. of bytes for each field type (start_code, address, etc.)
• Some are fixed length frames, some are variable with an additional length field type, some are variable depending on the identifier.
• Variances in integer-percentage state value mappings; many ways to represent state values (e.g. 0-255 for 0-100%).

This is more of a RFC as I have just done my first integration (the Grow fingerprint reader currently still in PR) and am just started on writing another integration for Dooya RS485 blinds, so if there's any other issues/considerations with this approach do let me know! This feature request will be revised accordingly as I get familiar with the intricacies in due time.

[sandervandegeijn]

This is a nice idea! It was also implemented by https://github.com/Instathings/modbus2mqtt but that project seems dead unfortunately... Some sort of templating for well known devices would be nice (like that project does)

https://github.com/esphome/feature-requests/issues/1024 seems related.

[loongyh]

Yep, the general idea is to have a base serial integration for each ESPHome core abstraction, with the command codes directly configurable in .yaml. Kinda like the Remote Transmitter/Receiver integrations combined but it will be and show up as an actual core component like light, fan, switch, cover, etc.

We may even pre-define the codes for common devices such that the user only has to define in .yaml:

cover:
  uart:
    device: dooya_rs485  

[martgras]

I have used a similar approach for my implementation of an epever solar controller component. (https://github.com/esphome/esphome/pull/1256)

There is already the option to define additional modbus registers in YAML. See https://github.com/martgras/esphome/tree/epever/esphome/components/epsolar under 'Define an additional register in YAML'

[loongyh]

On further reading, I understand that modbus is a standardized protocol over RS485, with registers already well-defined. My intention is to create an abstraction for "protocol-less" raw UART/RS485 frames.

Upon looking through different datasheets for various RS485 devices, I realized that the frame structures are too varied for a simple approach to such an abstraction. As such, I have taken inspiration from the existing modbus component and intend to create a base RS485 component to just route received frames to individual RS485Device components based on their defined header (e.g. start_code + address).

[sandervandegeijn]

Or just a really simple template with which you can implement regular esphome devices. I can do some of them, it isn't that hard, but as a C++ novice the esphome code is a bit heavy.

[loongyh]

@neographikal you mean to write an UART/RS485 abstraction that provides a set of templatable values over actions/triggers for the user to define a base ESPHome component on?

[sandervandegeijn]

I'd like to have a very simple template with which you could build your own components/devices with. The approach you are suggesting is too generic I think, there is a reason we have a ton of devices in esphome that communicate over serial/bluetooth/whatever and not just a bluetooth/serial component which you have to configure for each device you want to use. This is really user friendly and leaves too much work / variability for the user.

A simple template would enable novices like me to add components for Eastron Energy meters, modbus enabled relays, etc etc to be integrated with esphome.

[loongyh]

Hmm you have a point, in the first place the user shouldn't be expected to manually define each byte, it is crossing into developer role.

Not just the bytes, operation logic also matters. I'm working on another cover device aside from Dooya, and the device responds differently from Dooya's, when the cover opens/closes/stops it sends another frame to update the final position whereas for Dooya it doesn't so you have to regularly poll the device to update the state. We can write out separate logic for each of these behaviours and maybe some more, but how do we cover all use cases, and what do we even call this configurable option? How can this be made simple enough in the docs to be understood by everyone? Hence now I'm just taking the approach used by the PZEM/modbus component, creating a base RS485 class and have components inherit from this to route the packets via on_rs485_received callbacks.

Anyway for the simple template you're suggesting, how will such a template be used? Can you write out an example .yaml?

[sandervandegeijn]

I thought about it, it is a bit more complicated. On a single bus there can be multiple device types active provided they use the same serial settings. Let's assume esphome is the modbus master, this would be the most common scenario. Within esphome you could have a proxy like structure. The device components should not communicate through the serial port directly but through a proxy that catches the requests from the device components, puts them on the serial bus and distributes the answers based on slave ID / device type to the device components. The proxy should also be responsible to manage traffic when the bus is getting saturated.

Something like this:

[loongyh]

The modbus component already does this, simple to do since the frame format is standardized. Just that it's not documented. I'm not sure if you can access it via custom components, but modbus integrations should extend the ModbusDevice class.

[sandervandegeijn]

Ah, didn't know that sorry :) then I would need a simple reference project that demonstrates how to send requests over modbus as a master and to return the results to esphome/mqtt

[loongyh]

For now only PZEM AC/DC does so. Just search for ModbusDevice to find out which integrations use the modbus class.