search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.33k stars 7.2k forks source link

Extended CRC Functionality Including CRC-16/MODBUS Support (IDFGH-13183) #14122

Open hayschan opened 2 months ago

hayschan commented 2 months ago

Is your feature request related to a problem?

Yes, the current CRC capabilities in crc.h (in the rom component) within ESP-IDF are limited primarily to CRC-16-CCITT and do not include support for CRC-16/MODBUS, which is essential for industrial applications utilizing Modbus communication protocols.

I am always frustrated when trying to implement CRC-16/MODBUS for Modbus protocol applications, as the current implementation in crc.h within ESP-ROM only supports CRC-16-CCITT (polynomial 0x1021). This polynomial is different from the CRC-16/MODBUS polynomial (0xA001), requiring a separate implementation or external library to handle Modbus-related CRC calculations.

Describe the solution you'd like.

I would like ESP-IDF to include comprehensive CRC support, particularly for CRC-16/MODBUS, to facilitate easier implementation of Modbus protocols in industrial and IoT projects.

Proposed Solution:

  1. Extend the crc.h library or create a new component that provides configurable support for multiple CRC algorithms. This component would allow developers to specify parameters for each CRC type, accommodating a broader range of applications.
  2. Below is a list of some CRC algorithms I plan to include, showcasing the diversity of CRC configurations used across various industries:
CRC-8 CRC-16 CRC-32
CRC-8 CRC-16/ARC CRC-32
CRC-8/CDMA2000 CRC-16/AUG-CCITT CRC-32/BZIP2
CRC-8/DARC CRC-16/BUYPASS CRC-32C
CRC-8/DVB-S2 CRC-16/CCITT-FALSE CRC-32D
CRC-8/EBU CRC-16/CDMA2000 CRC-32/JAMCRC
CRC-8/I-CODE CRC-16/DDS-110 CRC-32/MPEG-2
CRC-8/ITU CRC-16/DECT-R CRC-32/POSIX
CRC-8/MAXIM CRC-16/DECT-X CRC-32/SATA
CRC-8/ROHC CRC-16/DNP CRC-32/XFER
CRC-8/WCDMA CRC-16/EN-13757
CRC-16/GENIBUS
CRC-16/KERMIT
CRC-16/MAXIM
CRC-16/MCRF4XX
CRC-16/MODBUS
CRC-16/RIELLO
CRC-16/T10-DIF
CRC-16/TELEDISK
CRC-16/TMS37157
CRC-16/USB
CRC-16/X-25
CRC-16/XMODEM
CRC-A

Describe alternatives you've considered.

Additional context.

Including a native, flexible CRC calculation tool within ESP-IDF would significantly enhance its utility for developers working on applications requiring various CRC standards. This feature would align ESP-IDF with other development platforms that offer extensive support for communication protocols and data integrity checks.

alisitsyn commented 1 month ago

@hayschan,

Thank you for this valuable issue, this issue was already discussed more than year ago but still is not addressed. This is possible to extend the CRC library to support additional configurations. The below are some thoughts related to it.

  1. Implement the required algorithm on the project level or component (Current approach)

This allows to implement this in the project (stack) to not rely to the ROM implementation or some other external code.

  1. Extension of ROM library

Comments: This does not look like good solution. It is more reliable to implement this in the project (or in stack) to not rely on the ROM implementation. The size of the implementation for concrete algorithm is very low.

  1. Implementation of the configurable component for checksum calculation: The idea with the separate officially maintained component or a library that supports all possible algorithms (configurations) for checksum calculation looks reasonable and can significantly increase user experience implementing the projects related to industrial applications. This library can include the endianness conversion routines and error correction algorithms as an example. The component can use some sub-module as a library and might have opportunity to override some functionality on user layer without touching to the official sources.

So, the option 3), looks like pretty flexible, reliable and maintainable solution. The existing algorithms can be easily extended/fixed by users and new realizations can be added faster. This is proposed to be selected as a major approach.

Let me know if you have other comments. Thanks.

hayschan commented 1 month ago

@hayschan,

Thank you for this valuable issue, this issue was already discussed more than year ago but still is not addressed. This is possible to extend the CRC library to support additional configurations. The below are some thoughts related to it.

  1. Implement the required algorithm on the project level or component (Current approach)

This allows to implement this in the project (stack) to not rely to the ROM implementation or some other external code.

  1. Extension of ROM library
  • Incompatibility with some previous revisions of chips due to differences in ROM code implementation. It is not guaranteed that the chip used on user side has the support of required configuration.
  • The ROM shall support the only most commonly used algorithms to limit its size. It might be not possible to support all the algorithms you proposed because even through the size of one implementation is low, the whole ROM image size that supports all above configuration of CRC might be increased significantly.
  • The ROM included routines should be maintainable and later new algorithms might be proposed to be added but updating the ROM image not fast and flexible way for this.

Comments: This does not look like good solution. It is more reliable to implement this in the project (or in stack) to not rely on the ROM implementation. The size of the implementation for concrete algorithm is very low.

  1. Implementation of the configurable component for checksum calculation: The idea with the separate officially maintained component or a library that supports all possible algorithms (configurations) for checksum calculation looks reasonable and can significantly increase user experience implementing the projects related to industrial applications. This library can include the endianness conversion routines and error correction algorithms as an example. The component can use some sub-module as a library and might have opportunity to override some functionality on user layer without touching to the official sources.

So, the option 3), looks like pretty flexible, reliable and maintainable solution. The existing algorithms can be easily extended/fixed by users and new realizations can be added faster. This is proposed to be selected as a major approach.

Let me know if you have other comments. Thanks.

Thanks for the detailed reply, @alisitsyn .

After creating the issue a month ago, I checked the ESP-IDF repo and its related repo and found that this algorithm is not included in any of them.

Then, I implemented CRC-16/MODBUS the algorithm myself in my project. To save up compute power, I used the lookup table (LUT) method. The code is also quite easy. I believe this approach is the first method you are referring to - "Implement the required algorithm on the project level or component".

Extension of ROM library

Regarding to baking this algorithm into the ROM library, I agree with your opinion. This is a very niche use case. It should not be included into the main code of ESP-IDF.

Implementation of the configurable component for checksum calculation

I like your idea of "Implementation of the configurable component for checksum calculation". For example, creating a Espressif maintained component repo about CRC checksum. User can include it through ESP component manager (aka idf_component.yml).

By using a lookup table for each algorithm, the code structure is very easy to maintain. When we need to add a new algorithm, all we need is add a new lookup table .h/.c code file. The main logic will stay the same in every algorithm.

I can also contribute as well, since I have quite a lot of experience of implementing this in my projects already.

Let me know if there is something I can do. By having such component/library, I hope we can save others' time in the future. This will help ESP-IDF to become more robust in industrial application.

igrr commented 1 month ago

This feature would align ESP-IDF with other development platforms that offer extensive support for communication protocols and data integrity checks.

Just out of curiosity, could you please give some pointers, which embedded platforms offer the functionality you describe as a built-in feature?

hayschan commented 1 month ago

This feature would align ESP-IDF with other development platforms that offer extensive support for communication protocols and data integrity checks.

Just out of curiosity, could you please give some pointers, which embedded platforms offer the functionality you describe as a built-in feature?

I consider both of the following method as built-in feature:

  1. in the software framework
  2. can be easily used through dependency or library

For ESP-IDF, it is clear that this is not integrated into ESP-IDF. This is understandable to me.

The inconvenience for me is that there is no one has implemented this as an ESP-IDF component. There is also no reference of this feature from any ESP32 open-source projects as well. This field of ESP32 code is vacant.

I had to build things from scratch, and I did.

Platform that have this feature

Arduino

Arduino has this library: https://www.arduino.cc/reference/en/libraries/crc/ . It supports CRC8, CRC12, CRC16, CRC32, CRC64. There is a lot of open-source libraries as well.

AVR

AVR has that, offered through software or hardware.

They have the code library in util/crc16.h in the avr-libc. You can check this: https://www.nongnu.org/avr-libc/user-manual/group__util__crc.html .

Some of their chips (AVR and PIC), by Microchip, offer this feature as a hardware feature as well. They have hardware on the chip for this purpose. Source: https://www.microchip.com/en-us/products/microcontrollers-and-microprocessors/8-bit-mcus/core-independent-and-analog-peripherals/safety-and-monitoring/cyclic-redundancy-check .

Many modern PIC and AVR microcontrollers (MCUs) have a built-in hardware module for CRC. This allows faster and lower-power CRC calculation without extra memory usage or additional CPU cycles when compared with a software implementation. This hardware module supports up to 32-bit polynomials and is implemented with a Linear Feedback Shift Register (LFSR) to generate a checksum of the data. PIC devices can directly feed data into the CRC module, which is useful for calculating the checksum of an outgoing datastream or for verifying an incoming datastream for transmission errors.

If the device is equipped with the SCAN feature, then the CRC module can directly read the memory on the device. This allows you to automatically validate specific regions of memory, such as boot or application, for system integrity. A more advanced version of the SCAN feature (available on select devices) is CRC On-Boot. This feature can be used to scan and verify areas, such as the boot, application, or data EEPROM with a preprogrammed checksum prior to the device running code.

STM32

The guide of STM32 is very clear. They included:

Using the CRC peripheral on STM32 microcontrollers: https://www.st.com/resource/en/application_note/an4187-using-the-crc-peripheral-on-stm32-microcontrollers-stmicroelectronics.pdf

A side note is that the application note of STM32 and AVR is well written. The note is very close to actual use case. I know that many software engineers will use the code and design very similar to what is written in the application note. This saves a lot of time in commercial use.

This is a part where Espressif can improve on. I like the examples on ESP-IDF. They are very informative enough to explain the usage of ESP-IDF. However, application note focus on the whole solution, from hardware to software and product design. Considering that the cost of ESP32 is higher than STM32, this is a part where Espressif can play catch-up.

MicroPython

I believe MicroPython maybe offer it as well, considering that MicroPython can use Python library. I am not a frequent user of MicroPython though.

There are many other platforms that I have never used. They might include this as built-in or as a library as well.

@igrr

hayschan commented 1 month ago

To choose between lookup-table method versus calculation method, I believe a benchmark should be done to determine the difference in performance.

hayschan commented 1 month ago

I have created a repo to fill in the gap for CRC in ESP ecosystem. Here is the link to the repo for this component: https://github.com/hayschan/esp-crc-suite .

Currently, it supports CRC-16. There are four configurable value:

  1. CRC16_POLYNOME
  2. CRC16_INITIAL
  3. CRC16_XOR_OUT
  4. CRC16_REV_IN
  5. CRC16_REV_OUT

This should make them compatible with any variant of algorithms in CRC-16.

alisitsyn commented 1 month ago

@hayschan,

For example, creating a Espressif maintained component repo about CRC checksum. User can include it through ESP component manager (aka idf_component.yml).

This field of ESP32 code is vacant.

This is a part where Espressif can improve on.

I know that many software engineers will use the code and design very similar to what is written in the application note.

I have created a repo to fill in the gap for CRC in ESP ecosystem.

Thank you for shared library and the information you provided above. I agree with the aspects mentioned above and I think we came to an agreement on the creation of the component. The issue is ready to go through the formal process.

To choose between lookup-table method versus calculation method, I believe a benchmark should be done to determine the difference in performance.

The lookup-table method is widely used and can increase performance of calculation significantly it is also flexible. The performance evaluation is required.

hayschan commented 1 month ago

The lookup-table method is widely used and can increase performance of calculation significantly it is also flexible.

The lookup-table method, though widely used, may or may not bring performance edge over calculation method. The lookup-table method uses fewer instructions to the CPU, but the performance depends on the cache. If there is a cache miss, then the performance will be affected.

We should do some benchmarking to that.

alisitsyn commented 2 weeks ago

Note: This task will be postponed for later period due to changes in the schedule. The status changes will be updated here.