license: check-copyright fails non-deterministically with comment_parser.parsers.common.UnterminatedCommentError or utf8 decoding error (IDFGH-10692)

[matu3ba]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [x] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0.1

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

No error and popup of checker results.

What is the actual behavior?

Non-deterministic error:

• 1. Option

     Traceback (most recent call last):
     File "../check-copyright/check_copyright.py", line 697, in <module>
     main()
     File "../check-copyright/check_copyright.py", line 659, in main
     wrong_header_files, modified_files, must_be_updated = check_copyrights(args, config)
     File "../check-copyright/check_copyright.py", line 520, in check_copyrights
     valid, modified = has_valid_copyright(file_name, mime, False, config[matched_section], args)
     File "../check-copyright/check_copyright.py", line 242, in has_valid_copyright
     code = f.read()
     File "/usr/lib/python3.8/codecs.py", line 322, in decode
     (result, consumed) = self._buffer_decode(data, self.errors, final)
     UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 59354: invalid start byte

• 2. Option

     Traceback (most recent call last):
     File "../check-copyright/check_copyright.py", line 697, in <module>
     main()
     File "../check-copyright/check_copyright.py", line 659, in main
     wrong_header_files, modified_files, must_be_updated = check_copyrights(args, config)
     File "../check-copyright/check_copyright.py", line 520, in check_copyrights
     valid, modified = has_valid_copyright(file_name, mime, False, config[matched_section], args)
     File "../check-copyright/check_copyright.py", line 243, in has_valid_copyright
     comments = get_comments(code, mime)
     File "../check-copyright/check_copyright.py", line 216, in get_comments
     comments = parser.extract_comments(code)
     File "/home/hafer/.local/lib/python3.8/site-packages/comment_parser/parsers/c_parser.py", line 66, in extract_comments
     raise common.UnterminatedCommentError()
     comment_parser.parsers.common.UnterminatedCommentError

Steps to reproduce.

• 1. Step Install python dependencies of check-copyright via ``.
• 2. Step Run python3 ../check-copyright/check_copyright.py --dry-run --config tools/ci/check_copyright_config.yaml --ignore tools/ci/ignore_build_warnings.txt ..

OS (uname -a): Linux ASG148 5.15.0-75-generic #82~20.04.1-Ubuntu SMP Wed Jun 7 19:37:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Python version: Python 3.8.10

Build or installation Logs.

Logs are way too noisy with skipped things, so I provide some example chunk. See "More Information".

``` "./components/bt/host/bluedroid/bta/include/bta/bta_hh_co.h" is using config s "./tools/idf_size_yaml/esp32h2_data_info.yaml" is not of a supported type! Ski "./components/bt/esp_ble_mesh/README.md" is not of a supported type! Skipping. "./tools/unit-test-app/configs/default_2_c2" is not of a supported type! Skipp "./components/openthread/openthread/tools/spi-hdlc-adapter/README.md" is not o "./components/mbedtls/test/crts/bad_md_crt.pem" is not of a supported type! Sk "./components/openthread/openthread/third_party/nlbuild-autotools/repo/automak "./components/bt/host/nimble/nimble/nimble/host/mesh/src/rpl.c" is using confi "./examples/bluetooth/esp_ble_mesh/ble_mesh_wifi_coexist/sdkconfig.ci.bluedroi "./examples/bluetooth/esp_ble_mesh/ble_mesh_fast_provision/fast_prov_server/tu "./components/pthread/CMakeLists.txt" is not of a supported type! Skipping. "./components/openthread/openthread/third_party/mbedtls/repo/tests/suites/test "./components/bt/host/nimble/nimble/porting/examples/linux_blemesh/main.c" is "./components/spi_flash/sim/Makefile.files" is not of a supported type! Skippi "./tools/ci/python_packages/idf_http_server_test/__init__.py" is empty! "./examples/wifi/getting_started/station/main/CMakeLists.txt" is not of a supp "./examples/bluetooth/bluedroid/ble/ble_ancs/CMakeLists.txt" is not of a suppo "./examples/bluetooth/bluedroid/ble/ble_spp_client/sdkconfig.defaults.esp32h2" "./docs/zh_CN/api-reference/peripherals/adc_continuous.rst" is not of a suppor "./components/mbedtls/mbedtls/tests/data_files/cert_sha1.crt" is not of a supp "./components/wpa_supplicant/src/tls/pkcs5.h" is using config section "ignore" "./components/bt/host/nimble/nimble/nimble/host/test/src/ble_os_test.c" is usi "./components/esp_rom/README.md" is not of a supported type! Skipping. "./components/mbedtls/mbedtls/tests/data_files/dir4/cert71.crt" is not of a su ./components/heap/tlsf/tlsf.c:4 License "BSD-3-Clause" is not allowed! Allowed "./examples/build_system/cmake/multi_config/sdkconfig.defaults" is not of a su "./components/openthread/openthread/third_party/mbedtls/repo/tests/suites/test "./components/openthread/openthread/third_party/mbedtls/repo/visualc/VS2010/ke "./examples/bluetooth/bluedroid/classic_bt/bt_spp_vfs_initiator/README.md" is "./components/bt/host/nimble/nimble/nimble/host/mesh/src/transport.c" is using "./examples/bluetooth/bluedroid/ble_50/multi-adv/sdkconfig.defaults" is not of "./components/openthread/openthread/third_party/mbedtls/repo/tests/data_files/ "./docs/_static/esp32-ethernet-kit-rmii-clk-from-phy.png" is not of a supporte "./components/bt/host/nimble/nimble/porting/examples/linux_blemesh/ble.c" is u "./docs/en/api-reference/system/inc/power_management_esp32.rst" is not of a su "./docs/_static/esp-wrover-kit-v2-layout-back.png" is not of a supported type! "./components/bt/host/nimble/nimble/nimble/host/services/ias/include/services/ "./components/wpa_supplicant/src/common/dragonfly.h" is using config section " "./components/heap/test_multi_heap_host/test_all_configs.sh" is not of a suppo "./examples/wifi/wifi_easy_connect/dpp-enrollee/CMakeLists.txt" is not of a su "./examples/bluetooth/nimble/ble_periodic_adv/README.md" is not of a supported "./components/efuse/test/three_key_blocks/esp_efuse_test_table.csv" is not of "./docs/en/api-guides/freertos-smp.rst" is not of a supported type! Skipping. "./examples/zigbee/light_sample/light_bulb/main/esp_zb_light.c" is using confi "./examples/protocols/sockets/udp_server/main/CMakeLists.txt" is not of a supp "./examples/bluetooth/bluedroid/ble_50/ble50_security_server/CMakeLists.txt" i "./components/soc/esp32c2/CMakeLists.txt" is not of a supported type! Skipping "./docs/zh_CN/hw-reference/esp32s2/user-guide-esp-lyrap-cam-v1.1.rst" is not o "./components/bt/host/nimble/nimble/docs/ble_hs/ble_hs.rst" is not of a supported type! Skipping. "./docs/_static/diagrams/twai/state_transition.diag" is not of a supported type! Skipping. "./docs/en/hw-reference/esp32s2/user-guide-esp-lyrap-cam-v1.1.rst" is not of a supported type! Skipping. "./components/cmock/CMock/test/unit/cmock_generator_plugin_ignore_test.rb" is not of a supported type! Skipping. "./components/mbedtls/mbedtls/tests/data_files/dir-maxpath/c01.pem" is not of a supported type! Skipping. "./components/cmock/CMock/vendor/unity/src/meson.build" is not of a supported type! Skipping. "./components/bt/host/bluedroid/bta/hf_ag/include/bta_ag_int.h" is using config section "ignore" which does not perform the check! Skipping. "./components/unity/unity/extras/fixture/readme.txt" is not of a supported type! Skipping. "./examples/ethernet/iperf/sdkconfig.defaults" is not of a supported type! Skipping. "./components/lwip/lwip/test/fuzz/output_to_pcap.sh" is not of a supported type! Skipping. "./components/bt/host/nimble/nimble/porting/nimble/Makefile.defs" is not of a supported type! Skipping. "./components/openthread/openthread/third_party/mbedtls/repo/tests/suites/test_suite_gcm.aes128_en.data" is not of a supported type! Skipping. "./docs/zh_CN/api-reference/peripherals/adc_calibration.rst" is not of a supported type! Skipping. "./docs/zh_CN/api-reference/bluetooth/esp_gattc.rst" is not of a supported type! Skipping. "./examples/peripherals/timer_group/gptimer/README.md" is not of a supported type! Skipping. "./components/mbedtls/mbedtls/tests/data_files/ecdsa_secp256r1.crt" is not of a supported type! Skipping. "./examples/protocols/https_mbedtls/sdkconfig.ci" is not of a supported type! Skipping. Traceback (most recent call last): File "../check-copyright/check_copyright.py", line 697, in main() File "../check-copyright/check_copyright.py", line 659, in main wrong_header_files, modified_files, must_be_updated = check_copyrights(args, config) File "../check-copyright/check_copyright.py", line 520, in check_copyrights valid, modified = has_valid_copyright(file_name, mime, False, config[matched_section], args) File "../check-copyright/check_copyright.py", line 243, in has_valid_copyright comments = get_comments(code, mime) File "../check-copyright/check_copyright.py", line 216, in get_comments comments = parser.extract_comments(code) File "/home/hafer/.local/lib/python3.8/site-packages/comment_parser/parsers/c_parser.py", line 66, in extract_comments raise common.UnterminatedCommentError() comment_parser.parsers.common.UnterminatedCommentError ```

More Information.

1. piping to file includes shell escape symbols, which makes them unreadable
2. During checks, there is a huge pile of empty space and in between the checker complains:

   ./components/heap/tlsf/tlsf.h:4 License "BSD-3-Clause" is not allowed! Allowed licenses: ['Apache-2.0'].

3. The installation README on https://github.com/espressif/check-copyright is missing installation instructions for the dependencies
4. https://github.com/espressif/check-copyright is missing compatibility a description for where it works and breaking changes etc.
5. Output should not contain skipped files, which are in an unreadable font with dark background. Those should only be shown with a flag --verbose.
6. As far as I understand, there is a license violation in m5.c
7. Having a sane way to know which libararies are being used without tracing all paths used for compilation would be nice. Right now, it is painful to be compliant, if optional modules are selected. If you think, humans should manually check, then please provide instructions how to trace accessed path within ninja or what you consider "supported build system" for this purpose.
8. The documentation of check-copyright does not mention, if piping files used for building to check_copyright works. I think most people would prefer this option.

[matu3ba]

This issue does not persist on master anymore, so I am closing this. I created this issue in case other people run into the same one.