zephyrproject-rtos / west

West, Zephyr's meta-tool
https://docs.zephyrproject.org/latest/guides/west/index.html
Apache License 2.0
215 stars 117 forks source link

Use 'backslashreplace' not to crash on malformed UTF from subprocess #700

Closed marc-hb closed 6 months ago

marc-hb commented 6 months ago

Giant commit https://github.com/zephyrproject-rtos/hal_nxp/commit/f9f0944bc2b4fce "Update to SDK 2.14" added files with malformed UTF-8, more precisely with the DEGREE SIGN (°) encoded in 8bit latin1/CP1252.

This crashes west grep.

Before this fix:

nxp$ west grep 'TEMPERATURE_CONV_FACTOR.*Will give'

Traceback (most recent call last):
  File ".local/bin/west", line 33, in <module>
    sys.exit(load_entry_point('west', 'console_scripts', 'west')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/app/main.py", line 1085, in main
    app.run(argv or sys.argv[1:])
  File "west/src/west/app/main.py", line 244, in run
    self.run_command(argv, early_args)
  File "west/src/west/app/main.py", line 503, in run_command
    self.run_builtin(args, unknown)
  File "west/src/west/app/main.py", line 611, in run_builtin
    self.cmd.run(args, unknown, self.topdir,
  File "west/src/west/commands.py", line 194, in run
    self.do_run(args, unknown)
  File "west/src/west/app/project.py", line 1765, in do_run
    completed_process = self.run_subprocess(
                        ^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/commands.py", line 325, in run_subprocess
    return subprocess.run(args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 2146, in _communicate
    stdout = self._translate_newlines(stdout,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1086, in _translate_newlines
    data = data.decode(encoding, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 385: invalid start byte

After this fix, no crash and no interruption and:

...
mcux-sdk/middleware/issdk/sensors/fxpq3115_drv.h:#define FXPQ3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
mcux-sdk/middleware/issdk/sensors/mpl3115_drv.h:#define MPL3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
...
marc-hb commented 6 months ago

west grep 'TEMPERATURE_CONV_FACTOR.*Will give'

Obviously, this is the not the search string I was using when I found this crash :-)