Windows: building samples reports "UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2"

[aaronyegx]

Describe the bug

After doing the "west update" and executing the "pip3 install -r %HOMEPATH%\zephyrproject\zephyr\scripts\requirements.txt", we cannot build the samples. It always report "UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 46".

To Reproduce

1. PS C:\ambiqcodes\ambiqzephyr\zephyr> west update
2. PS C:\ambiqcodes\ambiqzephyr\zephyr> pip3 install -r .\scripts\requirements.txt
3. PS C:\ambiqcodes\ambiqzephyr\zephyr> west build -b apollo4p_evb .\samples\hello_world\ -p

Expected behavior

No compiling error reports.

Impact

We cannot continue to develop until this issue getting resolved.

Logs and console output

PS C:\ambiqcodes\ambiqzephyr> west build -b apollo4p_evb .\zephyr\samples\hello_world\ -p -- west build: making build dir C:\ambiqcodes\ambiqzephyr\build pristine -- west build: generating a build system Loading Zephyr default modules (Zephyr base). -- Application: C:/ambiqcodes/ambiqzephyr/zephyr/samples/hello_world -- CMake version: 3.22.3 -- Found Python3: C:/Python310/python.exe (found suitable version "3.10.5", minimum required is "3.10") found components: Interpreter -- Cache files will be written to: C:/ambiqcodes/ambiqzephyr/zephyr/.cache -- Zephyr version: 3.7.99 (C:/ambiqcodes/ambiqzephyr/zephyr) -- Found west (found suitable version "1.0.0", minimum required is "0.14.0") CMake Error at C:/ambiqcodes/ambiqzephyr/zephyr/cmake/modules/boards.cmake:198 (message): Error finding board: apollo4p_evb

Error message: Traceback (most recent call last):

File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 387, in <module>
  dump_v2_boards(args)
File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 335, in dump_v2_boards
  boards = find_v2_boards(args)
File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 262, in find_v2_boards
  b = load_v2_boards(args.board, board_yml, systems)
File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 187, in load_v2_boards
  b = yaml.load(f.read(), Loader=SafeLoader)

UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 46: illegal multibyte sequence

Environment (please complete the following information):

• OS: Windows 10
• Toolchain: Zephyr SDK 0.16.8
• Commit SHA or Version used: 1ec5ce05f9f84b18cb8056d01917480a6ec39e52 (latest main branch)

Additional context

[aaronyegx]

Hi @tejlmand , can you take a look? Maybe we need a fix like this? or:

[pdgendt]

I'm not able to reproduce this, on main (1ec5ce05f9f) Environment:

OS: Windows 10 Toolchain: Zephyr SDK 0.16.8

Tried with python 3.10/ python 3.11/ west 1.0.0/ west 1.2.0/ west 1.3.0a2

[pdgendt]

Could you try to find out which board file is causing this? Are you sure it's in-tree?

[aaronyegx]

Could you try to find out which board file is causing this? Are you sure it's in-tree?

I think the issue is in the scripts/list_boards.py. I think some Windows does not use utf-8 encoding by default, could you check the changes in https://github.com/zephyrproject-rtos/zephyr/issues/79745#issuecomment-2410522982. It seems the change can address the compiling issue in my site.

[pdgendt]

I think the issue is in the scripts/list_boards.py. I think some Windows does not use utf-8 encoding by default

Sure, but it's the decoding of some board file which is causing this. I'd like to know which one

could you check the changes in #79745 (comment). It seems the change can address the compiling issue in my site.

I don't have issues to begin with, so I wouldn't know what I'm fixing.

[aaronyegx]

I think the issue is in the scripts/list_boards.py. I think some Windows does not use utf-8 encoding by default

Sure, but it's the decoding of some board file which is causing this. I'd like to know which one

could you check the changes in #79745 (comment). It seems the change can address the compiling issue in my site.

I don't have issues to begin with, so I wouldn't know what I'm fixing.

I am using apollo4p_evb.

[pdgendt]

So reading up on open in python, it can depend on the system encoding.

Linux:

Python 3.12.7 (main, Oct  1 2024, 02:05:46) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getencoding()
'UTF-8'

Windows:

Python 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getencoding()
'cp1252'

This does feel fragile, and maybe we should take a look at https://peps.python.org/pep-0540/

[pdgendt]

I am using apollo4p_evb.

Yes, but the commands fails when trying to dump/list boards, so it might be another board that is causing the issue.

What's the output of west -vvv boards?

[aaronyegx]

I am using apollo4p_evb.

Yes, but the commands fails when trying to dump/list boards, so it might be another board that is causing the issue.

What's the output of west -vvv boards?

PS C:\ambiqcodes\ambiqzephyr\zephyr> west -vvv boards args namespace: Namespace(help=None, zephyr_base=None, verbose=3, command='boards', format='{name}', name_re=None, arch_roots=[], board_roots=[], soc_roots=[], board=None, board_dir=None) Traceback (most recent call last): File "C:\Python310\Lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\Lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Python310\Scripts\west.exe__main__.py", line 7, in File "C:\Python310\Lib\site-packages\west\app\main.py", line 866, in main app.run(argv or sys.argv[1:]) File "C:\Python310\Lib\site-packages\west\app\main.py", line 111, in run self.run_command(argv) File "C:\Python310\Lib\site-packages\west\app\main.py", line 385, in run_command self.run_extension(args.command, argv) File "C:\Python310\Lib\site-packages\west\app\main.py", line 467, in run_extension self.cmd.run(args, unknown, self.topdir, manifest=self.manifest, File "C:\Python310\Lib\site-packages\west\commands.py", line 194, in run self.do_run(args, unknown) File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\west_commands\boards.py", line 100, in do_run for board in list_boards.find_v2_boards(args): File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 262, in find_v2_boards b = load_v2_boards(args.board, board_yml, systems) File "C:\ambiqcodes\ambiqzephyr\zephyr\scripts\list_boards.py", line 187, in load_v2_boards b = yaml.load(f.read(), Loader=SafeLoader) UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 46: illegal multibyte sequence PS C:\ambiqcodes\ambiqzephyr\zephyr>

[pdgendt]

As I expected, the apollo4p_evb board file itself has no issues, but other files with UTF-8 characters do:

$ rg "[^\x00-\x7F]" -g "board.yml"
boards/ezurio/mg100/board.yml
3:  full_name: Sentrius™ MG100 Gateway

boards/beagle/beaglev_fire/board.yml
3:  full_name: BeagleV®-Fire

boards/intel/socfpga_std/cyclonev_socdk/board.yml
3:  full_name: Cyclone® V SoC Development Kit

boards/intel/socfpga/agilex5_socdk/board.yml
3:  full_name: Agilex™ 5 SoC FPGA Development Kit

Opened #79832 to set the encoding.