Encoding problem?

[NightFurySL2001]

I am trying to build a font with makeotf as below:

PS E:\edit old dot\build_final_otf> makeotf -f cidfont.ps -omitMacNames -ff features.CN -fi cidfontinfo.CN -mf FontMenuNameDB.SUBSET -r -nS -cs 25 -ch UniSourceHanSansCN-UTF32-H -ci SourceHanSans_CN_sequences.txt
makeotf [Warning] Could not find an Adobe CMAP Mac encoding mapping file. If you want a Mac cmap subtable, please specify the Mac CMAP encoding file with the '-cm' option.
makeotf [Note] setting the USE_TYPO_METRICS OS/2 fsSelection bit 7 from fontinfo keyword.
makeotf [Note] setting the WEIGHT_WIDTH_SLOPE_ONLY OS/2 fsSelection bit 8 from fontinfo keyword.
makeotf [Note] setting the OBLIQUE OS/2 fsSelection bit 9 from fontinfo keyword.
Traceback (most recent call last):
  File "c:\users\username\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\username\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\Scripts\makeotf.exe\__main__.py", line 7, in <module>
  File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\afdko\makeotf.py", line 2775, in main
    runMakeOTF(makeOTFParams)
  File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\afdko\makeotf.py", line 2664, in runMakeOTF
    success = fdkutils.run_shell_command_logging(params)
  File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\afdko\fdkutils.py", line 112, in run_shell_command_logging
    out = proc.communicate()[0]
  File "c:\users\username\appdata\local\programs\python\python39\lib\subprocess.py", line 1117, in communicate
    stdout = self.stdout.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xab in position 554: illegal multibyte sequence

My system have used Chinese (PRC) in the settings for localization, and Python by default reads all files as GBK. Does this requires a fix?

[punchcutter]

Which version of Windows are you on? I tried with a clean Chinese install of the latest Windows 10, but I can't reproduce so far using the same command. I can get a 'gbk' error with Python if I do some other things like try to print Korean, but not with makeotf. Did you edit any of the Source Han Sans files before running makeotf? Are there other characters in your path?

[NightFurySL2001]

I was on Windows 10 20H2. Trying to open a UTF-8 file when system is configured to GBK will make Python open("filename") containing non-ASCII characters results in the same 'gbk' error. There was a fix around this issue, which is to turn on Beta: Use Unicode UTF-8 for worldwide language support in Control Panel - Clock and Region - Change date, time, or number formats - Administrative - Change system locale... which will force all programs to read files with UTF-8.

Source files for my project (which I did modified) is on https://github.com/NightFurySL2001/maoken-heavy-labourer.

There are no other characters in my path, it's just E:\edit old dot\build_final_otf.

[NightFurySL2001]

It seems to be related to a problem with subprocess that universal_newlines=True and stdout=PIPE will force decoding of the child process' output using locale.getpreferredencoding(False), which is not UTF-8 for language-locale system. Mine return cp936 instead of cp65001 when I didn't check the option above.

A fix could be as such: https://github.com/adobe-type-tools/afdko/blob/42ad89290a42015dcdf2dfd778d826e928bf9b9b/python/afdko/fdkutils.py#L112

out = proc.communicate().encode("utf-8")[0]

Reference