bvschaik / julius

An open source re-implementation of Caesar III
GNU Affero General Public License v3.0
2.85k stars 317 forks source link

Add /utf-8 for MSVC #672

Closed feiyunw closed 1 year ago

feiyunw commented 1 year ago

Some compile error on non-English MSVC:

2>------ Rebuild All started: Project: julius, Configuration: Debug x64 ------
...
2>japanese.c
2>D:\prj\julius\src\translation\japanese.c(1,1): warning C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss
2>D:\prj\julius\src\translation\japanese.c(9,9): error C2001: newline in constant
2>D:\prj\julius\src\translation\japanese.c(11,5): error C2059: syntax error: '{'
...
2>korean.c
2>D:\prj\julius\src\translation\korean.c(1,1): warning C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss
2>D:\prj\julius\src\translation\korean.c(29,32): error C2001: newline in constant
2>D:\prj\julius\src\translation\korean.c(30,5): error C2059: syntax error: '{'
...
2>simplified_chinese.c
2>D:\prj\julius\src\translation\simplified_chinese.c(1,1): warning C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss
2>D:\prj\julius\src\translation\simplified_chinese.c(7,9): error C2001: newline in constant
2>D:\prj\julius\src\translation\simplified_chinese.c(10,9): error C2001: newline in constant

"/utf-8" in *.vcxproj files can fix it like:

<AdditionalOptions>%(AdditionalOptions) /utf-8</AdditionalOptions>

Anyway, CMake add_compile_options() didn't work as expected, thus a CMAKE_C_FLAGS workaround is used in this PR.

FYI: /utf-8 add_compile_options

crudelios commented 1 year ago

I think this causes characters to not show at all, but I'll need to confirm

feiyunw commented 1 year ago

I verified that the built x64 version is compatible with Simplified Chinese. julius-log.txt julius

feiyunw commented 1 year ago

Some warnings after switching code page to utf-8:

2>D:\prj\julius\ext\png\png.h(1,1): warning C4828: The file contains a character starting at offset 0x355 that is illegal in the current source character set (codepage 65001).
2>D:\prj\julius\ext\zlib/zlib.h(1,1): warning C4828: The file contains a character starting at offset 0x567 that is illegal in the current source character set (codepage 65001).

png.h, line 22:

 * Changed for Julius by Jos?Cadete (crudelios) on 11 Feb 2020.

zlib.h, line 30:

  Changed for Julius by Jos?Cadete (crudelios) on 11 Feb 2020.

As "é" in José is incompatible with utf-8, I guess. Fortunately, it does not cause compile error.

crudelios commented 1 year ago

Afaik the files are not saved in utf8 format, rather on iso-8859-1 (except for the Chinese etc translation files), hence the warning.

bvschaik commented 1 year ago

All files in the Julius src directory should be UTF-8. Most of them are plain ASCII, which is basically the same as/compatible with ISO-8859-1. If the PNG files are another encoding, we should re-save them as UTF-8.