Transcode and normalize encoding of all files across the project

[stevewgr]

Description

Resolves the following issue: https://github.com/ko4life-net/ko/issues/240

This PR introduces the following changes:

• Standardized and normalized file encoding across the project by transcoding all text-based files to UTF-8. This ensures compatibility with modern systems and supports multiple languages, specifically Korean, Chinese, and Polish characters. Microsoft's resource files (*.rc) are exempt and transcoded to UTF-16 Little Endian (LE byte order) with a Byte Order Marker (BOM).

• Converted all Windows CRLF line endings to Unix LF to ensure cross-platform compatibility and consistency in various editors.

• Implemented a python script to automate the encoding standardization and line ending conversion.

• Configured .gitattributes to handle *.rc files as text with UTF-16 encoding, preventing them from being misinterpreted as binary and ensuring proper diff display.

• Added a .editorconfig file to enforce encoding consistency for future files and modifications.

Workflow

# 1.
git clone git@github.com:ko4life-net/ko.git

# 2. Run detect encoding to generate the csv files.
python .\ko\script\fix-encoding.py --root-dir .\ko --detect-encoding

# 3. Update and review the guessed encoding in ko-encoding-manual.csv file to match with the correct source encoding.
# Also merge whatever you found faulty in ko-encoding-invalid.csv into ko-encoding-manual.csv as needed.

# 4. Run ko-encoding-manual.csv -> check yourself -> all good? run the main ko-encoding.csv file
python .\ko\script\fix-encoding.py --encoding-file .\ko-encoding-manual.csv
python .\ko\script\fix-encoding.py --encoding-file .\ko-encoding.csv

# 5. Final touch: convert all files to use Unix LF line feed instead Windows CRLF
python .\ko\script\fix-encoding.py --root-dir .\ko --crlf-to-lf

# 6. Run format.ps1

Results

• ko-encoding-manual.csv

• ko-encoding.csv

Log files

• 1_fix-encoding_detection_2024-07-20_15-24-47.log
• 2_fix-encoding_transcode_manual_2024-07-20_15-29-42.log
• 3_fix-encoding_transcode_2024-07-20_15-34-25.log
• 4_fix-encoding_crlf_to_lf_2024-07-20_15-38-48.log

[stevewgr]

I ran some investigation on why would the new .gitattributes configurations messes up in the wrong way, where it tries to take a perfectly fine UTF-16-LE encoded RC file and rencode it from UTF-8 to UTF-16. This seems like an unexpected behavior from gitattributes, and hence I removed these configs: https://github.com/ko4life-net/ko/pull/241/commits/317d79598a439dbac4c3eada2e59302c005320a2

I initially added them so that when I run from the terminal git diff, I'll be able to see the content, since git doesn't handle well with UTF-16 encoded files and treat them as binary instead of text. However UI based diffing still works in vscode and some other diffing tools, so let's remove these configs from .gitattributes and merge this PR.

Also there are some work around in case someone still want to see the diff of UTF-16 encoded text files by changing the default diff tool: https://stackoverflow.com/questions/777949/can-i-make-git-recognize-a-utf-16-file-as-text

I tested the build on multiple scenarios and they seem to all pass perfectly fine:

• https://github.com/ko4life-net/ko/actions/runs/10025128918
• https://github.com/ko4life-net/ko/actions/runs/10025178930
• https://github.com/ko4life-net/ko/actions/runs/10025179949
• https://github.com/ko4life-net/ko/actions/runs/10025221456

Thanks for the review @xGuTeK and @srmeier 🚀