Encoding issues

[SoftCreatR]

While improving our automation process, i stumbled upon a bunch of files, that seem to be wrong encoded (UTF16). On Windows, this doesn't seem to be a big problem, but on Linux, these files are garbled with asian nonsense, e.g.:

Id: ChristianSchenk.MiKTeX
਍嘀攀爀猀椀漀渀㨀 ㈀⸀㤀⸀㜀㐀㐀㈀ഀഀ
Name: MiKTeX
਍倀甀戀氀椀猀栀攀爀㨀 䌀栀爀椀猀琀椀愀渀 匀挀栀攀渀欀ഀഀ
License: Redistributing MiKTeX
਍䰀椀挀攀渀猀攀唀爀氀㨀 栀琀琀瀀猀㨀⼀⼀洀椀欀琀攀砀⸀漀爀最⼀挀漀瀀礀椀渀最ഀഀ
AppMoniker: miktex
਍吀愀最猀㨀 琀攀砀Ⰰ 琀礀瀀攀猀攀琀琀椀渀最Ⰰ 氀愀琀攀砀Ⰰ 琀攀砀眀漀爀欀猀ഀഀ
Description: MiKTeX (pronounced mick-tech) is an up-to-date implementation of TeX/LaTeX and related programs. TeX is a typesetting system written by Donald Ervin Knuth who says that it is intended for the creation of beautiful books - and especially for books that contain a lot of mathematics.
਍䠀漀洀攀瀀愀最攀㨀 栀琀琀瀀猀㨀⼀⼀洀椀欀琀攀砀⸀漀爀最⼀ഀഀ
Installers:
਍  ⴀ 䄀爀挀栀㨀 砀㘀㐀ഀഀ
    Url: https://muug.ca/mirror/ctan/systems/win32/miktex/setup/windows-x64/basic-miktex-2.9.7442-x64.exe
਍    匀栀愀㈀㔀㘀㨀 䔀㌀㄀䔀㔀㐀㔀　㐀㔀㔀　㄀㠀㜀䈀㤀㈀㔀㤀䄀䄀㤀㔀䘀㤀㈀㄀㔀㜀㐀㔀㠀㘀㤀　　㌀　䐀　　㔀㤀㐀㜀㌀㈀㌀㠀䈀㠀䈀㔀㠀㜀㠀㔀㐀䄀㈀㌀㠀䈀ഀഀ
    InstallerType: exe
਍    匀眀椀琀挀栀攀猀㨀ഀഀ
      Silent: "--unattended"
਍      匀椀氀攀渀琀圀椀琀栀倀爀漀最爀攀猀猀㨀 ∀ⴀⴀ甀渀愀琀琀攀渀搀攀搀∀

List of affected files:

• https://github.com/microsoft/winget-pkgs/blob/master/manifests/ChristianSchenk/MiKTeX/2.9.7442.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/CodeLite/CodeLite/14.0.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/DasKeyboard/DasKeyboard/3.2.6.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Files-UWP/FilesUWP/0.8.2.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Files-UWP/FilesUWP/0.9.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Files-UWP/FilesUWP/0.9.1.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Firetrust/MailWasherPro/7.12.29.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Gauge/Gauge/1.0.6.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/GeoGebra/GeoGebraClassic/6.0.583.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/GeoGebra/GeoGebraClassic6/6.0.583.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/GeoGebra/Geometry/6.0.583.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/GeoGebra/GraphingCalculator/6.0.583.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/GnuWin32/MakeforWindows/3.81.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Insomnia/InsomniaDesigner/2020.2.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/IsWiX/IsWiX/4.14.5.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/KKBOX/KKBOX/7.2.30.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Krisp/Krisp/1.14.3.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/LyX/LyX/2.3.4.4.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/MarkText/MarkText/0.16.1.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Microsoft/MouseAndKeyboardCenter/12.181.137.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Microsoft/dotnet/5.0.100-preview.4.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/MongoDB/Compass/Community/1.21.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/MongoDB/Compass/Full/1.21.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/MongoDB/Compass/Isolated/1.21.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/MongoDB/Compass/Readonly/1.21.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Mu/Mu/1.0.3.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/NagleCode/PacketSender/6.2.3.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/NimbusWeb/NimbusWeb/6.10.34.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/ONLYOFFICE/DesktopEditors/5.5.1.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/OSGeo/QGIS/3.12.3.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/PicoTorrent/PicoTorrent/0.18.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Plex/Plexamp/3.0.3.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/RStudio/RStudio/1.3.959.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Renode/Renode/1.9.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/RevoUninstaller/RevoUninstaller/2.1.5.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/RevoUninstaller/RevoUninstallerPro/4.3.1.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/SMPlayer/SMPlayer/20.4.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TGRMNSoftware/BulkRenameUtility/3.3.1.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TTYPlus/MTPutty/1.7.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TechSmith/Camtasia/2020.0.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TechSmith/Snagit/6.10.34.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TeraTermProject/TeraTerm/4.105.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TheHandBrakeTeam/HandBrake/1.3.2.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/TonyPottier/ImageView/2.0.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/VivaldiTechnologies/Vivaldi/3.0.1874.38.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Zap/Zap/2.9.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/Zettlr/Zettlr/1.6.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/canonical/MicroK8s/2.0.0.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/canonical/Multipass/1.2.1.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/gnupg/GnuPG/2.2.20.yaml
• https://github.com/microsoft/winget-pkgs/blob/master/manifests/isaaclevin/presencelight/2.0.49.yaml

I already wrote something to fix this, however i'm limited to one file per PR, so i will have to create 51 pull requests.

@KevinLaMS @denelon I would prefer to create a branch where i push all fixed files at once. When done, I would link it so you could grab and merge it. Or is there no way around the 1 file / PR limitation?

[SoftCreatR]

Here's the commit with the fixed files: https://github.com/SoftCreatR/winget-pkgs/commit/0ecaf9679725921d2f63f37c676c273f6d636872

[JamieMagee]

There are also significantly more that are using us-ascii encoding

$ find manifests/ -type f -exec file --mime {} \; | grep "charset=us-ascii" | wc -l
553

Even more than are using UTF-8

$ find manifests/ -type f -exec file --mime {} \; | grep "charset=utf-8" | wc -l
156

I think requiring UTF-8 encoding is probably the best way forward.

[SoftCreatR]

I just found/fixed these, because they cannot be read on a linux machine. Good examples are https://wingetit.com and https://winstall.app where affected manifests are not listed. On https://www.winget.it I already fixed it, regardless of my PR.