"Unicode Highlight" feature suggests incorrect locale for ignoring common characters

[imax9000]

Type: Bug

1) Set Windows language to Ukrainian 2) Write a string constant "росія - держава-терорист" in an open code file 3) Hover over highlighted characters and click "Adjust settings"

In the popup it suggests to ignore characters common for "ru" locale, despite it not being used anywhere in the system. Moreover, manually specifying "uk" or "ua" locale in settings has no effect at all.

VS Code version: Code 1.80.1 (74f6148eb9ea00507ec113ec51c489d6ffb4b771, 2023-07-12T17:22:07.651Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 6.1.0-1-amd64 Remote OS version: Linux x64 6.1.0-1-amd64

System Info

|Item|Value| |---|---| |CPUs|AMD Ryzen 5 7600 6-Core Processor (12 x 3793)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|63.14GB (50.15GB free)| |Process Argv|--crash-reporter-id 6ae8d069-3bb0-4a67-82d2-6493bb923799| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: brick| |OS|Linux x64 6.1.0-1-amd64| |CPUs|Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (8 x 1200)| |Memory (System)|15.55GB (9.56GB free)| |VM|0%| |Item|Value| |---|---| |Remote|SSH: brick| |OS|Linux x64 6.1.0-1-amd64| |CPUs|Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (8 x 1197)| |Memory (System)|15.55GB (9.56GB free)| |VM|0%|

Extensions (19)

Extension|Author (truncated)|Version ---|---|--- jq-syntax-highlighting|jq-|0.0.2 autoconf|mae|0.2.0 remote-containers|ms-|0.299.0 remote-ssh|ms-|0.102.0 remote-ssh-edit|ms-|0.86.0 remote-wsl|ms-|0.80.2 vscode-remote-extensionpack|ms-|0.24.0 remote-explorer|ms-|0.4.1 remote-server|ms-|1.3.0 rewrap|stk|1.16.3 vscode-jq-playground|dav|4.3.5 go|gol|0.39.1 jq-syntax-highlighting|jq-|0.0.2 cmake-tools|ms-|1.14.34 cpptools|ms-|1.16.3 cpptools-extension-pack|ms-|1.3.0 advanced-new-file|pat|1.2.2 datetime|rid|2.2.2 cmake|twx|0.0.17 (1 theme extensions excluded)

A/B Experiments

``` vsliv368cf:30146710 vsreu685:30147344 python383:30185418 vspor879:30202332 vspor708:30202333 vspor363:30204092 vslsvsres303:30308271 vserr242:30382549 pythontb:30283811 vsjup518:30340749 pythonptprofiler:30281270 vshan820:30294714 vstes263cf:30335440 vscod805:30301674 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 vsaa593cf:30376535 pythonvs932:30410667 vscaac:30438847 vsclangdc:30486549 c4g48928:30535728 dsvsc012cf:30540253 pynewext54:30695312 azure-dev_surveyone:30548225 282f8724:30602487 f6dab269:30613381 a9j8j154:30646983 showlangstatbar:30737416 vsctsb:30748421 03d35959:30757346 7ij38806:30736111 pythonfmttext:30731395 pythoncmv:30756943 fixshowwlkth:30771522 hideindicator:30785051 pythongtdpath:30769146 i26e3531:30792625 gsofa:30797621 pythonnosmt12:30797651 e537b577:30795824 dsvsc013:30795093 dsvsc014:30797589 ```

[vscodenpa]

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.80.2. Please try upgrading to the latest version and checking whether this issue remains.

Happy Coding!

[imax9000]

Reproduces on 1.80.2 as well

[imax9000]

Looks like the culprit is the assumption that people only ever use vscode with localization set to their native language and don't ever have to type anything in any other language :) Which leads to https://github.com/hediet/vscode-unicode-data/blob/main/index.ts#L131 only exporting data for locales supported by vscode-loc. And that data then gets copy-pasted into vscode sources https://github.com/microsoft/vscode/blob/9a281018181dca942cc46c03f9795be00912e38d/src/vs/base/common/strings.ts#L1092

[image72]

same above problem, find string in vscode encoding: UTF-8 that strings all strict equal, but in find not match.

pcs\u53d8\u6d41\u5668
pcs变流器

The following are examples of testing the equality of these two strings in several common programming languages: 下面是几种常见编程语言中测试这两个字符串相等性的示例：

// JavaScript
const str1 = "pcs\u53d8\u6d41\u5668";
const str2 = "pcs变流器";

console.log(str1 === str2); // output: true

# Python
str1 = "pcs\u53d8\u6d41\u5668"
str2 = "pcs变流器"

print(str1 == str2)  # output: True

// Java
public class StringComparisonExample {
    public static void main(String[] args) {
        String str1 = "pcs\u53d8\u6d41\u5668";
        String str2 = "pcs变流器";

        System.out.println(str1.equals(str2)); // output: true
    }
}

// Csharp
using System;

class StringComparisonExample
{
    static void Main()
    {
        string str1 = "pcs\u53d8\u6d41\u5668";
        string str2 = "pcs变流器";

        Console.WriteLine(str1 == str2); // output: True
    }
}

These examples test the equality of these two strings in different programming languages and output the results. Based on these examples, it can be seen that these two strings are considered equal in various programming languages. 这些示例在不同的编程语言中测试了这两个字符串的相等性，并输出了结果。根据这些示例，可以看到这两个字符串在各个编程语言中都被认为是相等的。

• Version: 1.81.0
• Commit: 6445d93c81ebe42c4cbd7a60712e0b17d9463e97
• Date: 2023-08-02T12:38:28.722Z
• Electron: 22.3.18
• ElectronBuildId: 22689846
• Chromium: 108.0.5359.215
• Node.js: 16.17.1
• V8: 10.8.168.25-electron.0
• OS: Darwin arm64 22.5.0

[image72]

In most programming languages, these two strings are exactly equal when compared. Whether you use Unicode-encoded escaped characters to represent Chinese characters, or you use UTF-8 encoded Chinese characters directly, they are treated as the same string when compared.

[image72]

// GO

package main

import "fmt"

func main() {
    str1 := "pcs变流器"
    str2 := "pcs\u53d8\u6d41\u5668"

    fmt.Println(str1 == str2) // output: true
}

// Rust

fn main() {
    let str1 = "pcs变流器";
    let str2 = "pcs\u{53d8}\u{6d41}\u{5668}";

    println!("{}", str1 == str2); // output: true
}

// C

#include <stdio.h>
#include <string.h>

int main() {
    char str1[] = "pcs变流器";
    char str2[] = "pcs\u53d8\u6d41\u5668";

    if (strcmp(str1, str2) == 0) {
        printf("Strings are equal\n"); // output: Strings are equal
    } else {
        printf("Strings are not equal\n");
    }

    return 0;
}

[image72]

that problem also alike in search files by name, Search options

[imax9000]

Sorry, @image72 , I think yours is a separate issue. Mine is about incorrect locale detection in UnicodeHighlight specifically. Yours seems to be about treatment of Unicode and/or decoding Unicode escape sequences in the UI in general