Use wordPattern inspired from the markdown extension

[jlelong]

Related to microsoft/vscode/issues/144913

@Yarakashi-Kikohshi Could you test this new wordPattern?

Close #17

[Yarakashi-Kikohshi]

@jlelong

I'm a user of VS Code and LaTeX Workshop, but I don't know how to test this pull request works. I'm sorry, this is my lack of knowledge.

If you can show me a detailed way to test that it works, I can test that it works.

---

On the other hands, although this is an unusual method, I have made edits wordPattern in ~\.vscode\extensions\james-yu.latex-workshop-8.23.0\syntax\latex-language-configuration.json in currently installed LaTeX Woskrhop (8.23). In this case, the problem discussed in https://github.com/microsoft/vscode/issues/144913 don't seem to arise.

[jlelong]

You can actually directly edit ~\.vscode\extensions\james-yu.latex-workshop-8.23.0\syntax\latex-language-configuration.json as you did but note that the wordPattern used by LaTeX-Workshop and vscode basic LaTeX support are slightly different (in LW, we include ..., __ and ** because they are snippet prefixes)

In you want to directly test this PR,

• clone https://github.com/jlelong/vscode-latex-basics.git and checkout the branch 17-wordpattern

  git clone https://github.com/jlelong/vscode-latex-basics.git
  git checkout -b 17-wordpattern origin/17-wordpattern

• In VS Code, press F5 (or run Debug: Start Debugging from the command palette) and a new window will pop up where you can test the extension.

@Yarakashi-Kikohshi I am pretty convinced that this PR solves https://github.com/microsoft/vscode/issues/144913, but I really want to make sure it does not introduce other problems.

[Yarakashi-Kikohshi]

@jlelong, Thanks for the very clear explanation. I was able to test it successfully. I hope this test will be useful.

I tested with all extensions disabled.

Summary

• Occurrence highlighting

  • [ ] If the following characters and full-width space is on the left, the highlight does not show.

    ＃＄％＆＝～＾｜＜＞；：・￥＠＿＋＊　

    See gif of 、, 。, ＝ and ～ tested as representative:

    

• Suggestion

  • [x] Snippet works fine.
    • Snippets are not suggested immediately after strings, but this is normal behaviour.
  • [x] Snippet works fine in tabstops in snippet.

Test procedure

Note that these tests only consider Ascii characters and Japanese at best. There are numerous languages in the world, but only two have been tested.

1. I have drawn up lists of "the patterns in which suggestions are displayed" and "the characters immediately preceding the pattern in which suggestions are displayed".
2. I have tested all the patterns from each of the two lists.
3. Visually check Suggestion display and Occurrence highlighting.

list of the patterns in which suggestions are displayed

• Snippet of teacher
• Snippet of せんせい

Snippets

```json { "Teacher in Japanese": { "prefix": "せんせい", "body": "先生", "description": "先生: Teacher in Japanese" }, "Teacher in English": { "prefix": "teacher", "body": "TEACHER", "description": "Teacher in English" }, "Tabstop": { "prefix": "tabstop", "body": "test{$1}{$2}$0", "description": "test: Tabstops in snippet" } } ```

list of the characters immediately preceding the pattern in which suggestions are displayed

• Begining of line (Immediately after and after some whitespace characters)

• Tabstops in snippet

• Brakets (Immediately after each)

  {}[]()

• Ascii characters (Immediately after each)

  [*] Backslash is tested as double-backslash

  '`"#$%&=~^|<>;:,.!?/\\@+*_

• Multi-byte brakets (Immediately after each)

  ｛｝（）［］「」

• Multi-byte characters (Immediately after each)

  、。，．！？＃＄％＆＝～＾｜＜＞；：・￥＠＿＋＊

• Strings (Immediately after and after a whitespace)

  [**] Alphabet, Arabic numeral, Full-width alphabet, Full-width arabic numeral, Hiragana, Katakana, Kanji

  alphabet
  123
  ａｌｐｈａｂｅｔ
  １２３
  ひらがな
  カタカナ
  漢字

• Full-width space (Immediately after and after a whitespace characters)

Complete file for testing

```latex % Snippet of `teacher' % Snippet of `せんせい' %% IntelliSense starting with `\' %% IntelliSense starting with `@' %% IntelliSense starting with `**' %% IntelliSense starting with `__' %% IntelliSense starting with `...' \documentclass{article} \begin{document} {} [] () ｛｝ （） ［］ 「」 ' ` " # $ $ % & = ~ ^ | < > ; : , . ! ? / \\ @ + * _ 、 。 ， ． ！ ？ ＃ ＄ ％ ＆ ＝ ～ ＾ ｜ ＜ ＞ ； ： ・ ￥ ＠ ＿ ＋ ＊ alphabet 123 ａｌｐｈａｂｅｔ １２３ ひらがな カタカナ 漢字 　 \end{document} ```

About LaTeX Workshop's IntelliSense

I have also accidentally tested LaTeX Workshop's IntelliSense, so I am writing alongside it.

LaTeX Workshop's intelligences are as follows:

• IntelliSense provided by LaTeX Workshop
  • Starting with \
  • Starting with @
  • **
  • __
  • ...

The results of the suggestion display were in addition to the results above:

• Suggestion of IntelliSense provided by LaTeX Workshop
  • [x] IntelliSense starting with \ and @ works fine.
  • [ ] ** and __ and ... did not show suggestion in every situation.
    • It will be displayed when you press Ctrl+Space.
    • In ver. 8.23.0, suggestions are displayed everywhere.

[jlelong]

Thank you so much for your very thorough tests and detailed report.

** and __ and ... did not show suggestion in every situation.

This is because we do not use the same wordPattern in vscode-latex-basics and in LaTeX-Workshop. In LaTeX-Workshop, we add (**|__|\.\.\.) to the wordPattern to make sure these snippets show up automatically, see https://github.com/jlelong/LaTeX-Workshop/commit/e0ba934c9e0457861ce463c43a13deb9a34e4577. If you want to test it inside LaTeX-Workshop, the simplest approach is to directly edit ~\.vscode\extensions\james-yu.latex-workshop-8.23.0\syntax\latex-language-configuration.json because running the git version of LaTeX-Workshop requires npm (see https://github.com/James-Yu/LaTeX-Workshop/blob/master/CONTRIBUTING.md)