Non-ASCII character may provoke shifted error highlighting

[matze-dd]

Information

VIM version

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 18 2020 18:29:15) Inklusive der Patches: 1-1453

VIM - Vi IMproved 8.2 (2019 Dec 12 kompiliert am Mar 30 2020 22:53:24) Inklusive der Patches: 1-486

Operating Systems:

• Cygwin on Windows 7 (32 bit): Vim 8.2
• Lubuntu 18.04 (32 bit): Vim 8.0

What went wrong

If I use for instance the LanguageTool 'linter' on a plain-text file

This is a test..

then the double trailing dot is correctly highlighted as error location. As soon as I replace 'a' with 'ä', the hightlighting of '..' shifts by one position to the left. This marking of 't.' is not corrected with the next linter run, and immeditiately goes right if I replace the 'ä' with command 'r a'.

Perhaps, the non-ASCII character 'ä' breaks the one-to-one relation between character and byte offsets? As can be seen in the ALEInfo below, LanguageTool correctly reports the error location both for 'a' and 'ä'.

This problem appears whenever a non-ASCII character is present on the same text line at a place before the error highlighting. It disturbs in languages that commonly use such characters, in my case German.

Many thanks for this otherwise fantastic framework!

:ALEInfo

 Current Filetype: text
Available Linters: ['alex', 'languagetool', 'proselint', 'redpen', 'textlint',
'vale', 'writegood']
   Linter Aliases:
'writegood' -> ['write-good']
  Enabled Linters: ['languagetool']
 Suggested Fixers:
  'remove_trailing_lines' - Remove all blank lines at the end of a file.
  'textlint' - Fix text files with textlint --fix
  'trim_whitespace' - Remove all trailing whitespace characters at the end of e
very line.
 Linter Variables:
 Global Variables:
let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = 0
let g:ale_command_wrapper = ''
let g:ale_completion_delay = v:null
let g:ale_completion_enabled = 0
let g:ale_completion_max_suggestions = v:null
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%code: %%s'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_insert_leave = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'normal'
let g:ale_linter_aliases = {}
let g:ale_linters = {'text': ['languagetool']}
let g:ale_linters_explicit = 0
let g:ale_list_vertical = 0
let g:ale_list_window_size = 10
let g:ale_loclist_msg_format = '%code: %%s'
let g:ale_lsp_root = {}
let g:ale_max_buffer_history_size = 20
let g:ale_max_signs = -1
let g:ale_maximum_file_size = v:null
let g:ale_open_list = 0
let g:ale_pattern_options = v:null
let g:ale_pattern_options_enabled = v:null
let g:ale_set_balloons = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 0
let g:ale_sign_error = '>>'
let g:ale_sign_info = '--'
let g:ale_sign_offset = 1000000
let g:ale_sign_style_error = '>>'
let g:ale_sign_style_warning = '--'
let g:ale_sign_warning = '--'
let g:ale_sign_highlight_linenrs = 0
let g:ale_statusline_format = v:null
let g:ale_sign_style_warning = '--'
let g:ale_sign_warning = '--'
let g:ale_sign_highlight_linenrs = 0
let g:ale_statusline_format = v:null
let g:ale_type_map = {}
let g:ale_use_global_executables = v:null
let g:ale_virtualtext_cursor = 0
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1
  Command History:
(executable check - success) languagetool
(finished - exit code 0) ['/bin/bash', '-c', '''languagetool'' --autoDetect ''/
home/matthias/z.txt''']
<<<OUTPUT STARTS>>>
20:34:18.463 [main] DEBUG com.optimaize.langdetect.LanguageDetectorImpl - ==> [
DetectedLanguage[en:0.99999461651071]]
1.) Line 1, column 15, Rule ID: DOUBLE_PUNCTUATION
Message: Two consecutive dots
Suggestion: .
This is a test..
              ^^
Time: 5253ms for 1 sentences (0.2 sentences/sec)
<<<OUTPUT ENDS>>>
(finished - exit code 0) ['/bin/bash', '-c', '''languagetool'' --autoDetect ''/
home/matthias/z.txt''']
<<<OUTPUT STARTS>>>
20:35:03.927 [main] DEBUG com.optimaize.langdetect.LanguageDetectorImpl - ==> [
DetectedLanguage[en:0.9999606576754977]]
1.) Line 1, column 15, Rule ID: DOUBLE_PUNCTUATION
Message: Two consecutive dots
Suggestion: .
This is ä test..
              ^^
Time: 5067ms for 1 sentences (0.2 sentences/sec)
<<<OUTPUT ENDS>>>

[matze-dd]

The issue may be solved by the linter component. It should take care that column positions are passed as byte offsets that are expected by Vim function matchaddpos() (used by ALE for highlighting).

(Not sure whether this issue should be closed.)

[matze-dd]

In file autoload/ale/handlers/languagetool.vim, function ale#handlers#languagetool#HandleOutput(), a field 'vcol' should be set to 1 in dict l:item.