Highlight tests fail with >=tree-sitter 0.21

[MatthewGentoo]

With tree-sitter 0.20.6 as used by the CI, the tests all pass for me. However when upgrading to tree-sitter 0.21.0 and later, some of the highlight tests fail.

  attributes:
    ✓ Target attribute
    ✓ Attribute with multiple values
  cases:
    ✓ Case examples
    ✓ Case examples
    ✓ Pattern matching binaries with 'as'
    ✓ Case with boolean negation in a guard
  constants:
    ✓ Constants
    ✓ Public constants
    ✓ Scientific notation
  custom_types:
    ✓ Parser example custom types
    ✓ Other custom type examples
    ✓ Public custom type definitions
    ✓ Public opaque custom type definitions
  destructuring:
    ✓ Case with spread
  expressions:
    ✓ Bit-string expression
    ✓ Boolean Negation
    ✓ Integer Negation
    ✓ Concatenation
    ✓ Todo and panic 'as' with string expressions
    ✓ Nested field access
  external_functions:
    ✓ External functions
    ✓ Public external functions
    ✓ External function with attribute syntax
  external_types:
    ✓ External types
    ✓ Public external types
  functions:
    ✓ Function examples
    ✓ Public function examples
    ✓ Basic functions
    ✓ Cases
    ✓ Let expressions
    ✓ Complex binary expressions
    ✓ Complex nesting of field and tuple access
    ✓ Unusual function invocations
    ✓ Various discard variables
    ✓ Weird lists
    ✓ Comment in string
  imports:
    ✓ Imports
    ✓ Unqualified imports
    ✓ Aliased imports
    ✓ Type imports
    ✓ Discard module imports
  pipes:
    ✓ Pipes
  statements:
    ✓ Use
  strings:
    ✓ Escape sequences
  targets:
    ✓ Target groups
    ✓ Target group edge cases
  type_aliases:
    ✓ Type aliases
    ✓ Public type aliases
    ✓ Public opaque type aliases
  whole_files:
    ✓ Excerpt from stdlib's base.gleam
    ✓ Excerpt from stdlib's bool.gleam
    ✓ Trailing commas
syntax highlighting:
Warning: you should add a `highlights` entry pointing to the highlights path in `tree-sitter` language list in the grammar's package.json
See more here: https://tree-sitter.github.io/tree-sitter/syntax-highlighting#query-paths

    ✓ bit_strings.gleam (21 assertions)
    ✗ constants.gleam
      Failure - row: 5, column: 18, expected highlight 'warning', actual highlights: 'string.escape'
    ✗ destructuring.gleam
      Failure - row: 0, column: 13, expected highlight 'variable.parameter', actual highlights: 'variable'
    ✗ expressions.gleam
      Failure - row: 9, column: 3, expected highlight 'operator', actual highlights: 'punctuation.delimiter'
    ✗ functions.gleam
      Failure - row: 0, column: 7, expected highlight 'function', actual highlights: 'variable'
    ✗ modules.gleam
      Failure - row: 4, column: 22, expected highlight 'module', actual highlights: 'variable'
    ✗ records.gleam
      Failure - row: 8, column: 12, expected highlight 'variable.parameter', actual highlights: 'variable'
    ✗ reserved.gleam
      Failure - row: 0, column: 0, expected highlight 'error', actual highlights: 'variable'

There's something called out in the changelog of tree-sitter 0.21 which might be the issue:

Breaking

• Remove the apply-all-captures flag, make last-wins precedence the default \ \ NOTE: This change might cause breakage in your grammar's highlight tests. Just flip the order around of the relevant queries, and keep in mind that the last query that matches will win.

[MatthewGentoo]

I've made a bit of progress and got most of the tests to pass. It looks like the changelog entry is accurate. Here is a patch for highlight.scm that makes all tests but functions.gleam pass (just shuffling things around).

diff --git a/queries/highlights.scm b/queries/highlights.scm
index 20f809f..0365244 100644
--- a/queries/highlights.scm
+++ b/queries/highlights.scm
@@ -7,22 +7,11 @@
 (constant
   name: (identifier) @constant)

-; Modules
-(module) @module
-(import alias: (identifier) @module)
-(remote_type_identifier
-  module: (identifier) @module)
-(remote_constructor_name
-  module: (identifier) @module)
-((field_access
-  record: (identifier) @module
-  field: (label) @function)
- (#is-not? local))
+; Variables
+(identifier) @variable
+(discard) @comment.unused

 ; Functions
-(unqualified_import (identifier) @function)
-(unqualified_import "type" (type_identifier) @type)
-(unqualified_import (type_identifier) @constructor)
 (function
   name: (identifier) @function)
 (external_function
@@ -37,6 +26,18 @@
    right: (identifier) @function)
  (#is-not? local))

+; Modules
+(module) @module
+(import alias: (identifier) @module)
+(remote_type_identifier
+  module: (identifier) @module)
+(remote_constructor_name
+  module: (identifier) @module)
+((field_access
+  record: (identifier) @module
+  field: (label) @function)
+ (#is-not? local))
+
 ; "Properties"
 ; Assumed to be intended to refer to a name for a field; something that comes
 ; before ":" or after "."
@@ -59,12 +60,17 @@
 ; Data constructors
 (constructor_name) @constructor

+; Import
+(unqualified_import (identifier) @function)
+(unqualified_import (type_identifier) @constructor)
+(unqualified_import "type" (type_identifier) @type)
+
 ; Literals
 (string) @string
+(escape_sequence) @string.escape
 ((escape_sequence) @warning
  ; Deprecated in v0.33.0-rc2:
  (#eq? @warning "\\e"))
-(escape_sequence) @string.escape
 (bit_string_segment_option) @function.builtin
 (integer) @number
 (float) @number
@@ -75,10 +81,6 @@
 ((identifier) @error
  (#match? @error "^(auto|delegate|derive|else|implement|macro|test|echo)$"))

-; Variables
-(identifier) @variable
-(discard) @comment.unused
-
 ; Keywords
 [
   (visibility_modifier) ; "pub"
@@ -99,12 +101,6 @@
   "use"
 ] @keyword

-; Operators
-(binary_expression
-  operator: _ @operator)
-(boolean_negation "!" @operator)
-(integer_negation "-" @operator)
-
 ; Punctuation
 [
   "("
@@ -128,3 +124,9 @@
   "-"
   "<-"
 ] @punctuation.delimiter
+
+; Operators
+(binary_expression
+  operator: _ @operator)
+(boolean_negation "!" @operator)
+(integer_negation "-" @operator)

The failure I'm stuck with is that function names in calls are being highlighted as property accesses, e.g. this test:

  string.replace(in: original, each: pattern, with: replacement)
  // <- module
  //        ^ function

    ✗ functions.gleam
      Failure - row: 24, column: 12, expected highlight 'function', actual highlights: 'property'

[the-mikedavis]

It looks like the #is-not? local check is causing the issue here. I'll take a closer look when I get a chance. This might be a bug upstream in tree-sitter