TSX (web-mode) tree-sitter-hl not working

loafofpiecrust commented 3 years ago

I've been using tree-sitter-hl-mode for rustic-mode and js2-mode and it seems to be working great. When I try to enable it in a .tsx file (typescript-tsx-mode), however, I get an error and the mode fails to start.

font-lock-eval-keywords: Wrong number of arguments: #<subr quote>, 0

Here's the backtrace:

Debugger entered--Lisp error: (wrong-number-of-arguments #<subr quote> 0)
  quote()
  font-lock-eval-keywords(quote)
  tree-sitter-hl--minimize-font-lock-keywords()
  #<subr tree-sitter-hl--setup>()
  apply(#<subr tree-sitter-hl--setup> nil)
  tree-sitter-hl--setup()
  tree-sitter-hl-mode(toggle)
  funcall-interactively(tree-sitter-hl-mode toggle)
  command-execute(tree-sitter-hl-mode record)
  counsel-M-x-action("tree-sitter-hl-mode")
  #f(compiled-function (x) #<bytecode -0x20a815e057386a3>)("tree-sitter-hl-mode")
...

ubolonton commented 3 years ago

Can you provide more details to reproduce the issue? For example:

Your OS and Emacs build, e.g. the result of (emacs-version).
The installation mechanism, e.g. package.el vs. straight.el.
What is typescript-tsx-mode? Where does it come from? It doesn't seem to be on MELPA.
A minimal init.el with the issue. This can be a good start: https://gist.github.com/ubolonton/a4cfe85782f40e280570b3ef393ded5f
The values of font-lock-keywords and font-lock-defaults before enabling tree-sitter-hl.

loafofpiecrust commented 3 years ago

Sorry that I'm just now responding to this. It still doesn't seem to work on the latest commit.

Result of M-x emacs-version: GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.21, cairo version 1.16.0), this is native-comp running on NixOS 20.09 with doom.
Installation: straight.el, which comes with doom
typescript-tsx-mode comes from DOOM, just derived from `web-mode, described like so:
```
Major mode derived from web-mode by define-derived-mode.
```

It inherits all of the parent's attributes, but has its own keymap, abbrev table and syntax table:

typescript-tsx-mode-map, typescript-tsx-mode-abbrev-table and typescript-tsx-mode-syntax-table


Here's the relevant part of my config:
```el
(use-package tree-sitter
  :hook ((rustic-mode python-mode json-mode js-mode js2-mode typescript-mode go-mode sh-mode) . tree-sitter-mode)
  :config
  (require 'tree-sitter-langs)
  (push '(typescript-tsx-mode . typescript) tree-sitter-major-mode-language-alist)
  (add-hook 'tree-sitter-after-on-hook #'tree-sitter-hl-mode))

Here's font-lock-keywords:

(t
 (web-mode-fontify
  (whitespace-point--flush-used)
  ("\\( +\\)" 1 whitespace-tab t))
 (web-mode-fontify
  (0 font-lock-keyword-face))
 (whitespace-point--flush-used
  (0 nil))
 ("\\(  +\\)"
  (1 whitespace-tab t)))

Here's font-lock-defaults:

('(web-mode-fontify)
 t)

EDIT 2: I just tried it with plain web-mode and got the same error that I put in my first comment.

srcreigh commented 3 years ago

typescript and typescript-tsx have different syntax see tree-sitter-typescript

It's a conflict between <div></div> JSX syntax and the <number>foo Typescript casting syntax. In typescript-tsx you cast using foo as number. see here https://www.typescriptlang.org/docs/handbook/jsx.html#the-as-operator

I think the fix is to have separate mode, grammar, and HL queries for typescript-tsx

I also get parsing errors from a normal TSX file using typescript-mode, even though it compiles correctly.

srcreigh commented 3 years ago

Turns out tree-sitter-langs already comes with the TSX grammar, you can set it up with a typescript-tsx mode.

Still missing TSX highlighting queries I think. but this is a pretty good start.

Here's my config that lets me parse TSX

;; Typescript
(setq typescript-indent-level 2)
;; you can also use the DOOM one if you wish
(define-derived-mode typescript-tsx-mode typescript-mode "TSX"
  "Major mode for editing TSX files.

Refer to Typescript documentation for syntactic differences between normal and TSX
variants of Typescript.")
(add-to-list 'auto-mode-alist '("\\.tsx?\\'" . typescript-tsx-mode))

(use-package tree-sitter
  :ensure t)

(use-package tree-sitter-langs
  :ensure t
  :after tree-sitter
  :config
  (tree-sitter-require 'tsx)
  (add-to-list 'tree-sitter-major-mode-language-alist '(typescript-tsx-mode . tsx)))

With this config, if I open a typescript file and check M-x tree-sitter-debug-mode, I get some nice jsx nodes in the parse tree:

program:
  import_statement:
    import_clause:
      namespace_import:
        identifier:
    string:
  import_statement:
    string:
  import_statement:
    import_clause:
      identifier:
    string:
  lexical_declaration:
    variable_declarator:
      identifier:
      arrow_function:
        formal_parameters:
        statement_block:
          return_statement:
            parenthesized_expression:
              jsx_element:
                jsx_opening_element:
                  identifier:
                jsx_text:
                jsx_element:
                  jsx_opening_element:
                    identifier:
                  jsx_text:
                  jsx_closing_element:
                    identifier:
                jsx_text:
                jsx_element:
                  jsx_opening_element:
                    identifier:
                    jsx_attribute:
                      property_identifier:
                      string:
                  jsx_text:
                  jsx_element:
                    jsx_opening_element:
                      identifier:
                      jsx_attribute:
                        property_identifier:
                        string:
                    jsx_text:
                    jsx_closing_element:
                      identifier:
                  jsx_text:
                  jsx_element:
                    jsx_opening_element:
                      identifier:
                      jsx_attribute:
                        property_identifier:
                        string:
                    jsx_text:
                    jsx_closing_element:
                      identifier:
                  jsx_text:
                  jsx_element:
                    jsx_opening_element:
                      identifier:
                      jsx_attribute:
                        property_identifier:
                        string:
                    jsx_text:
                    jsx_closing_element:
                      identifier:
                  jsx_text:
                  jsx_closing_element:
                    identifier:
                jsx_text:
                jsx_closing_element:
                  identifier:
  export_statement:
    identifier:

dpassen commented 3 years ago

@srcreigh I wonder if a very small MELPA package that sets up the derived mode as a target for hooks wouldn't be such a bad idea.

vconcat commented 3 years ago

https://github.com/tree-sitter/tree-sitter-javascript/blob/master/queries/highlights-jsx.scm

simple jsx highlights.scm, can be used with existed typescript and javascript query.

Update: the query for typescript may need some changes since the grammer for typescript and tsx are different.

sangaline commented 3 years ago

This is a crude workaround, but if you set

(setq tree-sitter-hl-use-font-lock-keywords nil)

then you'll bypass the logic in tree-sitter-hl--minimize-font-lock-keywords that causes the error. Highlighting works correctly after that.

My value of font-lock-keywords when I get the error is:

 (("\\_<[[:digit:]]+\\(?:\\.[0-9]*\\)?\\_>" quote highlight-numbers-number)
  web-mode-fontify
  ((lambda
     (bound)
     (hl-todo--search nil bound))
   (1
    (hl-todo--get-face)
    t t))
  (whitespace-point--flush-used)
  ("\\(  +\\)" 1 whitespace-tab t))

Unlike what @loafofpiecrust posted, this does include quote as seen in the error, so maybe that's helpful for somebody who understands the tree-sitter-hl--minimize-font-lock-keywords code better than I do.

Abdillah commented 3 years ago

Thanks @sangaline for the workaround.

Oh It seems my concern below actually has been handled well. The problem came on the (dolist (keyword keywords-list) ...) construct, which still iterate on quote of '(web-mode-fontify).

Previously:

I'm exploring a bit and found this bug is mainly due to this construct, right?

;; In web-mode
(setq font-lock-defaults '('(web-mode-fontify) t))
;; In emacs-tree-sitter
(setq keywords-spec (car font-lock-defaults))
(car (cdr keywords-spec)))
;; Which led to an equivalent
(car (car '('(web-mode-fontify)))) ;; => quote

If font-lock-defaults value allow to have quote, I think any mode will be broken on this part. Maybe someone has the idea on what value font-lock-defaults allows?

OrionRandD commented 3 years ago

Perhaps, related to this issue? https://hungyi.net/posts/use-emacs-tree-sitter-doom-emacs/

Abdillah commented 3 years ago

Yes, albeit not a solution. I remember started using this after reading the post. I'm assuming that web-mode quite dependent on font-lock or syntax table, thus sometime using tree-sitter make some commands go awry.

mkcode commented 1 year ago

In case anyone else comes across this -

The following pattern works well enough for me as a workaround for tree-sitter in a derived web-mode.

(defun typescript-tsx-mode-fix-tree-sitter()
   (set (make-local-variable 'tree-sitter-hl-use-font-lock-keywords) nil))
(add-hook 'typescript-tsx-mode-hook #'typescript-tsx-mode-fix-tree-sitter)

emacs-tree-sitter / tree-sitter-langs

TSX (web-mode) tree-sitter-hl not working #23