search

radian-software / straight.el

🍀 Next-generation, purely functional package manager for the Emacs hacker.
MIT License
2.74k stars 150 forks source link

tree-sitter tsc module will be re-built instead of being downloaded with straight-rebuild-package #1131

Open scturtle opened 11 months ago

scturtle commented 11 months ago

What's wrong

Tree-sitter tsc module will be re-built instead of being downloaded with straight-rebuild-package.

Directions to reproduce.

.emacs.d/init.el:

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'use-package)
(setq straight-use-package-by-default t)

(use-package tree-sitter)
(use-package tree-sitter-langs)

Run emacs --init-directory /tmp/.emacs.d the first time, the tsc module should be downloaded correctly.

In Emacs, run C-u M-x straight-rebuild-package and enter tree-sitter to rebuild tree-sitter and tsc. The tsc module will be built from source this time.

details

*straight-byte-compilation*:

In toplevel form:
tree-sitter-query.el:15:2: Error: Module is not GPL compatible: "/tmp/.emacs.d/straight/build/tree-sitter-langs/bin/scheme.dylib"
Compiling /tmp/.emacs.d/straight/build/tree-sitter/tree-sitter.el...
Done (Total of 6 files compiled, 1 failed, 1 skipped)

it seems that .emacs.d/straight/build/tree-sitter-langs/bin is included in load-path while doing straight-rebuild-package. Then during downloading tsc module with url-copy-file, not knowing why, Emacs will try to load the tree-sitter language files *.dylib / *.so as Emacs module and fail. Then tree-sitter will turn to compile tsc from source.

Version information

raxod502 commented 11 months ago

I think you're right. Furthermore I think the reason the subdirectory is (incorrectly) added to load-path is this:

https://github.com/radian-software/straight.el/blob/e0f3ba8e3fcf66b746c99418e916a3d2fe7a3c35/straight.el#L5270

It seems that as a convenience we add all subdirectories in ~/.emacs.d/straight/build to load-path, which isn't right. We should only be adding the packages that are dependencies of the one being built; furthermore, even amongst those, we shouldn't be adding subdirectories recursively, only the top level ones.

progfolio commented 11 months ago

I think you're right. Furthermore I think the reason the subdirectory is (incorrectly) added to load-path is this:

https://github.com/radian-software/straight.el/blob/e0f3ba8e3fcf66b746c99418e916a3d2fe7a3c35/straight.el#L5270

It seems that as a convenience we add all subdirectories in ~/.emacs.d/straight/build to load-path, which isn't right.

This was a compromise added in #812. Notes from 01eb849125afca684ebefd4a62793df7484685c2:

Previous method involved looking up each package's dependencies. It did not account for local-repo names and transitive dependencies.

I can't recall why the :local-repo keyword was causing issues (if it was at all). It looks like straight--symlink-package uses (straight--build-dir package), so I don't think it should be a problem.

We should only be adding the packages that are dependencies of the one being built; furthermore, even amongst those, we shouldn't be adding subdirectories recursively, only the top level ones.

Agreed. Now that we have straight-dependencies we can easily get the transitive dependencies (it looks like my initial attempt relied on straight--get-dependencies, which did not return transitive dependencies, leading to the compromise).

1132 seems to work for the case in this issue (see below), but it could use some more testing.

Test Case ```emacs-lisp (straight-bug-report :pre-bootstrap (setq straight-repository-user "progfolio" straight-repository-branch "fix/compile") :post-bootstrap (straight-use-package 'tree-sitter) (straight-use-package 'tree-sitter-langs) (straight-rebuild-package "tree-sitter" 'recusrive) (with-current-buffer (get-buffer-create "*straight-byte-compilation*") (message "%S" (buffer-substring-no-properties (point-min) (point-max))))) ```
Output ```emacs-lisp Bootstrapping straight.el... Bootstrapping straight.el...done Looking for gnu-elpa-mirror recipe → Cloning melpa... Looking for gnu-elpa-mirror recipe → Cloning melpa...done Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror... Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa... Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done Looking for emacsmirror-mirror recipe → Cloning el-get... Looking for emacsmirror-mirror recipe → Cloning el-get...done Looking for straight recipe → Cloning emacsmirror-mirror... Looking for straight recipe → Cloning emacsmirror-mirror...done Building straight... Building straight...done Test run with version: prerelease (HEAD -> fix/compile, origin/fix/compile) 058bcdd 2023-11-17 Cloning elisp-tree-sitter (for tree-sitter)... Cloning elisp-tree-sitter (for tree-sitter)...done Building tree-sitter... Building tree-sitter → Building tsc... Building tree-sitter → Building tsc...done Building tree-sitter... Building tree-sitter...done Cloning tree-sitter-langs... Cloning tree-sitter-langs...done Building tree-sitter-langs... Building tree-sitter-langs...done " $ /usr/bin/emacs -Q --batch --eval \\ \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/straight/\\\") 'nil load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/straight/\\\" 0 'force))\" Checking /tmp/straight.el-test-86hhVW/straight/build/straight/... Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-autoloads.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-ert-print-hack.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-x.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight.el... Done (Total of 3 files compiled, 1 skipped) $ /usr/bin/emacs -Q --batch --eval \\ \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\") 'nil load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\" 0 'force))\" Checking /tmp/straight.el-test-86hhVW/straight/build/tsc/... Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-autoloads.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-dyn-get.el... In tsc-dyn-get--build-version: tsc-dyn-get.el:185:2: Warning: docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...') Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-obsolete.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc.el... tsc-dyn-get: Using source :github (:loaded nil :recorded nil :requested 0.18.0) tsc-dyn-get: Downloading https://github.com/emacs-tree-sitter/elisp-tree-sitter/releases/download/0.18.0/tsc-dyn.x86_64-unknown-linux-gnu.so Contacting host: github.com:443 Checking /tmp/straight.el-test-86hhVW/straight/build/tsc/src... Done (Total of 3 files compiled, 1 skipped) $ /usr/bin/emacs -Q --batch --eval \\ \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\") '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\") load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\" 0 'force))\" Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-autoloads.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-cli.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-debug.el... tsc-dyn-get: Using source :github (:loaded nil :recorded 0.18.0 :requested 0.18.0) tsc-dyn-get: Recorded version already satifies requested -> loading Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-extras.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-hl.el... In toplevel form: tree-sitter-hl.el:263:12: Warning: custom-declare-variable ‘tree-sitter-hl-face-mapping-function’ docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...') In tree-sitter-hl--highlight-region: tree-sitter-hl.el:487:12: Warning: ‘inhibit-point-motion-hooks’ is an obsolete variable (as of 25.1); use ‘cursor-intangible-mode’ or ‘cursor-sensor-mode’ instead In tree-sitter-hl-mode: tree-sitter-hl.el:639:2: Warning: docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...') Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-load.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-query.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter.el... Done (Total of 7 files compiled, 1 skipped) $ /usr/bin/emacs -Q --batch --eval \\ \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/\\\") '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\" \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\") load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/\\\" 0 'force))\" Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs-autoloads.el... Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs-build.el... In tree-sitter-langs-install-grammars: tree-sitter-langs-build.el:500:22: Warning: ‘unless’ with empty body In tree-sitter-langs-install-latest-grammar: tree-sitter-langs-build.el:550:2: Warning: docstring wider than 80 characters Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs.el... tsc-dyn-get: Using source :github (:loaded nil :recorded 0.18.0 :requested 0.18.0) tsc-dyn-get: Recorded version already satifies requested -> loading tree-sitter-langs: Installing grammar bundle v0.12.79 (was vnil) Contacting host: github.com:443 [tree-sitter-langs] Running (tar -xvzf tree-sitter-grammars.x86_64-unknown-linux-gnu.v0.12.79.tar.gz) in /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/bin/ In tree-sitter-langs--hl-default-patterns: tree-sitter-langs.el:245:28: Warning: ‘ignore-error’ condition argument should not be quoted: 'file-missing Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/agda... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/arduino... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/asm... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/astro... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/bash... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/bibtex... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/c... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/clojure... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/cmake... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/cpp... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/css... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/csv... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/dart... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/dockerfile... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/elixir... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/elm... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/fish... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gdscript... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/git-rebase... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitattributes... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitcommit... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitignore... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/glsl... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/go... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/groovy... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/haskell... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/hcl... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/heex... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/hlsl... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/html... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/jai... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/java... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/javascript... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/json... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/kotlin... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/lua... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/make... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/matlab... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/mermaid... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ocaml... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ocaml-interface... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/pascal... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/pgn... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/php... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/psv... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/python... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/r... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ruby... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/rust... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/solidity... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/sql... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tcl... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tsv... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tsx... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/twig... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/typescript... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/typst... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/xml... Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/yaml... Done (Total of 2 files compiled, 1 skipped) BUNDLE-VERSION ada.so agda.so arduino.so asm.so astro.so bash.so beancount.so bibtex.so c-sharp.so c.so clojure.so cmake.so commonlisp.so cpp.so css.so csv.so d.so dart.so dockerfile.so dtd.so elisp.so elixir.so elm.so erlang.so fennel.so fish.so fluent.so fortran.so gdscript.so git-rebase.so gitattributes.so gitcommit.so gitignore.so gleam.so glsl.so go.so groovy.so haskell.so hcl.so heex.so hlsl.so html.so jai.so janet-simple.so java.so javascript.so jsdoc.so json.so jsonnet.so julia.so kotlin.so latex.so lua.so make.so markdown.so matlab.so mermaid.so meson.so nix.so noir.so ocaml-interface.so ocaml.so org.so pascal.so perl.so pgn.so php.so prisma.so python.so r.so rst.so ruby.so rust.so scala.so scheme.so smithy.so solidity.so sql.so swift.so tcl.so toml.so tsx.so twig.so typescript.so typst.so verilog.so vhdl.so xml.so yaml.so zig.so " Packages: "straight" n/a fix/compile 058bcdd 2023-11-17 "org-elpa" n/a n/a "melpa" n/a master d01c726f 2023-11-12 "gnu-elpa-mirror" n/a master 3e9da80 2023-11-08 "nongnu-elpa" n/a main e9d06c3 2023-11-07 "el-get" melpa master a5ba5852 2023-10-20 "emacsmirror-mirror" n/a master 5cbbb0a 2023-11-13 "tree-sitter" melpa release 3cfab8a 2022-02-13 "tsc" melpa release 3cfab8a 2022-02-13 "tree-sitter-langs" melpa release 3a3ad05 2023-11-15 ```
raxod502 commented 11 months ago

Excellent, thank you so much! I was not looking forward to that refactor.