Freezes with 100% CPU when trying to open a specific Nix file

[max-privatevoid]

Summary

Probably related to https://github.com/helix-editor/helix/issues/2997

Reproduction Steps

• Create a file with the content listed below
• In another terminal, open the file in Helix
• Observe it getting stuck before launching
• Edit the file: Remove a single character from a string
• Try opening the file with Helix again

• Observe it now working even though there should be no change to the AST

  { config, lib, pkgs, ... }:
  with lib;
  let
  # most of the things here should probably be incorporated into a module
  
  # TODO: make a proper type for env vars or pathspecs
  coerceToEnv = val: let
  parsed = match "^\\$([a-zA-Z0-9_]*)(/(.*))?$" val;
  key = elemAt parsed 0;
  append = elemAt parsed 2;
  in if isString val && parsed != null then
    if append != null then
      concat (env key) (concat "/" (coerceToEnv append))
    else
      env key
  else if isAttrs val && val ? _sloth then
    val._sloth
  else
    val;
  
  instanceId = { type = "instanceId"; };
  concat = a: b: { type = "concat"; inherit a b; };
  env = key: { type = "env"; inherit key; };
  
  splitPath = path: if isList path then {
  a = elemAt path 0;
  b = elemAt path 1;
  } else {
  a = path;
  b = path;
  };
  
  bind' = arg: path: let
  split = splitPath path;
  coerced = mapAttrs (_: coerceToEnv) split;
  in [ arg coerced.a coerced.b ];
  bind = bind' "--bind-try";
  bindRo = bind' "--ro-bind-try";
  bindDev = bind' "--dev-bind-try";
  setEnv = key: val: [ "--setenv" key val ];
  mountTmpfs = path: [ "--tmpfs" path ];
  
  bindPaths = map bind config.bubblewrap.bind.rw;
  bindRoPaths = map bindRo config.bubblewrap.bind.ro;
  bindDevPaths = map bindDev config.bubblewrap.bind.dev;
  envVars = mapAttrsToList setEnv config.bubblewrap.env;
  tmpfs = map mountTmpfs config.bubblewrap.tmpfs;
  
  app = config.app.package;
  info = pkgs.closureInfo { rootPaths = app; };
  launcher = pkgs.callPackage ../launcher {};
  dbusOutsidePath = concat (env "XDG_RUNTIME_DIR") (concat "/nixpak-bus-" instanceId);
  
  bwrapArgs = flatten [
  "--unshare-all"
  bindPaths
  bindRoPaths
  envVars
  tmpfs
  
  (optionals config.bubblewrap.network "--share-net")
  (optionals config.bubblewrap.apivfs.dev ["--dev" "/dev"])
  (optionals config.bubblewrap.apivfs.proc ["--proc" "/proc"])
  
  bindDevPaths
  
  (optionals config.dbus.enable [
    (bind [ dbusOutsidePath "$XDG_RUNTIME_DIR/nixpak-bus" ])
    "--setenv" "DBUS_SESSION_BUS_ADDRESS"
    (concat "unix:path=" (coerceToEnv "$XDG_RUNTIME_DIR/nixpak-bus"))
  ])
  
  [ "--ro-bind" config.flatpak.infoFile "/.flatpak-info" ]
  
  # TODO: use closureInfo instead
  [ (bindRo "/nix/store") "${app}/${config.app.binPath}" ]
  ];
  dbusProxyArgs = [ (env "DBUS_SESSION_BUS_ADDRESS") dbusOutsidePath ] ++ config.dbus.args ++ [ "--filter" ];
  
  bwrapArgsJson = pkgs.writeText "bwrap-args.json" (builtins.toJSON bwrapArgs);
  dbusProxyArgsJson = pkgs.writeText "xdg-dbus-proxy-args.json" (builtins.toJSON dbusProxyArgs);
  
  mainProgram = builtins.baseNameOf config.app.binPath;
  
  envOverrides = pkgs.runCommand "nixpak-overrides-${app.name}" {} ''
  mkdir $out
  cd ${app}
  grep -Rl ${app}/${config.app.binPath} | xargs -r -I {} cp -r --parents {} $out || true
  find $out -type f | while read line; do
    substituteInPlace $line --replace ${app}/${config.app.binPath} ${config.script}/${config.app.binPath}
  done
  
  desktopFileRel=share/applications/${config.flatpak.appId}.desktop
  if [[ -e $desktopFileRel ]]; then
    echo $out
    echo -e 'nX-Flatpak=${config.flatpak.appId}' >> $out/$desktopFileRel
  fi
  '';
  
  # This is required because the Portal service reads /proc/$pid/root/.flatpak-info
  # from the calling PID, when dbus-proxy is in use, this PID is the dbus-proxy process
  in {
  }

Helix log

No response

Platform

Linux

Terminal Emulator

Tilix 1.9.5

Helix Version

helix 22.08.1

[the-mikedavis]

This is probably from the tree-sitter-bash injections within strings. Tree-sitter-bash has an external scanner that looks like other problematic scanners we've seen (https://github.com/helix-editor/helix/issues/3504, https://github.com/helix-editor/helix/issues/3376, https://github.com/helix-editor/helix/issues/2979), and looking at the source now, it looks like there is the usual workaround in place which prevents scanning if in error recovery: https://github.com/tree-sitter/tree-sitter-bash/blob/77cf8a7cab8904baf1a721762e012644ac1d4c7b/src/scanner.cc#L171-L191. We can probably fix this by updating tree-sitter-bash

EDIT: we're already on a version of tree-sitter-bash with that scanner fix. It looks like updating tree-sitter-bash isn't enough to fix this.

[max-privatevoid]

The "fix" also works in strings that are used as derivation names, or even when renaming variables. Even deleting things from the comment near the end makes the file openable. I don't think it's related to bash TS.

[the-mikedavis]

Looking at a flamegraph for this, it looks like we're stuck on tree-sitter's ts_parser__recover function which is the error recovery routine that caused problems in other languages. On the revision that we use (and master), tree-sitter-nix's scanner is mitigating the error recovery problem: https://github.com/cstrahan/tree-sitter-nix/blob/1b69cf1fa92366eefbe6863c184e5d2ece5f187d/src/scanner.c#L180-L190

If I comment out this block in nix injections then the problem goes away: https://github.com/helix-editor/helix/blob/590a628460c8fa8a4e101541d67289fa2fac95b5/runtime/queries/nix/injections.scm#L21-L28

The block I was pointing to in tree-sitter-bash isn't preventing scanning in error recovery, it's just part of the scanner 😅. ~Adding the usual change to the scanner that checks if we're in error recovery and returns false seems to fix this. I'll fork tree-sitter-bash and make a PR.~ Actually that doesn't fix it either. Something else must be keeping us in recovery mode.

Very strange that deleting from the comment removes the problem 🤔

[the-mikedavis]

It looks like this was a bug in tree-sitter itself and was fixed in https://github.com/tree-sitter/tree-sitter/pull/1952 (also see https://github.com/cstrahan/tree-sitter-nix/issues/34). Running from that commit fixes this and the other linked issues for me. Once tree-sitter cuts a new version we should be able to upgrade and close this.

(If you're building from source and want to use that change before it is published and updated within Helix, you can add a patch to the Cargo.toml in the Helix repository root:)

[patch.crates-io]
tree-sitter = { git = "https://github.com/tree-sitter/tree-sitter", rev = "e021d6e9799903312dfa3a1ed39e160008fe52ac" }

[archseer]

We could switch to a git dependency here too since tree-sitter hasn't been updated in 6 months. neovim did so too https://github.com/neovim/neovim/pull/21858