search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.92k stars 13.95k forks source link

python-fx fails at runtime (import stage) #333173

Closed jcaesar closed 2 months ago

jcaesar commented 2 months ago

Describe the bug

python-fx fails at runtime, while doing its initial imports.

Steps To Reproduce

Steps to reproduce the behavior:

  1. echo [1,2,3] | nix run github:NixOS/nixpkgs/f5129fb42b9c262318130a97b47516946da3e7d7#python311Packages.python-fx (hash is of as-of-writing nixpkgs-unstable)

Expected behavior

pyfx tui opens.

"Screenshots"

Traceback (most recent call last):
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/bin/.pyfx-wrapped", line 6, in <module>
    from pyfx.cli import main
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/__init__.py", line 1, in <module>
    from .app import PyfxApp
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/app.py", line 13, in <module>
    from pyfx.model import Model
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/__init__.py", line 9, in <module>
    from .model import Model
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/model.py", line 4, in <module>
    from .autocomplete import autocomplete
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/autocomplete/__init__.py", line 11, in <module>
    from .autocomplete_listener import autocomplete
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/autocomplete/autocomplete_listener.py", line 8, in <module>
    from ..common.jsonpath import JSONPathLexer, JSONPathParser, JSONPathListener
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/common/jsonpath/__init__.py", line 7, in <module>
    from .JSONPathParser import JSONPathParser
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/common/jsonpath/JSONPathParser.py", line 75, in <module>
    class JSONPathParser(Parser):
  File "/nix/store/77hvxda4jrx0cchzkv9g2j58j0l6hrqk-python3.11-python-fx-0.3.1/lib/python3.11/site-packages/pyfx/model/common/jsonpath/JSONPathParser.py", line 78, in JSONPathParser
    atn = ATNDeserializer().deserialize(serializedATN())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/5qwaqv89wbd22cmpibvicg0gi58bkkk3-python3.11-antlr4-python3-runtime-4.13.0/lib/python3.11/site-packages/antlr4/atn/ATNDeserializer.py", line 28, in deserialize
    self.checkVersion()
  File "/nix/store/5qwaqv89wbd22cmpibvicg0gi58bkkk3-python3.11-antlr4-python3-runtime-4.13.0/lib/python3.11/site-packages/antlr4/atn/ATNDeserializer.py", line 50, in checkVersion
    raise Exception("Could not deserialize ATN with version " + str(version) + " (expected " + str(SERIALIZED_VERSION) + ").")
Exception: Could not deserialize ATN with version  (expected 4).

Additional context

Doesn't happen when e.g. installing current python-fx with pip on arch linux.

(I considered packaging fx instead, but only found more breakage there. Anyway, I wouldn't be terribly sad if the answer to this is "Package abandonned, will be removed".)

Notify maintainers

@fabaff

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.43, NixOS, 24.11 (Vicuna), j_5925a98_20240805234416`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - nixpkgs: `/nix/store/li7gsayjjdp78xm0x7crzcqnafz8n735-source`

Add a :+1: reaction to issues you find important.

fabaff commented 2 months ago

I have not used the TUI as python-fx so far only served as dependency for qiling.

Looks like https://github.com/antlr/antlr4/issues/4041

jcaesar commented 2 months ago

I poked around and I don't think that's the same issue at all. (It's the same cause, an old generated file is lying around, but it's a different old generated file.)

The python-fx prePatch is pretty fishy, it first removes the antlr-related files and then invokes antlr.

Fixing that runs into a few type errors, which can be fixed:

diff --git a/pkgs/development/python-modules/python-fx/default.nix b/pkgs/development/python-modules/python-fx/default.nix
index b3a15ff913ca..2200d01d9e9a 100644
--- a/pkgs/development/python-modules/python-fx/default.nix
+++ b/pkgs/development/python-modules/python-fx/default.nix
@@ -43,8 +43,12 @@ buildPythonPackage rec {
   };

   postPatch = ''
-    rm src/pyfx/model/common/jsonpath/*.{g4,interp,tokens}
-    antlr -Dlanguage=Python3 -visitor -o src/pyfx/model/common/jsonpath/ *.g4
+    antlr -Dlanguage=Python3 -visitor src/pyfx/model/common/jsonpath/*.g4
+    substituteInPlace src/pyfx/model/autocomplete/autocomplete_listener.py --replace-fail \
+      "from ..common.jsonpath import JSONPathLexer, JSONPathParser, JSONPathListener" \
+      $'from ..common.jsonpath.JSONPathLexer import JSONPathLexer\nfrom ..common.jsonpath.JSONPathParser import JSONPathParser\nfrom ..common.jsonpath.JSONPathListener import JSONPathListener\n'
+    substituteInPlace src/pyfx/view/common/selectable_text.py --replace-warn '@overrides' ""
+    substituteInPlace src/pyfx/view/common/frame.py --replace-warn '@overrides' ""
   '';

   pythonRelaxDeps = true;
@@ -81,13 +85,9 @@ buildPythonPackage rec {
     parameterized
   ];

-  # antlr4 issue prevents us from running the tests
-  # https://github.com/antlr/antlr4/issues/4041
-  doCheck = false;
-
-  # pythonImportsCheck = [
-  #   "pyfx"
-  # ];
+  pythonImportsCheck = [
+    "pyfx"
+  ];

   meta = with lib; {
     description = "Module to view JSON in a TUI";

Now you'd think those type errors are due to the new antler version, but that's only partially true? Downgrading still requires some patching.

diff --git a/pkgs/development/python-modules/python-fx/default.nix b/pkgs/development/python-modules/python-fx/default.nix
index b3a15ff913ca..fe4360bb8f53 100644
--- a/pkgs/development/python-modules/python-fx/default.nix
+++ b/pkgs/development/python-modules/python-fx/default.nix
@@ -18,7 +18,7 @@
   pyperclip,
   pytestCheckHook,
   pythonOlder,
-  antlr4,
+  antlr4_8,
   pyyaml,
   setuptools,
   six,
@@ -26,9 +26,16 @@
   parameterized,
   wcwidth,
   yamale,
-}:
+}: let

-buildPythonPackage rec {
+  antlr4-python3-runtime' = antlr4-python3-runtime.override { antlr4 = antlr4_8; };
+  antlr4-python3-runtime'' = antlr4-python3-runtime'.overrideAttrs {
+    postPatch = "";
+    prePatch = "mkdir tests; touch tests/run.py;";
+    doCheck = false;
+  };
+
+in buildPythonPackage rec {
   pname = "python-fx";
   version = "0.3.1";
   format = "setuptools";
@@ -43,19 +50,19 @@ buildPythonPackage rec {
   };

   postPatch = ''
-    rm src/pyfx/model/common/jsonpath/*.{g4,interp,tokens}
-    antlr -Dlanguage=Python3 -visitor -o src/pyfx/model/common/jsonpath/ *.g4
+    substituteInPlace src/pyfx/view/common/selectable_text.py --replace-warn '@overrides' ""
+    substituteInPlace src/pyfx/view/common/frame.py --replace-warn '@overrides' ""
   '';

   pythonRelaxDeps = true;

   nativeBuildInputs = [
-    antlr4
+    antlr4_8
     setuptools
   ];

   propagatedBuildInputs = [
-    antlr4-python3-runtime
+    antlr4-python3-runtime''
     asciimatics
     click
     dacite

If I have time I'll submit a PR fixing this, but just patching out the type checks from the @overrides feels wrong, so I might consult with upstream first.