Closed onekey-sec-bot closed 7 months ago
So Nix restored runtime dependency checking for python packages (see https://discourse.nixos.org/t/breaking-changes-announcement-for-unstable/17574/40?page=2).
We're hitting a wall with LIEF for two reasons:
pythonRelaxDepsHook
because LIEF project structure is dumb (remember the postInstall
and postBuild
fixups), which makes python-relax-deps-hook
fail:lief> [739/739] Linking CXX shared library /build/source/api/python/build/lief.sondroid/pyVersion.cpp.oKKted.c.oe.cpp.o
lief> Copying /build/source/api/python/build/lief.so into build/lib.linux-x86_64-cpython-311/lief/_lief.cpython-311-x86_64-linux-gnu.so
lief> copying build/lib.linux-x86_64-cpython-311/lief/_lief.cpython-311-x86_64-linux-gnu.so -> lief
lief> running build_scripts
lief> creating build/scripts-3.11
lief> copying and adjusting examples/elf_reader.py -> build/scripts-3.11
lief> copying and adjusting examples/pe_reader.py -> build/scripts-3.11
lief> copying and adjusting examples/macho_reader.py -> build/scripts-3.11
lief> changing mode of build/scripts-3.11/elf_reader.py from 644 to 755
lief> changing mode of build/scripts-3.11/pe_reader.py from 644 to 755
lief> changing mode of build/scripts-3.11/macho_reader.py from 644 to 755
lief> /build/source/build
lief> /nix/store/7wwxxw7d04vfx1gjr0jyw0958zib890j-python-relax-deps-hook/nix-support/setup-hook: line 78: pushd: dist: No such file or directory
lief> /nix/store/v5irq7wvkr7kih0hhnch5nnv2dcq8c4f-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/1z7nxz8b8g59zbhq3pcn6g8fhr78vbyk-lief-0.13.2.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/0ny787y0sv65w49inlzj9pkv7z70bsgk-unblob-tests-24.1.4.drv' failed to build
Since I was working on upgrading some packages, I decided to take care of all of them. Pyright is now reporting more issues, but these are relevant and should be either handled or ignored depending on importance.
We were already talking about ditching LIEF as a dependency internally, this package issue might be a good excuse to kickstart this change.
So Nix restored runtime dependency checking for python packages (see https://discourse.nixos.org/t/breaking-changes-announcement-for-unstable/17574/40?page=2).
We're hitting a wall with LIEF for two reasons:
* we can't upgrade LIEF because they do not provide wheels for python3.12 for versions after 0.12.3 * we cannot use `pythonRelaxDepsHook` because LIEF project structure is dumb (remember the `postInstall` and `postBuild` fixups), which makes `python-relax-deps-hook` fail:
I don't understand this. We need to relax in unblob:
diff --git a/nix/unblob/default.nix b/nix/unblob/default.nix
index 0be6c88..b2c0c0d 100644
--- a/nix/unblob/default.nix
+++ b/nix/unblob/default.nix
@@ -108,6 +108,7 @@ let
pythonRelaxDeps = [
"dissect.cstruct"
+ "lief"
"structlog"
];
makeWrapperArgs = [
This did the trick for me.
Regarding pyright errors, I think the type hint in pyperscan
's match callback is overzealous.
I could get rid of it by applying the following change there:
diff --git a/python/pyperscan/_pyperscan.pyi b/python/pyperscan/_pyperscan.pyi
index c5d1e64..142f554 100644
--- a/python/pyperscan/_pyperscan.pyi
+++ b/python/pyperscan/_pyperscan.pyi
@@ -176,12 +176,12 @@ class OnMatch(Protocol, Generic[_TContext_contra]):
"""Callback called on match."""
def __call__(
- self, context: _TContext_contra, tag: Any, start: int, end: Any
+ self, context: _TContext_contra, tag: Any, start: int, end: int, /
) -> Scan:
"""Called when a match happens.
Note:
- Call parameters are passed positonally.
+ Call parameters are passed positionally.
Args:
context: Object passed to [Database.build][pyperscan._pyperscan.Database.build]
I'd suggest ignoring those for now, and it will be fixed in an upcoming version.
Looked into the pyright issues. Some of them were present before but wasn't reported, because they were disabled. In pyright 1.1.339 the default typeCheckingMode
has changed from basic
to the new standard
setting^1. Setting it back to basic
eliminates a new category of issues.
It makes sense to resolve these, but maybe not in a dependency update.
Thanks for the pointers @vlaci ! Will look into it this week-end.
Nitpick: lz4 and lark are updated in the same commit
Split into two separate commits.
We could also update dissect.cstruct to 3.x, as the nix derivation already uses 3.10 without issues and can get rid of one pythonRelaxDeps entry.
Upgrading dissect.cstruct
would force us to drop support for python 3.8. Maybe later.
Regarding pyright errors, I think the type hint in pyperscan's match callback is overzealous. I'd suggest ignoring those for now, and it will be fixed in an upcoming version.
Done in 42cdc5a
In pyright 1.1.339 the default typeCheckingMode has changed from basic to the new standard setting1. Setting it back to basic eliminates a new category of issues.
Set back to basic
in 42cdc5a
Automated changes by the update-flake-lock GitHub Action.
Running GitHub Actions on this PR
GitHub Actions will not run workflows on pull requests which are opened by a GitHub Action.
To run GitHub Actions workflows on this PR, run: