Consider using python-nix

Mic92 / nixpkgs-review

Review pull-requests on https://github.com/NixOS/nixpkgs

MIT License

350 stars 59 forks source link

Consider using python-nix #358

Closed figsoda closed 9 months ago

figsoda commented 10 months ago

It still looks to be a WIP, but maybe this is something we can consider in the future to try to improve performance: https://github.com/tweag/python-nix

Mic92 commented 10 months ago

I am not quite sure how this would save performance though? I wouldn't expect a well-optimized json or xml parser for a language will be the bottleneck. I wouldn't be surprised if this was even faster in some circumstances because the core developers might use some not so known tricks to create objects faster. Since I maintained python-nix before I would be more concerned that we would had to fix python-nix ourself at some point i.e. when the people that currently work on python-nix at tweag left to make the build work with the latest nix version. So far it's nice that nixpkgs-review probably just works with every nix version that is in the nixpkgs tree.

SuperSandro2000 commented 9 months ago

How likely would it be to cause unexpected outcomes eg when python shells out to some program?

Mic92 commented 9 months ago

Yet another issue: Memory leaks. Once we would have evaluated nixpkgs, we would likely still have all the memory in our nixpkgs-review since nix/nixpkgs is not good at returning memory. This would mean, we would still have to do the eval in a forked process.

Mic92 commented 9 months ago

How likely would it be to cause unexpected outcomes eg when python shells out to some program?

I do not quite understand that question?

SuperSandro2000 commented 9 months ago

I do not quite understand that question?

I confused the link with https://github.com/GuillaumeDesforges/fix-python and didn't click on it.

Mic92 commented 9 months ago

I now watched the talk about python-nix: https://media.ccc.de/v/nixcon-2023-36407-using-nix-from-python-using-python-from-nix#t=634

And my concerns are still the following:

we now have a gc that might leak memory in nixpkgs-review
nix-env internally evaluates nixpkgs in a special way that's not easy to re-implement (i.e. catching throws and broken flags)
also the C-API might become stable in a few years time, nix developers might still break internal stuff (I had fun with nix-eval-jobs in this area)
It's not clear to me this brings any performance advantage. Nix-env might be faster than having to switch back and force between nix and python. But of course I am looking forward to be convinced of the opposite by some proof-of-concept code.

Looks like a nice project though for other projects.

figsoda commented 9 months ago

seems like it might be too much effort without enough improvements