search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.15k stars 14.17k forks source link

Meta issue about lua ecosystem progress #56398

Open teto opened 5 years ago

teto commented 5 years ago

Issue description

Meta issue about changes to how lua is handled in nixpkgs (obsoletes https://github.com/NixOS/nixpkgs/pull/33903).

Possibie improvements

Shados commented 5 years ago

Some nice things that would require upstream luarocks changes (from a brief preliminary investigation):

None of these are super necessary, but all would be potentially useful.

These would require changes to luarocks to make it capable of searching multiple rocks trees for the relevant operations. It already does this for luarocks make/luarocks install if you use --deps-mode=all (and explicitly specify the locations of the rocks trees), but it does not do this for other operations AFAICT.

They would also need changes to add a way to have luarocks find these rocks trees to begin with.

doronbehar commented 5 years ago

I noticed that some packages which use Lua, (e.g AwesomeWM and Neovim) use an old version of Lua (5.2 && 5.1 - respectively). As a beginner Nixos user I'm not sure if it is easily override-able but I think it'd be worth mentioning that nixpkgs should use 5.3 as the default version of Lua for these packages' inputs.

teto commented 5 years ago

why is that ? lua5.2 looks standard; 5.3 qnd 5.4 seem to have breaking changes. As for neovim they explicitly require 5.1

doronbehar commented 5 years ago

As for Neovim, you are correct but as for AwesomeWM, anything higher than 5.1 is fine. They both support LuaJIT as well.

I'm not sure what should be dictated as nixpkgs' policy but my opinion is that Lua 5.3 should be treated as the default as it's considered the latest version upstream. You are right that it has breaking changes and that's why Neovim Neovim explicitly requires 5.1 but this is an upstream issue, not nixpkgs'.

Additionally, I think that in the specific case of Neovim, since LuaJIT is supported as well ~I would switch to it for this package specifically, it is certainly seems superior then Lua 5.1~, it's nice that it's build uses LuaJIT by default. but I think we should just rename the luaPackages name space to refer to lua53Packages. What do you think?

EDIT: The default Neovim build uses LuaJIT - it surprised me since it's inputs are lua and not luajit.

teto commented 5 years ago

I suspect moving to lua53 would break many things. lua versions are different between eachother.

I've added the support in nix for luarocks mirrors so don't be surprised if luarocks-nix generates some strange urls. I've also moved the luarocks-nix repository to nix-community. I had a qucik look at supporting luarocks test but seems like it deals with installed packages ? not too complex but just enough for me to skip it. I wanted to take the opportunity to thank @shados who improved the infra a lot ! I was able to tick many boxes thanks to his work !

teto commented 5 years ago

@grwlf Many packages are marked broken in pkgs/applications/science/machine-learning/torch/torch-distro.nix . What do you think about replacing them with the current lua infrastructure ?
it unifies the interface and may unbreak some modules. I can guide you if you have any questions.

lblasc commented 5 years ago

@teto how about removing torch-distro completely? Sadly upstream didn't have any activity since 2017.

stites commented 5 years ago

lua-torch is no longer maintained (the codebase was migrated to pytorch) and communities have been marked as non-existent. I think removing torch is valid. Doing so would also close #71888.

See https://github.com/torch/torch7#development-status and final commit https://github.com/torch/torch7/commit/dde9e56fb61eee040d7f3dba2331c6d6c095aee8

sergei-mironov commented 5 years ago

@teto Hi. I think it is OK to remove old torch files. I may be listed as a maintainer of some of them, but actually I don't plan to maintain anything because my focus shifted towards TF/pytorch nowdays..

teto commented 5 years ago

ok thanks for the update. I guess we can remove it then. I was hoping someone could move some of the packages to the current lua infra to salvage some of the work (like finding the buildInputs). But that might not be worth the effort. I don't plan to do anything as I have already other issues to address but I can advise anyone interested in cleaning up.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

teto commented 3 years ago

@Mic92 @timokau with the rise of a the lua ecosystem with neovim 0.5, I am looking at ways to consolidate dependency management in that ecosystem (with https://github.com/NixOS/nixpkgs/pull/131499 for instance). The "maintainers/scripts/update-luarocks-packages: shellscript is not up to the task anymore so I was considerng adaptin pkgs/misc/vim-plugins/update.py and pluginupdate.py to generate pkgs/development/lua-modules/generated-packages.nix as well. Any objection ?

timokau commented 3 years ago

Sorry for the late response. It has been a while since I touched the vim plugin infrastructure and I don't know much about lua. I think you're more up to date on both of these things than me, so I defer to you :)

teto commented 3 years ago

status update: I've recently merged :

While investigating how to reconcile vim plugins and lua packages, I found out this luarocks config element

-- to create a flat hierarchy
lua_modules_path = ""

that creates a flat hierarchy and could reduce the number of nested folders in our nix packages.

teto commented 3 years ago

big changes in https://github.com/NixOS/nixpkgs/pull/140801 to remove the use of with self; in generated-packages.nix, like is done in haskell.

EDIT: fixed link

doronbehar commented 3 years ago

big changes in #56398 to remove the use of with self; in generated-packages.nix, like is done in haskell.

@teto #56398 is this issue :).

teto commented 2 years ago

https://github.com/NixOS/nixpkgs/pull/166162 fixes the issue we had on luv that was blocking luarocks to 3.2. We can now bump it to 3.8. I had a nice PR that can automatically add neovim plugin dependencies https://github.com/NixOS/nixpkgs/pull/160484 (need a rebase) which could serve as a nice poc for https://github.com/nvim-lua/nvim-package-specification. To make it mergeable I would need to make it possible to generate flat lua installs.

One problem is that handling of LUA_PATH across nixpkgs is very heterogeneous, and so it makes harder to make some global changes. I think it's heterogeneous partly because lua utilities are broken. For instance the LUA_PATH generation hook removes the defaults from the lua interpreter (";;"). Many modules hardcode the different lua paths instead of using e.g. wrapLuaProgram, luaPathList from pkgs/development/lua-modules/lib.nix . I believe wrapLuaProgram is broken so let's first fix this and add some tests. I would welcome some help as I wont likely be able to do it anytime.

teto commented 2 years ago

I've merged https://github.com/NixOS/nixpkgs/pull/195869 that adds a testing infra for the interpreter, which I hope will help prevent naive regressions in the future. I've also merged https://github.com/NixOS/nixpkgs/pull/195732 which should hopefully allow to have the best of both worlds for neovim plugins: automatic dependency management from the rockspec + uptodate plugins via the vim plugin updater.

Next step will be to track every per-package custom nix code for lua and replace it with code from the luaLib so that I can configure flat installs in luaLib and it will affect all of nixpkgs.

tomodachi94 commented 3 weeks ago

Here's a GitHub Search query that can perhaps help with that: https://github.com/search?q=repo%3ANixOS%2Fnixpkgs+lang%3Anix+lua+NOT+wrapLuaPrograms+NOT+withPackages&type=code

There are definitely some false positives, but I'm also seeing some other packages could use some love.

tomodachi94 commented 3 weeks ago

More packages that could use TLC (pure Lua packages, most likely compatible with the neo-builders):