Alien-Luarocks supporting several lua versions

[AitorATuin]

Does Alien or the luarocks Alien wrapper support different versions of lua at the moment? I want to support the three latest version (5.1, 5.2 and 5.3) and I was wondering what's the best approach to achieve it, i have in mind some alternatives:

1. Let Alien-Luarocks to use the binary luarocks that is currently linked and support an utility in the Lua package/Recipe to switch the lua version somehow. We can also have different recipes for older versions of lua (Lua1, Lua2) and let Lua to point to the latest version (5.3 now) (we will need the same for the Luarocks package/Recipe obviously)
2. Add support into the Alien-Luarocks wrapper to specify the version/binary of lua we want to use.

if the versioning is not supported which option makes more sense for you?

[hishamhm]

Great question, we didn't dive into multi-versioning for Alien components yet.

As it stands, to work with multiple Lua/LuaRocks versions, one would have to SymlinkProgram both Lua and LuaRocks to switch versions each time. Clearly not ideal. (I never attacked this problem myself because I'm usually running a development copy of LuaRocks in my $HOME, but that's not of course the "right solution").

The LuaRocks approach for multi-versioning is that it installs itself as both luarocks and luarocks-5.x.

Here's an idea: what if Alien runs lua -e 'print(_VERSION)' to get the current Lua version and then calls luarocks-5.x accordingly? Switching Lua versions would require only calling SymlinkProgram for Lua only.

It's still not perfect because one would have to install LuaRocks three times, once for each Lua version (which leads us to the next question, how to install the same version of LuaRocks three times on GoboLinux cleanly... a klugde would be something like /Programs/LuaRocks/2.3.0_5.3/)

[AitorATuin]

Here's an idea: what if Alien runs lua -e 'print(_VERSION)' to get the current Lua version and then calls luarocks-5.x accordingly? Switching Lua versions would require only calling SymlinkProgram for Lua only.

Yeah, sounds fine but I think that calling SymlinkProgram just to change the version of lua at user scope is a bit overkill. That's fine if you want to make the default lua at system level but for users i was thinking more in adding a small utility (Environment function) to change between different versions of lua (aliases for example).

It's still not perfect because one would have to install LuaRocks three times, once for each Lua version (which leads us to the next question, how to install the same version of LuaRocks three times on GoboLinux cleanly... a klugde would be something like /Programs/LuaRocks/2.3.0_5.3/)

What about including luarocks with the lua recipe? for new versions of luarocks we can just release a new revision of the recipe. I think that the most widely use of lua is together with luarocks so I think that will work for most of the people. Python3 now includes pip by default, we could do the same thing with Python2 and with lua.

We can also provide a standalone Lua and Luarocks recipe for those who need something special (they will need to deal then with the conflicts i guess).

[lucasvr]

While revisiting open issues I noticed this one and thought about giving my 2 cents. How about spawning LuaRocks through Runner to have the right version of Lua automatically configured?

[hishamhm]

What about including luarocks with the lua recipe?

@AitorATuin This is what Homebrew does on macOS. It's definitely an option, but bundling things under the same prefix is not too Gobo-like.

While revisiting open issues I noticed this one and thought about giving my 2 cents. How about spawning LuaRocks through Runner to have the right version of Lua automatically configured?

@lucasvr LuaRocks has no problem finding the Lua version it was configured with. The trouble is having up to three copies of the same version of LuaRocks to handle different Lua versions.

The LuaRocks codebase works the same for Lua 5.1-5.3, the only difference is that its modules need to be installed in share/lua/5.x/ for the correct value of 5.x, plus one config file that's generated at install time (but which could be overwritten by Settings/luarocks/config-5.x.lua).

No solution I can think of right now makes me fully satisfied. I might end up going with Aitor's suggestion for simplicity's sake.

[lucasvr]

Ahhhh, thanks for the clarification @hishamhm.

[AitorATuin]

@hishamhm I know it's not the better solution but I think it's the simplest solution, at least for the moment, we can think about a better solution in the future if we can.

[AitorATuin]

Actually we have the same problem with python. I think that over and above the specific lua problem we should take a decision regarding on how to deal with multiple versions for the same language (in the languages for which this make sense of course). I was thinking in adding a parameter where you can specify a version (similar to what virtualenv does) but while that fixes the problem when using the cli it still does not solve the problem when using Alien as a dependency which is the most important case (all deps will be PIP:package for example). Another approach is to create a new namespace for different versions (PIP3, since python2 is default in gobolinux) so we can explicitly tell that a package requires PIP3:something. I prefer this one.

[AitorATuin]

A PR implementing a new PIP3 namespace. https://github.com/gobolinux/Scripts/pull/23

[mwh]

PIP3 is probably necessary anyway, but the broader issue is different minor versions used in parallel, which can't feasibly be dealt with with new prefixes every time.

Python 3.5 vs 3.6 or Lua 5.2 vs 5.3 can potentially be handled by detecting the corresponding language version in use from the dependencies. Bundling the package manager with it is one way to manage that, but it could be built in as a feature separately too. There may be odd corner cases where versions of more than one package matter and not just the language itself, which bundling can't deal with (for example, alien packages may vary according to both the language and the VM backing it).

There's a fundamental problem anyway when you upgrade and change the default system version too, when all of these packages instantly disappear. I don't know how to solve that. Declaring a single specific version of each is supported and ignoring the rest is one option, but that just means you're stuck forever.

[AitorATuin]

PIP3 is probably necessary anyway, but the broader issue is different minor versions used in parallel, which can't feasibly be dealt with with new prefixes every time.

Well, that should be something the concrete package manager should deal with. In lua it's easier, the versioning is much better imho, lua 5.2 is different from lua 5.3 and you have two different binaries. In python there is not nice way to difference 3.5 from 3.6, both are python3 but seems to be a decision in the language itself so an upgrade in the python version is a big change and the user should know it.

There may be odd corner cases where versions of more than one package matter and not just the language itself, which bundling can't deal with (for example, alien packages may vary according to both the language and the VM backing it).

I don't see this could happen in lua (which is the case we were discussing regarding the package manager bundled) but maybe I'm missing something. Can you provide an example, please?

There's a fundamental problem anyway when you upgrade and change the default system version too, when all of these packages instantly disappear.

Yeah, not sure if we should even solve this in Alien, it's a problem related to the language anyway. In my option the only use for Alien is to deal nicely with dependencies in Recipes, so it should be enough to specify the dependencies there making sure that the correct version of the interpreter is used by Compile.

[mwh]

you have two different binaries. In python there is not nice way to difference 3.5 from 3.6, both are python3

They're also python3.5 and python3.6 (and more if you turn on autoVersionExecutables). Python scripts could all be patched by Compile to run a particular version, it's just annoying to do so.

Can you provide an example, please?

I can imagine a JVM language having libraries that depended on both the JVM version and the language version, or something building on Parrot or Mono. None of those are supported at the moment anyway, so it's a future issue only. For Lua and Python it's not a problem. A long-term solution shouldn't exclude the possibility though.

[hishamhm]

Here's a sketch of a general solution in a way that may cover the cases that @AitorATuin and @mwh brought up:

• Each package manager has its own unversioned namespace, as we do today: LuaRocks, CPAN, PIP. These are used in Dependencies (LuaRocks:luafilesystem >= 1.5.0), in the command-line tool (Alien-PIP) and the /S/A dir (/System/Aliens/CPAN).
• Under each alien's /S/A dir, they define a versioning directory, at its own discretion. It refers to the specifics of the language. We could use the x notation to make it clear this refers to version families and not specific versions. Possible examples: /System/Aliens/LuaRocks/5.1.x, /System/Aliens/PIP/3.x, /System/Aliens/FooLanguage/foo-1.3.x-jdk1.8.x
• The command-line Alien tool will optionally receive a Dependencies file as an argument and use it to determine which versioned subtree to use, as in Alien -d /path/to/Dependencies --install PIP:burn

For example, given a Dependencies file that says:

Lua >= 5.3
LuaRocks:luafilesystem >= 1.5

then Alien-LuaRocks should be able to infer that /System/Aliens/LuaRocks/5.3.x is the tree to be used. The resolution process for this can be as simple or as complicated as necessary for the language (to help implementing the resolver, we can use tools like FindDependencies -d Lua /path/to/Dependencies).

• For direct command-line use, the Alien tool could simply fail if there are multiple trees under one namespace:

] Alien --install PIP:numpy
Alien-PIP: There are multiple trees for namespace PIP. Please specify one of these with flag --tree:
2.x
3.x
] Alien --install PIP:numpy --tree 2.x
Alien-PIP: Installing numpy to /System/Aliens/PIP/2.x/
...

• If there is only one tree under the namespace, use it directly as the default.

] Alien --install LuaRocks:luafilesystem
Alien-LuaRocks: Installing luafilesystem to /System/Aliens/LuaRocks/5.3.x/

• A tricky situation arises when there are no trees under the namespace yet. I suppose it's up to the Alien driver to fail, give the user sensible --tree options for the language, or to detect which language version is currently installed.

] Alien --install LuaRocks:luafilesystem
Alien-LuaRocks: Creating /System/Aliens/LuaRocks
Alien-LuaRocks: Detected that Lua version is 5.3.4
Alien-LuaRocks: Installing luafilesystem to /System/Aliens/LuaRocks/5.3.x/

---

Does this all sound sensible? Or is it too over-engineered? Or does it shift the burden too much on the Alien drivers?

[AitorATuin]

Does this all sound sensible? Or is it too over-engineered? Or does it shift the burden too much on the Alien drivers?

In general it sounds good enough for me with some comments / suggestions:

• I don't like the idea of having -d argument in Alien, I think that's coupling Alien with the dependencies system too much, I would rather prefer that a party component (GetDependencies or whatever in Compile) taking care of resolving the --tree version for the language (maybe giving the binary to use for example to avoid to duplicate code in 2 different parts).

A tricky situation arises when there are no trees under the namespace yet. I suppose it's up to the Alien driver to fail, give the user sensible --tree options for the language, or to detect which language version is currently installed.

Yeah, in that case it should expect explicitly a tree parameter i guess.

[hishamhm]

I don't like the idea of having -d argument in Alien, I think that's coupling Alien with the dependencies system too much, I would rather prefer that a party component (GetDependencies or whatever in Compile) taking care of resolving the --tree version for the language (maybe giving the binary to use for example to avoid to duplicate code in 2 different parts).

I agree that decoupling things is preferrable, but moving the context resolution into a component in Compile would mean that this component needs to know rules for all supported languages.

My criterion is that the logic for knowing how to resolve a set of dependencies into a tree is language-specific, and therefore belongs in each specific Alien. The knowledge about the dependencies format should be separate, that's why I suggested using the FindDependencies program. This way, Alien doesn't even need to open the dependencies file or how to solve dependencies. It can just query for specific programs it cares about.

The tree resolution is very language-specifiic. Typically it would query just the language, as in the Lua example, or as in @mwh's example, it could check for two programs; FooLanguage and JVM. Also, it is a language-specific thing to determine if we care about the major version only (Python 3.x to launch pip3) or the middle version (Lua 5.1.x) or something else entirely.

I'll cook up a PR to see how the implementation looks in practice. That way we'll be able to better judge if things got too coupled or not.

Yeah, in that case it should expect explicitly a tree parameter i guess.

That is sensible and not too magical.

[hishamhm]

Uploaded the PR: https://github.com/gobolinux/Scripts/pull/24