BEAM packages support umbrella tracking issue (Erlang, Elixir, LFE, etc.)

gleber commented 8 years ago

I am starting this issue to track progress of the effort started in #11651. I will be updating this issue with further developments and my implementation plans. If someone wants to jump in and work on it, please let me know (here or privately) to avoid work duplication.

Assumptions

I am working on getting Erlang packages into Nix. The approach is similar to what is done for Haskell. Here are main points:

use Hex.pm as a source of tarballs (that's the only reasonable package respository for Erlang AFAIK)
use rebar3 and Mix as foundation
aim at automatically importing all of newest packages on Hex (including their transitive closures) use not-yet-developed set of tools
packages should include both source code and bytecode files

Current status (2017-09-24)

The following basic features are implemented already:

fetchHex automates fetching packages from Hex.pm CDN
buildHex which automates definition of Erlang packages to be fetched and built from Hex.pm CDN
buildMix to build Mix packages
support building packages which depend on other Erlang packages
support adding rebar3 plugins for compilation of packages
support building of packages with ports (using rebar3-pc plugin)
implemented hermetic bootstrap and build for rebar3
two versions of rebar3 are provided:
- rebar3 is one with applied patch which makes it hermetic (forbids any attempts to download stuff from the internet)
- rebar3-open which is unmodified version of rebar3
implemented support for nix-shell using ERL_LIBS (just run nix-shell -A erlangPackages.lager.env) #11818
[NEEDS UPDATE] hex2nix automates creation of pkgs/development/erlang-modules/hex-packages.nix file based on hex.pm registry, which includes all buildable versions of packages, including multiple versions of the packages
doc/erlang-users-guide.xml contains documentation
support for multiple erlang versions
support for Elixir and LFE
support building Elixir/LFE and all packages from hex-packages.nix with various Erlang versions.

Rebar3 depends on a concept of registry, which is usually fetched from S3. I've created a separate github repo in which I plan to snapshot that single file from S3. This is automated in https://github.com/erlang-nix/hex-pm-registry-snapshots/blob/master/update.sh

Further work

Short term

extract out of hex2nix Nix expression creation for Erlang/Mix packages into a common library;
adapting hex2nix to use the common library;
reuse the library to implement rebar2nix as a rebar3_nix plugin able to create shell.nix and default.nix in two modes:
- self-contained with all deps explicitly written in the default.nix (like in https://github.com/NixOS/nixpkgs/blob/b1140f92ccf83aad933e2a25f94206f71b9a710f/pkgs/development/tools/build-managers/rebar3/default.nix#L10 ) - useful e.g. to upgrade rebar3 and mix in nixpkgs
- reusing beamPackages as much as possible - the default mode;
implement mix2nix (or a mix plugin) doing equivalent of rebar3_nix;
add "shell" command to both rebar3 and mix plugins to enter a self-contained shell mode with a specific OTP version and all deps available in PATH and in ERL_LIBS
upgrading all of the non-mass-imported Beam packages in nixpkgs using the self-contained mode;
upgrade the mass-imported beamPackages using the default mode.

Long term

implement running tests as part of the build
investigate if it makes sense to provide "binary" and/or "binary and sources" or just "sources" packages (there are some packages on Hex which provide some binaries which are useful to be put into user's environment)
investigate integration Erlang releases with Nix ecosystem
instigate discussion with Erlang community about some sort of LTS package set

Support for `nix-env`

Support for nix-env has been discouraged by @peti and @domenkozar as unsustainable and giving very little benefits.

gleber commented 8 years ago

I've redone https://github.com/rebar/rebar3/pull/982 according to owners of rebar3, waiting for feedback.

gleber commented 8 years ago

Tooling for this will be placed in: https://github.com/erlang-nix

gleber commented 8 years ago

Here are various changes to the tooling from last week:

https://github.com/NixOS/nixpkgs/pull/12429 https://github.com/NixOS/nixpkgs/pull/12444 https://github.com/erlang-nix/hex2nix/pull/6 https://github.com/erlang-nix/rebar3-nix-bootstrap/pull/1 https://github.com/erlang-nix/rebar3-nix-bootstrap/pull/2 https://github.com/erlang-nix/nixpkgs/pull/2

rasendubi commented 8 years ago

(triage) @gleber any update?

rasendubi commented 7 years ago

Closing because of inactivity.

domenkozar commented 7 years ago

@rasendubi let's not close meta issues. We can't expect people just deliver code every week or month.

rasendubi commented 7 years ago

@domenkozar, no problem. I've closed because the issue hasn't been updated for ~9 months - seems like nobody cares about it and the work is abandoned. I would be glad to be proven wrong.

gleber commented 7 years ago

Updates which I have forgotten to link to this issue:

Current rough plan of actions is outlined in this comment: https://github.com/NixOS/nixpkgs/issues/26279#issuecomment-313846466

ankhers commented 7 years ago

I'm also going to add the following PR's to this list.

2017
- 26716
- 27295

gleber commented 6 years ago

@ankhers Thanks for linking to those PRs which I've missed.

I've updated the issue main text with current status and future plans.

ankhers commented 6 years ago

I will start working on the common library and the mix2nix plugin.

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

NixOS / nixpkgs