RFC fixing build-mix - Githubissues

happysalada commented 3 years ago

Describe the bug Running the build-mix helper on a Phoenix project fails with ** (Mix) The task "compile.phoenix" could not be found. Did you mean "compile.elixir"? The problem is that the depencies are not available to the builder. Therefore this problem is not unique to Phoenix projects.

To Reproduce Steps to reproduce the behavior:

On a fresh phoenix project

try to build with the following for example

let
nixpkgs = import '<nixpkgs>';
 packages = nixpkgs.beam.packagesWith nixpkgs.beam.interpreters.erlangR23;
in
packages.buildMix {
name = "union";
src = builtins.fetchGit {
url = "ssh://git@github.com/happysalada/union";
rev = "5c1253231b7d37a45e14671ff37888173980e082";
};
elixir = nixpkgs.beam.packages.erlangR23.elixir_1_11;
version = "0.3.8";
}

This particular repo is private, but any public phoenix repo will do

I don't know how long this has been broken. Perhaps it's not supposed to be used in the way I'm using. Perhaps @DianaOlympos @NobbZ @Parasrah have an opinion

My idea with this issue would be more to discuss ways of solving this.

I think everybody is aware of https://github.com/hauleth/nix-elixir. It seems to me that the override provided there in buildMix' is the best alternative so far. Of course we can't integrate those changes upstream without @hauleth 's approval. I was under the impression he did not want to maintain it.

If that works for everybody, I'm proposing that we integrate those changes into nixpkgs.

I'm aware that everybody is busy and that this might take quite some time to figure out. I'm available to make PRs, be a maintainer, or just test things.

For the good of the community I hope we can move this issue forward.

@ericbmerritt @yorickvP @elitak Would be happy to hear your thoughts if you have time

Additional context Add any other context about the problem here.

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

yorickvP commented 3 years ago

I vaguely remember something like this breaking and us working around it by copying in the deps manually. At least, we don't have this issue somehow. cc @balsoft ?

balsoft commented 3 years ago

We're using https://github.com/transumption/mix-to-nix

NobbZ commented 3 years ago

As I do not have any phoenix project by myself, would you mind sharing a minimal public repository which one can clone, enter and nix-build?

It can be really minimal, perhaps even a just one created with phx_new as you describe the issue.

DianaOlympos commented 3 years ago

Quite honestly, my current stance is to get rid of build-mix and let 3rd party like mix-to-nix or other things deal with it.

At this point, there is no easy way to bring the dependencies that works well. The NixOS community have been mostly against using an intermediate FOD derivation like buildGoModule does. It would be the easy path forward. Flakes are not ready and do not solve that problem. And any other solution need us to translate a lot of mix and hex work into nix itself.

If i am missing some options, feel free to tell me, i would love to get some.

Now on the short run, there are probably ways to fix this particularly bug, i can have a look. But the whole setup of the beam packages and their tools is a mess. I would be for ripping it all up and see what the community consensus become.

Of note, beginning with otp23, rebar2 will be broken for everyone.

Tbh at this point my biggest problem touching all of this is that i have no idea who use what, what is needed and what is desired to not be available. I tried here, but getting any traction have proven... difficult.

https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202

DianaOlympos commented 3 years ago

As a note, if some way forward need us to simplify/modify/extend mix to make it easier to validate and use it for reproducible builds, i think the mix team would be open to change. But that need us to have a direction.

happysalada commented 3 years ago

@NobbZ Thanks for joining the RFC! I've prepared a phx.new repo with some nix basics https://github.com/happysalada/nix_demo It includes a shell.nix (start the server with iex -S mix phx.server) I included 3 scripts to test the various tools (nix-elixir, mix-to-nix, build_mix) All the relevant instructions to test are in the readme. I tried using nixpkgs-unstable to test the water. All 3 tools fail the build. As far as I know, the nix-elixir failure seems to be due to an old rebar3 package, so it should be a matter of just waiting for the next update of the channel (the rebar3 version update has been merged recently).

@DianaOlympos Thank you for your comment. Have you checked the https://github.com/hauleth/nix-elixir. tool? It seems to me the closest to getting something that would work. To relate to the discussion you posted, nix-elixir is not trying to be a Fixed Output Derivation tool, that makes it simpler, that's why I think it's a simpler task to integrate it into nixpkgs I think. That discussion made me discover this repo https://github.com/cw789/elixir_nix_seed It looks quite complete, I'm going to add a deeper look. (Thanks also for posting that discussion, I'll read it in more details. I'm definitely game to help in the maintenance of the ecosystem).

To summarize my point.

I think there are many things that can be done, however integrating something like what nix-elixir is doing seems to be the easiest step forward. (it definitely leaves some problems to be addressed.)

Last, personally, I would really like a helper or an example to build a Phoenix project. I think it could remove a tremendous hurdle for the Elixir community to join the Nix ecosystem.

DianaOlympos commented 3 years ago

@happysalada it does build one, that is what the fetchMixDeps function does. I have no problem with this, it is what i would like to have in nixpkgs. But the nix community have been relatively against that kind of ideas. Does not mean we can't get it merged ofc, just that it is not the path they are going for.

happysalada commented 3 years ago

@DianaOlympos If I understand correcty.

There is nobody with merge right that is working in the BEAM space. I'm new to the community. Is it hard to get someone to get merge rights? Are there 10 people only with merge rights or more like 100?
The nix community would prefer a tool that would do a lot of what the package managers already does. This is a problem since the mix.lock format has never been publicly stated as stable.

I think we should try to merge something like fetchMix then. The second step would be to integrate the changes to the buildMix, to get it to build properly.

If you think this can be merged, I'm all for trying. Who do you think would be the best candidate to help us merge it?

@hauleth we will probably take a lot of the work you have done. My impression is that you don't want to be involved in maintaining this, that's perfectly fine! I hope I'm not misunderstanding your consent to us using your work.

DianaOlympos commented 3 years ago

@happysalada The most active maintainer officially written down for the erlang stuff is probably @ankhers . But as far as i know, there is noone that understand well the current whole setup of beampackages whole library and setup. And noone with the time to maintain it over the long run.

Trying to merge something like fetchMix is on my list since last year. Just noone with the time to write it, integrate it with the rest of the beam stuff, shepherd it in and keep maintaining it afterward, has been found. I definitely had no time this year despite trying to fix a ton of stuff.

hauleth commented 3 years ago

Sorry for being late to the discussion.

Of course we can't integrate those changes upstream without @hauleth 's approval.

I forgot to add license (which I will fix shortly), but in general - are you f****g kidding me? Of course I would approve something like that.

I was under the impression he did not want to maintain it.

I can, however recently I got different obligations and I cannot keep up with all my projects. That is why there is a little bit stagnation there. I will try to get everything straight.

ankhers commented 3 years ago

So I don't really have a horse in this race, but I personally like the approach of having a tool that reads a lock file and output your nix dependencies from that (cabal2nix, node2nix, pypi2nix, etc). I started writing rebar32nix but never got around to finishing it. Maybe I should just to get the ball rolling on one approach.

Either way, I am more than happy to help out in whichever direction is deemed the most appropriate.

As for the big ball of mud that is the current beampackages, I have been wanting to tear that out for a while and start anew.

happysalada commented 3 years ago

Amazing to see so many people pitching in!

I would suggest the following.

As a short term effort, integrate nix-elixir into nixpkgs. I am personally interested in fetch-mix and build-mix. I think other things can be integrated too.
As a longer term effort, like @ankhers is suggesting it would be good to have a more elaborate tool. As the effort with this is a little higher, I feel it's better to do it either in parallel or as a second step.

If nobody is against it, I can start anytime with a PR for fetch-mix. As soon as It is merged, I want to integrate build-mix. @hauleth since it is your code, you have priority if you want to make a PR. If you would rather wait later to make a PR yourself (if you are time contrained) that works fine for me too.

As a reminder the repo https://github.com/happysalada/nix_demo is there for anybody who wants to test.

happysalada commented 3 years ago

One question I have is how tools like node2nix and build-mix are going to work together to build a phoenix project for example. Of course we can figure that out later.

minijackson commented 3 years ago

Also coming late to the discussion (sorry), but I spent some time yesterday trying to package Mobilizon, and have managed to build the Elixir part. It needed some changes in the buildMix and I did it without knowing @hauleth's project, which is much better than my small modifications!

Some thoughts, though:

I have tried to use mixnix to avoid the fixed output derivation, but I had two problems:

The mix.lock file is to my knowledge neither stable (as said @happysalada) nor documented, so I had to modify mixnix so that it could take in the new format
mixnix tries to compile the dependencies, but Elixir can execute arbitrary code at build time, so it makes fixing "sneakiness" very difficult without modifying the generated mix.nix file. This also means that having an Elixir overlay with compiled dependencies might be very difficult...

In the case of Mobilizon, some dependencies depend on git, cmake, and other things. We also have the dependency ex_cldr which downloads some locale files at build time. With the mixSha256 method, it is very easy to fix, we specify the buildInputs in the package derivation, and we can download the locale files in advance, and place them in the right directory of the dependency.

But like the rust nixpkgs ecosystem, we could have both the fixed-output derivation method and the generator method (carnix for Rust). I'm not sure whether this is a good idea to split the packaging method, though.

I'm exited to have a build-mix with dependencies, with that I just need to make the yarn2nix work, and we could have a nice event organization software :smiley:

minijackson commented 3 years ago

I'd also love to be tagged if/when we have PRs to merge some nix-elixir in nixpkgs.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcing-mixnix-build-elixir-projects-with-nix/2444/18

hauleth commented 3 years ago

@ankhers the problem with implementing something like mix2nix is that Mix do not have any guarantees about format of lock file. So we would work in a grey area of "it maybe will work in future". However mix itself guarantees semi-reproducibility if lock file is present (semi part is when someone uses custom SCMs or when Git SCM is used where additional steps are required, which fetchMixDeps does). If there would be a stable and documented way to query lock file programmatically or in machine readable way, then we could use that, but till then, the only stable way I can think of is approach of using fetchMixDeps-like tooling.

ankhers commented 3 years ago

I can try reaching out the the Elixir team and see if they would be willing to stablise/document the format of the lock file for our purposes.

DianaOlympos commented 3 years ago

@ankhers you can but i can give you the answer: no. Have been asked before.

What we may be able to get is an interface through mix. Maybe. If we can document explicitly what we need and how to deal with it on the long run

happysalada commented 3 years ago

@minijackson Thanks for joining! I'm doing a similar thing trying to package a phoenix application. Let's coordinate! My idea was to have a build-phoenix helper directly in nixpkgs. Even if it doesn't happen, we can probably add in the documentation of how to build a phoenix app (as an example).

@ankhers I don't think it's a bad idea to ask the elixir team. They will probably say no like @DianaOlympos is saying, but at least, they know there is more interest in this. It's definitely something they will get to in the long term, so letting them know people are interested might push it up the priority list.

@hauleth When you have a moment, let us know if you can make a PR for fetch-mix, or if you would like somebody else to do it. No rush if you're busy.

DianaOlympos commented 3 years ago

If you don't believe me @happysalada and @ankhers , then join the "build and packaging" working group of the erlef in the erlang slack, (https://erlef.org/slack-invite/erlanger) and feel free to ask :)

happysalada commented 3 years ago

@DianaOlympos I personally absolutely believe you! I was just thinking it would be good if several people asked, so that they would understand it is important for more people than just one.

DianaOlympos commented 3 years ago

@happysalada it is not a problem of number of people, they totally get it. It is that stabilising the lock file format has a lot of cost and they are not in any case ready to do that, which i totally understand.

hauleth commented 3 years ago

Well, there is an API that offers what we want, however it is non-public, so again, we cannot use it in safe way.

happysalada commented 3 years ago

@hauleth When you have a moment, do you think you can make a PR to integrate the fetch-mix code? If you are busy, and you would rather have someone else do it, let me know and I will do it. (If you need a bit of time, that's perfectly fine too).

happysalada commented 3 years ago

I've just added a PR for a first pass on a 'fetch-mix-deps' https://github.com/NixOS/nixpkgs/pull/107440 It's not rocket surgery, but it should be a good first steps. Comments are more than welcome! Let's make elixir a first class citizen of Nix!

happysalada commented 3 years ago

Happy new Year! Just a quick reminder of that #107440 PR if you guys want to take a look. It's very simple and mostly similar to existing things. When you have a moment of course, no rush.

happysalada commented 3 years ago

@hauleth just found one issue with the buildMix. I was personally getting trying to change the current version. Here is the issue for reference https://github.com/elixir-lang/elixir/issues/10517 It will be fixed in elixir 1.12. Not sure yet how to proceed in the meanwhile. After the fetch-mix PR is merged, It should be given more considerations.

happysalada commented 3 years ago

So far the previous issue means that elixir compilation will be broken with rebar3 3.14 which is currently the default version. I'm wondering if it would be possible to revert to rebar3 3.13.2 which was released in April 2020. rebar3 releases

This seems to be that all the elixir versions before 1.12 would be broken with rebar3 3.14. Not sure yet how to proceed with this one. Perhaps we would need to keep 2 different versions of rebar3 for compatibility purposes. The other option would be not to care if everybody would use the latest version of elixir anyway. Then perhaps we could revert rebar3 until elixir 1.12 comes out.

NobbZ commented 3 years ago

Perhaps it is as fine to just use overlays/overrides where applicable?

I'm not sure if we should block erlangers who might "need" the new features of 3.14, just because it doesn't play well with elixir.

So one side of the ecosystem will have to bite in the overrides anyway.

happysalada commented 3 years ago

I think you're right. When I integrate the buildMix part, I will add another version of rebar that people will be able to target for older versions of elixir.

happysalada commented 3 years ago

@yurrriq had the awesome idea of creating a beam maintainers group. I will have time to create a PR on monday probably, feel free to comment if you want to be part of it.

DianaOlympos commented 3 years ago

Sure thing

yurrriq commented 3 years ago

... creating a beam maintainers group.

I've started the process here: https://github.com/NixOS/nixpkgs/pull/110623

happysalada commented 3 years ago

Here is an initial PR on trying to fix buildMix https://github.com/NixOS/nixpkgs/pull/112477 I have included an example with a full build for an example Phoenix Project. That was the good news. The less good news is that there are still a couple of things I'm not sure about, so interested to have feedback. I haven't done the documentation. I should have some time tomorrow to have a go at it.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-elixir-and-dependencies-with-port-compilation/11208/5

happysalada commented 3 years ago

closed by #112477 please re-open if you feel this hasn't been properly addressed.

ghost commented 3 years ago

Uhm, sorry I'm late to the party but I've been using buildMix since months to build my pleroma. I'm not entirely sure how to use the new expression though.

ghost commented 3 years ago

Ah nevermind, while I did use buildMix for one component in a way that's probably not possible with the new releaseMix, the tool I vendored for the rest of pleroma actually ships an own expression for this that I could switch to:

https://git.petabyte.dev/petabyteboy/nixfiles/src/branch/main/pkgs/pleroma/mixnix/mix2nix.nix#L51

It would still be great to have something like this in nixpkgs, but I can live without it.

happysalada commented 3 years ago

@petabyteboy here is a PR for packaging mobilizon https://github.com/NixOS/nixpkgs/pull/119132/files It includes a lot of other stuff, but the gist of it is there. I want to include a PR for a more traditional elixir app, I'll tag you so you can have a look (probably next week). pleroma is on my list of things to do at some point. If you are interested, we can even collaborate on it :-)

ghost commented 3 years ago

Thanks, but I'm not super-interested because I don't like the fixed-output-derivation hack for fetching dependencies, and I already have a working source-built package that does not rely on it.

happysalada commented 3 years ago

Just to be sure, when you talk about the hack, you're talking about removing the .git files from the dependencies?

ghost commented 3 years ago

I'm talking about this: https://github.com/NixOS/nix/issues/2270

DianaOlympos commented 3 years ago

Yeah this is simply not something we ill accept to fix in nixpkgs beam subsystem anytime soon. The mix lockfile is not stable and we do not have a nice interface to it, so any mix2nix will be unstable and not properly reproducible. So we rely on FOD for the time being here. If you want to deal with the breakage when it happens in your own project, no problem ofc :smile:

happysalada commented 3 years ago

Just summerizing the argument here. the current fetchmixdeps fetches dependencies from the internet. Building in this way makes it nowhere apparent that the derivation has dependencies on the internet. The ideal solution should have nix aware of the external dependencies. Let me know if I've misinterpreted something.

On a different note, I can see that you have a different usage of buildMix. If you are interested in maintaining your different usage, I would be willing to help you get it merged. I think both usage can coexist as long as we document properly what they do. Of course, I realize this is quite a commitment, so no worries if you don't want to go through it.

ghost commented 3 years ago

the current fetchmixdeps fetches dependencies from the internet. Building in this way makes it nowhere apparent that the derivation has dependencies on the internet

I think it's pretty clear it depends on the internet, but mix has more logic. What happened for example when the version of mix run in the FOD changes the output format? We wouldn't notice for a long time, because if a build result for the specified output hash is cached, it will not attempt to re-build the derivation, even if the "build recipe" changes. It could also happen that the FOD is not reproducible on other platforms like darwin, so it can only be built in the specific environment where Mix produces the expected hash. These are some of the flaws of Nix (or rather peoples usage of Nix) that Eelo describes in https://github.com/NixOS/nix/issues/2270.

If you are interested in maintaining your different usage, I would be willing to help you get it merged

Nah, I have my own expression (part of my vendor copy of mixnix) and it works for my purposes. I was just confused because I used the nixpkgs buildMix for one dependency that mixnix didn't catch, and now buildMix was gone, but in fact I had another buildMix-like expression in my repository all along.

I disagree with DianaOlympos about that it's a problem for mixnix that the lockfile format is unstable. Sure, over the last two years there have been one or two changes, but it was not too hard to adapt for them, and even when it doesn't work this will not break builds of the already packaged versions of software.

I think the best way to set up Elixir packaging in the more-distant future would be a Nix expression that can directly read the dependencies from a mix.lock file and translate that into many small FOD.

DianaOlympos commented 3 years ago

I disagree with DianaOlympos about that it's a problem for mixnix that the lockfile format is unstable. Sure, over the last two years there have been one or two changes, but it was not too hard to adapt for them, and even when it doesn't work this will not break builds of the already packaged versions of software.

I mean yeah this is where we disagree. The current decision is that both the lock file and the mix behaviour may change, which in both case has proven too problematic for nearly everyone out there. On top of that, translating the mix.lock to a nix format keep being a really problematic problem, relying on really brittle code. Nix is simply not designed to interact well with this kind of stuff. Any solution will be brittle. You have to keep in mind that there is zero maintainer for the BEAM stuff on nixpkgs, so anything that need regular change like the lockfile format evolving is simply not sustainable. It is not "super high" to maintain (in theory we would need to maintain the different formats so things get harder), but it is high.

It also need us to support all the different way to fetch dependencies that are allowed in mix.lock, which is higher than you think

I think the best way to set up Elixir packaging in the more-distant future would be a Nix expression that can directly read the dependencies from a mix.lock file and translate that into many small FOD.

Before we get there, we need maintainers in nixpkg for the BEAM. There is a lot of tech debt to clean up and change that noone has been able to so far.

A better solution would probably be to engage with the mix and hex team and try to define a spec and cli tool on their side that would allow to get a fixed well defined output for mix.lock. I would like very much for mix to allow us to use a cli tool to query mix.lock. That would probably be a nicer interface for us.

I would particularly like to get a list of what would be needed so i can bring it to the build and packaging WG of the EEF. This kind of work could probably be funded. May be a job for @hauleth , @petabyteboy or @happysalada as you dealt with this recently ?

happysalada commented 3 years ago

If I understand well, the arguments around a Fixed Output Derivation are the following.

it accesses the internet and so we have no guarantee of what it could be doing. The way I understand things, mix only accesses hex and git urls, so not sure this is a real issue.
it does not get rebuilt unless a sha256 is updated. This can create issues where breakage could be detected much later, until the sha256 is updated. If I understand well, this is mitigated by the fact that if there is a new mix, elixir, or hex version, things should be rebuilt and so potentially breakage could be detected earlier. The fact that things can break 'under your feet' is still a problem.
It does not provide a dependency graph. The main disadvantage here is that any dependency update will cause all the dependencies to be downloaded and rebuilt. For bigger projects, this could be an issue. I do not have any data on how a small project would be affected.
it does not depend on the unspecified mix.lock file. This could potentially change (not sure how often this has happened in the past) and therefore things might break.

There is disagreement around how much maintenance would actually be required to depend on the mix.lock format. Potential solutions would be

Rely on the current format, that would need a maintainer to step forward.
Engage with the elixir team to define an api to interact with the mix.lock, potentially involving a cli.

Having a well defined API with mix could potentially mean that some funding might be requested to do the actual work of a mix-to-nix function.

Please let me know if I misrepresented or forgot anything.

DianaOlympos commented 3 years ago

Having a well defined API with mix could potentially mean that some funding might be requested to do the actual work of a mix-to-nix function.

More like we could probably get funding to implement that API in mix, but we would need to know what would a mix to nix tool need from that API so we can explore the design of it.

happysalada commented 3 years ago

I started a discussion on the elixir google group. I was intentionally vague in order to let people contribute their opinions. I suggested perhaps outputting a json object. My general idea is something akin what niv produces. A json object with perhaps a rev, url, and sha256. I'll post updates if there are any.

NixOS / nixpkgs

RFC fixing build-mix #105002