Supporting extra repositories

[CohenCyril]

Dear @timbertson, first, thank you for this tool! I am trying to use it to compile coq and coq related projects using the opam extra repositories from https://github.com/coq/opam-coq-archive (3 repositories might be added). I did not find where to specify the repositories opam2nix uses, could you tell me? Best wishes!

[timbertson]

Unfortunately this is currently hardcoded. The earlier version (v0) allowed this, but it got quite complex. The newer version keeps a cache of "the opam repository" and generated selections reference a path within "the repo", so it's now hardcoded to work with a single repo (the official one). This is the first use case I've heard which actually makes use of additional repos.

It would be possible to support multiple repos, but it would be quite a lot of work. It may be simpler to support a single but non-hardcoded repo. Is it possible to aggregate all of the repos you want to use into a single git repository (or .tgz archive)?

[CohenCyril]

@timbertson thanks for your answer. I am mostly an opam repository user and have little experience about maintaining them / setting them up. I have no clue if what you are saying is possible. I will ask around when I have time.

[balsoft]

Hi!

I have faced exactly the same issue as you: the need to use many opam repositories. Because of that, I have written my own (very jank and crude) opam-nix . I don't know OCaml and I had very little time, so I just used what I knew (Haskell) to parse the opam files. The code quality is awful, but it worked for me (traversed 3 interdependent opam repos, including the upstream opam-repository, and built the package with little overrides). Limitations: currently it doesn't do any version resolution, that is left to the user. Also, it's very jank, help would be highly appreciated.

[timbertson]

You know, I thought the one place people would be familiar with OCaml was opam users :laughing:

[avsm]

The good news is that combining multiple opam-repository overlays should be possible with a GitHub Action, since you just need to combine multiple directory trees and deal with conflicts with a simple priority queue. I need something similar for another project I'm working on, so I'll see if I can get something working over at https://github.com/dune-universe/opam-overlays and update this issue when it's done.

[thufschmitt]

If combining repos is that easy, a nice way to allow using multiple repositories without making the code more complicated than needs to be is to give the possibility to use a Nix expression to build the opam repository to use. On the Nix side, combining repositories should just be one symlinkJoin away. Something like

{ pkgs ? import <nixpkgs> {} }:
let
  upstream = builtins.fetchGit {
    url = "https://github.com/ocaml/opam-repository";
    rev = "513d3767ad627b97c4b58c6669fb5bc55238371e";
  };
  dune-universe = builtins.fetchGit {
    url = "https://github.com/dune-universe/opam-overlays";
    rev = "05ec075202d3fead23859d191dff094e3b4ae81d";
  };
in
  pkgs.symlinkJoin {
    name = "combined-opam-repository";
    paths = [ upstream dune-universe ];
  }

I have a poc of this at https://github.com/regnat/opam2nix/commit/2ff2395f0a9ecba7b7316907748dae1d1c9b631d if you're interested

[timbertson]

Thanks @regnat , that's a nice idea - and using a local repo is probably a useful feature regardless of its use in multi-repo. I'm also tentatively working on this (multiple repos) in opam2nix itself.

[timbertson]

If anyone would like to take the multirepo branch for a spin and provide feedback, it's looking like this would be reasonable to support natively. You can add another repo with the --repo flag, and even override the default opam-packages if needed.

Also I remembered why I don't support a local repo already - if you embed the path in the selections document, only people who already have that path (store path or otherwise) will be able to build it, which is not a great experience. So it currently only supports github repos for simplicity.

[timbertson]

Well, I didn't get any feedback good or bad, but I'm pretty happy with the above branch, so I've merged it into the mainline v1 branch now. If you start using it, let me know how it works :)

[CohenCyril]

Hi @timbertson happy new year and thanks for the addition, and sorry I did not have the time to test before, as it is this does not work for me as is. I will open a new issue as it is yet another feature request (and I cannot reopen this one).