ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.24k stars 356 forks source link

Map of repo-name to repo-url/path is global, which breaks git-worktree workflows #5126

Open ELLIOTTCABLE opened 2 years ago

ELLIOTTCABLE commented 2 years ago

I use git worktree fairly heavily, to concurrently work on multiple branches on our monorepo. Unfortunately, this means our opam repo — which exists in that monorepo — doesn't play nice.

Our initial git-repository setup does something a lil' like this:

$ opam repo add --dont-select theproj "./backend/opam" || \
      opam repo set-url theproj "./backend/opam"
$ opam repo add --dont-select mirror "https://opam.oursite.net" || \
      opam repo set-url mirror "https://opam.oursite.net"

Unfortunately, if I run this initial setup in ~/code/monorepo, that means opam now has this understanding:

$ opam repo list --all
# Repository # Url                                                 # Switches(rank)
theproj      file:///home/elliott.cable/code/monorepo/backend/opam /home/elliott.cable/code/monorepo(1/2)
default      https://opam.ocaml.org                                <default> _globals our-placeholder
mirror       https://opam.oursite.net                              /home/elliott.cable/code/monorepo(2/2)

Note that the path ~/code/monorepo is now hard-coded in a global way.

Meanwhile, if someone subsequently sets up another clone of the repository and attempts to give it a clean copy of the switch (which is the entire point of directory-local switches, yes?) ...

$ git worktree add ~/code/monorepo.worktrees/alt && cd !#3
$ opam repo add --dont-select theproj "./backend/opam" || \
      opam repo set-url theproj "./backend/opam"

Wait, hold up. Now the theproj repo points at the alt worktree; both for us, and for the original git-clone. That's bad.


One fix I attempted, was setting the repo during switch-creation; so it's hopefully locally to only that switch:

$ git worktree add ~/code/monorepo.worktrees/alt && cd !#3
$ make init
rm -rf _opam backend/_build
opam init --bare --enable-shell-hook 
make opam-setup
make[1]: Entering directory '/home/elliott.cable/code/monorepo.worktrees/alt'
# adding the repo if it doesn't exist || setting the url of the repo
opam repo add --dont-select mirror  || opam repo set-url mirror "https://opam.oursite.net"
opam update -a # we need to update the repo to make sure that the latest version of the compiler package is detected

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[theproj] no changes from file:///home/elliott.cable/code/monorepo/backend/opam
[default] no changes from https://opam.ocaml.org
[mirror] no changes from https://opam.oursite.net
# setting theproj-setup and theproj-dev-tools-deps as invariant so that they aren't uninstalled by accident later on
opam sw -y --repositories "theproj=./backend/opam,mirror" create . --packages=theproj-setup,theproj-dev-tools-deps,ocaml-base-compiler.4.14.0 --no-install
Creating repository theproj...
[ERROR] Repository theproj is already set up and points to file:///home/elliott.cable/code/monorepo/backend/opam. To change that, use 'opam repository set-url theproj
        file:///home/elliott.cable/code/monorepo.worktrees/alt/backend/opam'.
make[1]: *** [Makefile:26: opam-setup] Error 2
make[1]: Leaving directory '/home/elliott.cable/code/monorepo.worktrees/alt'
make: *** [Makefile:32: init] Error 2

Okay, nope, no suck luck — even if I attempt to create a new theproj repo in opam switch create, it still conflicts with a completely separate theproj repo in another switch.


My last attempt involved using $PWD as the name of the repo, because that's somewhat honest — it is, indeed, a directory-local repo specific to that worktree. Something like the following:

$ opam repo add --dont-select "$PWD" "./backend/opam" || \
      opam repo set-url "$PWD" "./backend/opam"

Unfortunately, this leads — somewhat unsurprisingly — to build-failures like the following, due to non-existent directories like /tmp/opam-NNN//home/elliott.cable/code/monorepo.worktrees/alt/packages...:

#=== ERROR while compiling ragel.6.8+20190930 =================================#
Sys_error("/tmp/opam-25708-888da3//home/elliott.cable/code/monorepo.worktrees/alt/packages/ragel.6.8+20190930/files/ragel.install: No such file or directory")

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ ⬇ fetch   sqlgg    20220401+git
│ λ build   camlp4   4.14+1
│ λ build   ctypes   0.20.1
│ λ build   lwt      5.5.0
│ λ build   ocamlmod 0.0.9
│ λ build   ocamlnet 4.1.9-2
│ λ build   ptime    1.0.0
│ λ build   ragel    6.8+20190930
│ λ build   rresult  0.7.0
│ ∗ install geoip    0.0.3
└─ 

So, tl;dr, I'd love to see one of the following:

  1. Lowering the 'name'-to-'url' map of opam repositories into individual switches, or …
  2. allow some opam repo 'names' to have different values in different switches, or …
  3. allow repo 'names' to have slashes in them, and automatically make that play nice with the filesystem.
opam config report ``` $ opam config report # opam config report # opam-version 2.1.2 # self-upgrade no # system arch=x86_64 os=linux os-distribution=debian os-version=10 [ERROR] No switch is currently set. Please use 'opam switch' to set or install a switch # read-state OpamStd.OpamSys.Exit(50) ```
dra27 commented 2 years ago

This is something which it would be nice to fix properly in opam, but we're a bit nervous that making a small tweak here runs a risk of making the already confusing switch repo selections more confusing which rules out 1 and 2, at least for now. In terms of switch naming, is it possible in your workflow either to be hashing the worktree path or transforming it to create unique repo names for each worktree?