opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
At present includes #5991, #5994 and #5997. The diff becomes slightly less hideous with #5997 merged. The PR includes the behaviour of #5996 and may need #5998 for CI to pass properly.
This PR considerably extends the ideas from #5963. The primary goal is to extend opam init on Windows to recognise as many scenarios for getting either Cygwin or MSYS2 as we at present know to exist, so that users are presented with meaningful choices. In particular, this addresses running opam init from with a Cygwin or MSYS2 bash shell (in particular, #5952 is fixed).
I think it's helpful to start with some scenarios with beta2, and then explain the fixes from there:
From cmd/pwsh, with nothing other than opam installed, we already have an excellent experience which recommends pausing to install Git for Windows (with helpful advice on that added #5963) and which installs Cygwin.
If the user has installed Git for Windows using its (non-default) "Use Git and optional Unix tools from the Command Prompt" setting (or run choco install git /GitAndUnixToolsOnPath), then beta2 displays rather more Git options than might be nice, but #5963 also makes this a quite elegant experience by default (the bash.exe included on PATH is correctly shadowed by the internal Cygwin installation, which is recommended).
If the user installs things with Scoop - e.g. scoop install cygwin msys2 git - then opam init presently works, but neither Git for Windows, MSYS2 or Cygwin are recognised by opam.
Similarly, if the user installs with Chocolatey - e.g. choco install cyg-get msys2 git - then opam init does detect Git for Windows, but does not detect either the MSYS2 or Cygwin installations.
Similarly, if the user installs with winget - e.g. winget install Cygwin.Cygwin MSYS2.MSYS2 Git.Git - then we have an experience similar to Chocolatey, although as Cygwin is installed to the default C:\cygwin64, that is recognised by opam.
If the user runs opam init from Git Bash (not a recommended thing to do), then we get hit by #5984, but if we work around that then the experience is OK. This is a scenario where there are tools on the PATH, and running with --no-cygwin-setup is not as OK - opam var os-distribution reporting win32 instead of msys2. That said, using Git Bash to run opam init is never going to be a good idea (it has no package manager - it's intended for interaction with Git only).
If the user has elected to configure their PATH to have MSYS2's main bin directory permanently installed, then the default behaviour of opam init is pretty good, unless MSYS2's curl shadows Windows curl.
If the user runs opam init from within MSYS2's bash the experience both with and without --no-cygwin-setup is pretty good.
If the user has elected to configure their PATH to have Cygwin's main bin directory permanently installed, then the default behaviour is affected by #5952.
Running opam init from within Cygwin's bash is similarly affected by #5952.
In all of these scenarios where the user has preinstalled either MSYS2 or Cygwin, we do have the issue that opam init --no-cygwin-setup (i.e. just using what's found) will typically fail because patch, unzip and so forth are not installed by default.
The changes here seek to make all of those scenarios as awesome as possible.
At a very high-level the changes are:
MSYS2 and Cygwin are now treated equally where sys-pkg-manager-cmd is concerned (if both have been defined, Cygwin takes priority). In particular that means that MSYS2 gets added to PATH in exactly the same way as Cygwin.
os-distribution is no longer explicitly set by opam init. Instead, os-distribution is always determined by looking at cygpath.exe. The internal or --cygwin-location treatment merely affects PATH resolution (i.e. the internal cygbin addition) - once cygpath.exe is resolved, it becomes the source of truth. Thus, from an MSYS2 shell, opam var os-distribution is now msys2by default.
For now, I've removed the prompting for Cygwin setup. The idea here is that we treat Cygwin setup as being equal between both an internal and external installation of Cygwin. In particular, this fixes a bug in depext treatment when initialising with --cygwin-location as Cygwin setup was never downloaded.
The final, somewhat horrific, refactor in OpamClient attempts to deal with the fact that the Git menu and the Unix tools decisions are not independent. The Unix tools are now determined first, and that feeds into the decision about what may need to be done for Git.
Various tricks are now exploited to find Cygwin and MSYS2 installations:
Cygwin's two registry keys are inspected
The default locations of C:\cygwin64 and C:\msys64 are used
A path search for msys2.exe provides an heuristic for seeeking an MSYS2 installation
Special handling for Scoop is added, both to identify its Git for Windows and to identify its MSYS2 installation
A key feature of all of these changes is that we only ever cygcheck and cygpath while inspecting things. In particular, we have to be very careful about running bash since we have no idea what it may do.
Chocolatey and winget both correctly initialise their MSYS2 installations, but unfortunately Scoop leaves this as an instruction to the user. Again, I've added some special handling to detect this - but this is treated just as running a depext command, so opam asks about it first.
The depext system is now integrated into opam init so opam will offer to add the required packages to both Cygwin and MSYS2 for existing installations, rather than simply erroring.
Put together, the changes mean that the user is presented with meaningful choices in all 10 of those scenarios and, in particular, they will see all of the packages they have already installed. If they agree with everything opam suggests, they should then always end up with initialised opam with a compiler 🥳
On my system, when added to #5992, opam init run from Cygwin's bash now offers:
DRA@Tau /cygdrive/c/Devel/opam
$ ./opam init
<><> Unix support infrastructure ><><><><><><><><><><><><><><><><><><><><><> 🐫
opam and the OCaml ecosystem in general require various Unix tools in order to operate correctly. At present, this requires the installation of Cygwin to provide these tools.
How should opam obtain Unix tools?
> 1. Use tools found in PATH (Cygwin installation at C:\cygwin64)
2. Automatically create an internal Cygwin installation that will be managed by opam (recommended)
3. Use Cygwin installation found in C:\cygwin64
4. Use Cygwin installation found in C:\OCaml64
5. Use Cygwin installation found in C:\cygwin64-2
6. Use Cygwin installation found in C:\Users\DRA\AppData\Local\opam\.cygwin\root
7. Use Cygwin installation found in C:\Devel\Roots\beta2-testing\.cygwin\root
8. Use Cygwin installation found in C:\Cygwin64-beta2-testing
9. Use Cygwin installation found in C:\cygwin64-throw
a. Use Cygwin installation found in C:\Devel\Roots\opam-testing\.cygwin\root
b. Use Cygwin installation found in C:\Devel\Roots\final-test\.cygwin\root
c. Use Cygwin installation found in C:\Devel\Roots\env-testing\.cygwin\root
d. Use Cygwin installation found in C:\cygwin64-throw2
e. Use Cygwin installation found in C:\Devel\Roots\msys2-native\.cygwin\root
f. Use Cygwin installation found in C:\Devel\Roots\os-dist-cygwin\.cygwin\root
g. Use MSYS2 installation found in C:\msys64
h. Use another existing Cygwin/MSYS2 installation
i. Abort initialisation
[1/2/3/4/5/6/7/8/9/a/b/c/d/e/f/g/h/i]
kit-ty-kate
commented
3 weeks ago
At present includes #5991, #5994 and #5997. The diff becomes slightly less hideous with #5997 merged. The PR includes the behaviour of #5996 and may need #5998 for CI to pass properly.
This PR considerably extends the ideas from #5963. The primary goal is to extend
opam initon Windows to recognise as many scenarios for getting either Cygwin or MSYS2 as we at present know to exist, so that users are presented with meaningful choices. In particular, this addresses runningopam initfrom with a Cygwin or MSYS2 bash shell (in particular, #5952 is fixed).I think it's helpful to start with some scenarios with beta2, and then explain the fixes from there:
opaminstalled, we already have an excellent experience which recommends pausing to install Git for Windows (with helpful advice on that added #5963) and which installs Cygwin.choco install git /GitAndUnixToolsOnPath), then beta2 displays rather more Git options than might be nice, but #5963 also makes this a quite elegant experience by default (thebash.exeincluded on PATH is correctly shadowed by the internal Cygwin installation, which is recommended).scoop install cygwin msys2 git- thenopam initpresently works, but neither Git for Windows, MSYS2 or Cygwin are recognised by opam.choco install cyg-get msys2 git- thenopam initdoes detect Git for Windows, but does not detect either the MSYS2 or Cygwin installations.winget install Cygwin.Cygwin MSYS2.MSYS2 Git.Git- then we have an experience similar to Chocolatey, although as Cygwin is installed to the default C:\cygwin64, that is recognised by opam.opam initfrom Git Bash (not a recommended thing to do), then we get hit by #5984, but if we work around that then the experience is OK. This is a scenario where there are tools on the PATH, and running with--no-cygwin-setupis not as OK -opam var os-distributionreportingwin32instead ofmsys2. That said, using Git Bash to runopam initis never going to be a good idea (it has no package manager - it's intended for interaction with Git only).PATHto have MSYS2's main bin directory permanently installed, then the default behaviour ofopam initis pretty good, unless MSYS2'scurlshadows Windowscurl.opam initfrom within MSYS2'sbashthe experience both with and without--no-cygwin-setupis pretty good.PATHto have Cygwin's main bin directory permanently installed, then the default behaviour is affected by #5952.opam initfrom within Cygwin'sbashis similarly affected by #5952.In all of these scenarios where the user has preinstalled either MSYS2 or Cygwin, we do have the issue that
opam init --no-cygwin-setup(i.e. just using what's found) will typically fail becausepatch,unzipand so forth are not installed by default.The changes here seek to make all of those scenarios as awesome as possible.
At a very high-level the changes are:
sys-pkg-manager-cmdis concerned (if both have been defined, Cygwin takes priority). In particular that means that MSYS2 gets added to PATH in exactly the same way as Cygwin.os-distributionis no longer explicitly set byopam init. Instead,os-distributionis always determined by looking atcygpath.exe. The internal or--cygwin-locationtreatment merely affects PATH resolution (i.e. the internal cygbin addition) - once cygpath.exe is resolved, it becomes the source of truth. Thus, from an MSYS2 shell,opam var os-distributionis nowmsys2by default.--cygwin-locationas Cygwin setup was never downloaded.bashsince we have no idea what it may do.opam initso opam will offer to add the required packages to both Cygwin and MSYS2 for existing installations, rather than simply erroring.Put together, the changes mean that the user is presented with meaningful choices in all 10 of those scenarios and, in particular, they will see all of the packages they have already installed. If they agree with everything opam suggests, they should then always end up with initialised opam with a compiler 🥳
On my system, when added to #5992,
opam initrun from Cygwin's bash now offers: