NetBSD installation issues

[maroneze]

I just installed opam on a NetBSD 7.1.2 (via pkg_add ocaml-opam), and it seemed to work fine, however I cannot do opam init since I have no solvers, and I cannot find NetBSD packages for aspcud, packup nor mccs.

I wonder if I missed something?

I found a mention to NetBSD in this issue, but the CUDF Solver Farm link mentioned by rneswold is broken.

In the Specifying solver dependencies page in the opam website, the link to aspcud is also broken.

I managed to get packup and compile it (after installing gmp, configuring CPATH and LIBRARY_PATH to include the /usr/pkg/include and /usr/pkg/lib respectively), then I installed it, tried opam init again, and this time I had another error:

=-=- Fetching repository information
[ERROR] Could not update repository "default": "/usr/pkg/bin/curl
 --write-out %{http_code}\\n --retry 3 --retry-delay 2 --user-agent opam/2.0.0~beta5
 -L -o /tmp/opam-3783-c61e7b/index.tar.gz.part https://opam.ocaml.org/index.tar.gz"
 exited with code 60
[ERROR] Initial download of repository failed

The error was due to the fact that I did not have the NetBSD CA certificates, however it was not obvious from the error message.

After doing all of that, I finally had a (mostly) working opam installation, with OCaml 4.06.0. Still, when trying to install several packages, I have opam freezing systematically while applying patches to package num. With OPAMDEBUG=1, I get:

00:15.530 PARALLEL  Next task in job 980392853: /usr/bin/patch
 -p1 -i /root/.opam/default/.opam-switch/build/num.1.0/findlib-install.patch
Processing 6/24: [num: patch]

My virtual machine indicates no disk nor network accesses, and there is no CPU usage. It really seems frozen. If I Ctrl+C and try again, the same thing happens.

I tried installing another package which also has a patch, such as camlzip, but it worked as expected.

At this point, I have no idea if it's an issue with opam 2, with the num package, with NetBSD, my installation, or something else. I'd like a suggestion on what to try next.

[rjbou]

It seems that you had a succession of tiny errors & incompatibility.

Regarding certificates and curl error, we don't have more information than curl output and return code, it is difficult to specialise & maintain the correct internal curl error. However, opam download error are more human understandable.

We already had issues with patch in {Open,Free}BSD. In the end, we choose to use gpatch instead on those platform (cf. threads https://github.com/ocaml/opam/issues/3433#issuecomment-400346546 & https://github.com/ocaml/opam/pull/3444#issuecomment-405874603).

[dra27]

@maroneze - it would be worth pinging the package maintainer for ocaml-opam since beta5 is very out-of-date. It would also be worth them making two changes:

• The binary package should be built with an internal solver, which just requires a C++ compiler
• This could be made mandatory by adding a dependency on a C++ compiler and also ensuring configure is invoked with --with-mccs. By default opam chooses to build without an internal solver if no C++ compiler is found (or the ocaml-mccs package is not found) - specifying --with-mccs turns this into a hard error.

[maroneze]

This time I tried downloading https://github.com/ocaml/opam/archive/2.0.0-rc4.tar.gz directly and running gmake cold, and it failed with:

env MAKE=gmake ./shell/bootstrap-ocaml.sh 
--2018-07-25 15:53:53--  http://caml.inria.fr/pub/distrib/ocaml-4.06/ocaml-4.06.0.tar.gz
Resolving caml.inria.fr (caml.inria.fr)... 128.93.101.34
Connecting to caml.inria.fr (caml.inria.fr)|128.93.101.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4041107 (3.9M) [application/x-gzip]
Saving to: 'ocaml-4.06.0.tar.gz'

ocaml-4.06.0.tar.gz                  100%[==========>]   3.85M  10.5MB/s    in 0.4s    

2018-07-25 15:53:54 (10.5 MB/s) - 'ocaml-4.06.0.tar.gz' saved [4041107/4041107]

Bad checksum for http://caml.inria.fr/pub/distrib/ocaml-4.06/ocaml-4.06.0.tar.gz.tar.gz:
- expected: 66e5439eb63dbb8b8224cba5d1b20947
- actual:  

The culprit is a call to sed inside shell/bootstrap-ocaml.sh, which does not work well with BSD sed, so the replacement does not take place and a double .tar.gz is appended to the URL. By replacing the call with gsed, the bootstrap script works fine until it reaches jbuilder, at which point it fails (and I've been told this is a known issue).

[maroneze]

Further report: this time I used NetBSD 8.0 (which seems to have no ocaml-opam package, but then again, it was released 2 days ago), installed ocaml-4.06.0 and several ocaml packages with pkgin, and I was then able to compile the master branch of opam and run it. I was also able to install num without patch freezing.

Some packages require patches (e.g. conf-gmp), but overall opam itself is mostly working.

[maroneze]

I'd like to ask NetBSD enthusiasts if there is a better way to handle the issues I'm having with several packages.

NetBSD, as far as I understand, by default installs all files in /usr/pkg. Which means that all OCaml packages with hardcoded paths have to include an extra line to handle this "exotic" configuration. Several of them already do this for macOS or other BSDs, but this is a repetitive and cumbersome approach.

I believe it could make more sense to simply ask NetBSD users to make a symbolic link from /usr/local to /usr/pkg, so that packages would almost work. Almost, because since netbsd is not freebsd | openbsd, the opam rules will default to Linux, and so the paths will be incorrect. But simply adding | netbsd would make them work, instead of requiring an extra line of copy-pasted content, plus the modification to the matching rules (e.g. adding & os != "netbsd", then {os = "netbsd"}, etc).

In both cases, patches would be required:

1. Small patches, but the user must add a symbolic link to /usr (I don't know the implications of this);
2. Large patches, but no outside modifications required.

Is there another alternative?

[dra27]

opam, by default, maintains everything in ~/.opam - which packages are you referring to?

[maroneze]

Packages which do not use pkg-config, such as conf-gmp and its derived packages (e.g. zarith), and generally every package which assumes something will be in /usr or /usr/local (from what I can see in opam-repository, it seems that most conf- packages do it, such as conf-lldb, conf-tk, conf-openssl, conf-openblas, conf-netsnmp, etc, and a few other packages as well).

Looking again in more detail, I see that in fact most non-conf packages which include hardcoded paths are actually versions 0.x, which indicates they are not very mature, or old packages using ocaml <=4.00. So it may be in fact doable to patch each file as needed.

[krytarowski]

/usr/pkg is pkgsrc specific path, it's the same on other pkgsrc platforms, and /usr/local is for local software out of package management.

In general you should install and handle software through pkgsrc. Please do not hardcode any paths like /usr/pkg in 3rd party software, as it's not correct. People are free to install pkgsrc into $HOME/pkg, /opt/pkg etc.

[maroneze]

Thanks for the info, I am ruling out the link from /usr/local.

Is there an environment variable or something like pkg-config that allows discovering where the user installed pkgsrc? Or should I assume that the user must have defined a specific variable pointing to it prior to running opam?

As in, conf-gmp works by trying to compile a short file containing #include <gmp.h>. If I try to compile it via cc test.c it fails, so the compilation command-line must include both -I<path/to/pkgsrc>/include and -L<path/to/pkgsrc>/lib. Currently this is done with hardcoded paths in the opam file, so for instance in macOS, I believe it adds /opt/local in case MacPorts is installed.

Other than with some external tool or assumption about the user environment, I don't see how to discover this configuration. There is even an unanswered SuperUser question about it.

[krytarowski]

I think that a user shall not detect pkgsrc from 3rd party software. If we want to integrate 3rd party programs with pkgsrc-aware paths, we must install it from pkgsrc; not the other way around. Hardcoding /usr/local is the correct approach, as a user is free to install OCaml bypassing pkgsrc or any other package management.

This means that the correct approach is to use directly the following package: http://pkgsrc.se/misc/ocaml-opam

It is actively maintained as the last commit and upgrade is few days old.

Regarding OCaml specific questions in the combination of pkgsrc, please send a mail to tech-pkg@netbsd.org. I'm not a programmer using OCaml.

[dra27]

@krytarowski - so the only way to use libz, say, if installed via pkgsrc is yourself to be built using pkgsrc? That's unworkable for development tools - how are you supposed to compile anything?

[krytarowski]

In NetBSD we have a clean separation of base and packages (ports / pkgsrc).

Base ships with the kernel, bootloader, toolchain, userland tools, X and some useful services and its dependencies like dhcpcd, ldap, ftpd, bozohttpd etc.

From a programmer point of view, you do not (usually) maintain the base programs, but just maintain a set of externally installed software from said pkgsrc. Upstream NetBSD is responsible to keep the base system in order, you are free to install your local software in /usr/local or through pkgsrc.

libz is inside base as a requirement for many programs, but if you need some special version you are free to use the pkgsrc one and you are guaranteed that they will cohabit peacefully without conflicts (unless there are some bugs in a set of packages, it might be possible for so elementary software dependency).

[krytarowski]

It's different to the model from a typical linux distro, where you maintain the entire system in packages. You can treat BSD like Windows or MacOSX, that you get a foundation of your OS and just install addons on top of it in whatever version you like, and whatever paths you want.

Hardcoding some paths in 3rd party software like MacPorts or similar in opam would be as wrong as hardcoding pkgsrc paths on NetBSD.

I would ask a similar question. is Windows or MacOSX unworkable? How is it possible to build anything on top of them?

Thanks to pkgsrc we can maintain several versions of the same software, without conflicting with userland. We also do not care whether base GCC is too old or similar, we can get a wide range of compilers from pkgsrc and even configure the framework to use them for building the pkgsrc packages.

I know that BSD is mind-blow for typical Linux schemes.

[maroneze]

Sorry if I misunderstood your message, but it seems that it's by design impossible to have something similar to Homebrew's brew --prefix, or npm root?

Now that you mention it, apt and dnf (two distribution-level package managers I use often) also do not seem to give this information, but these are (as far as I know) hardcoded to be installed in the same path every time, very rarely being installed in other places. So in practice I have never needed to find out similar information about them.

I think one of the issues is that opam, unlike a distribution-level package manager such as apt/dnf/pacman/etc, can only manage OCaml packages, so while it does know where all of those are installed, it cannot handle C-level bindings, such as those needed by OCaml packages using them (e.g. GMP). So the user must install these C-level packages using their own package manager, and then opam will refer to the location where those C bindings were installed. Without some help from the distribution-level package manager, opam is unable to know where to find them. So the solution found for some package managers was to hardcode the path according to where each distribution usually puts them, in a best-effort approach.

I agree that installing ocaml-opam is the right approach, but how to continue after that? You'd either need to have one pkgsrc package for each opam package, which kind of defeats the purpose of having opam in the first place; or should opam try to find its own path and assume that prefix is being used to install the other packages? As in, if it finds itself installed in /usr/pkg/bin, should it suppose that the C bindings it requires will also be under /usr/pkg?

Overall, opam differs from distribution-level package managers because it is specific to a programming language, similar to npm, I believe, but with extra requirements concerning C bindings coming from the system. These extra requirements impose the need for external information or assumptions about paths.

[krytarowski]

I recommend to get in touch with tech-pkg@, there are several people using OCaml and can suggest proper approaches how to handle this particular example.

It's a typical problem not related to pkgsrc, that some programming languages reinvent their packaging system (maven, luarocks, quicklisp, cabal, npm etc) that attempts to be orthogonal to the set of installed software, but it tends to generate issues. I try to defer suggestions how to configure your setup as I miss some details on OCaml.

BTW, Gentoo portage-prefix and Nix are similar to pkgsrc. Both were inspired by the BSD system.

[dra27]

@krytarowski - thanks for taking the time to explain this! If I get the correct impression, the point is (as on Windows) that the libraries could be installed anywhere, so either opam needs to require the user to configure a symlink in a standard place (@maroneze's original suggestion) or opam needs to have this location fed into it when the conf packages are installed in order to use the installed libraries for C bindings.

@hannesm and @avsm (who is away at the moment) can both help out here.

(As far as I can tell, the issue is now morphing over into an opam-repository issue, though for now it's probably more sensible to keep this discussion in one place)

[krytarowski]

My blind guess would be - if we really want to not refer to a pkgsrc version of opam for some reasons - to add a configure option to opam like '--additional-path-with-libraries' '--additional-path-with-includes', but it's a shot in the dark.

Please note that NetBSD by design does not have ldconfig (at least comparable to the Linux one), that instructs the dynamic linker where are the libraries. You must use RPATH (-R linker option) against your executable to instruct the linker to search paths in proper locations.

[hannesm]

@dra27 I'm not sure how to help, it is a current issue with opam and non-GNU/Linux systems (that install their packages in a prefix that is not automatically added to calls of cc (-I DIR) and ld (-L DIR).

The current approach to "fix" this is to add -I /usr/local/include -L /usr/local/lib to package build instructions (opam files) in opam-repository for FreeBSD + OpenBSD. Another set of -I .. -L .. is added for MacOSX (homebrew IIUC) support (please have a look at https://github.com/ocaml/opam-repository/blob/master/packages/zarith/zarith.1.7/opam#L13 for example). If NetBSD uses /usr/pkg for ports/package installations (such as libgmp), we'll need to extend those packages that depend on 3rd party installed libraries with the right paths.

I'm not convinced this is a good solution. I'm not sure whether opam2 is fixing this (maybe @AltGr has some input here?). @krytarowski mentioned --additional-path-with-libraries / includes, that's a good idea, but there's still the plumbing needed to pass these flags from opam to the build systems - either by embedding this into opam files or to extend opam with information what/how to pass these flags to specific build systems.

[krytarowski]

Please remind that ld needs -R next to -L. Probably using the pkgsrc package handles this.

[mcuee]

Please remind that ld needs -R next to -L. Probably using the pkgsrc package handles this.

I thought avrdude NetBSD issue is the same but it turns out to be different issue. Somehow LD_LIBRARY_PATH is not set to /usr/pkg/lib and that is the reason.

Edit, replacing -L with -R does sort out the issue, without the need to setting up LD_LIBRARY_PATH.