search

Bioconductor / BBS

The Bioconductor Build System
9 stars 11 forks source link

openssl on Mac #378

Open hpages opened 9 months ago

hpages commented 9 months ago

See https://github.com/grimbough/Rhdf5lib/issues/58 and https://github.com/Bioconductor/BBS/issues/245 for some background.

A central idea to the business of building and distributing Mac binaries for CRAN and Bioconductor packages is that these binaries should work on a pristine Mac where those optional system libraries are not necessarily available.

On the Mac builders we want to install Simon's openssl binary from https://mac.r-project.org/bin/. These binaries contain static libraries only (.a files, e.g. /opt/R/arm64/lib/libcrypto.a). They don't contain dynamic libraries (.dylib files). Bioconductor packages that need to link against the crypto library should link against /opt/R/arm64/lib/libcrypto.a.

Also, for the sake of keeping the Mac builders as clean as possible, and to avoid the risk of linking to the wrong libraries, we should try to avoid having the brewed openssl. Note that this one provides both libcrypto.a and libcrypto.dylib.

jwokaty commented 9 months ago

In order to brew uninstall openssl, I had to use the --ignore-dependencies option since it's a dependency for mono and open-babel. I think other packages will also install it. I want to confirm again that these two are fine without it. I checked open-babel yesterday but I didn't run the packages that they are associated with.

Note: (mostly for myself) brew uninstall won't remove dependencies installed for a package; however, you can run brew autoremove for clean up.

I think the documentation is mostly correct now; however, there is one section https://github.com/Bioconductor/BBS/blob/devel/Doc/Prepare-macOS-Big-Sur-HOWTO.md#46-install-the-mysql-client that states you should have a "brewed" openssl. Maybe we need an alternative way to install mysql-client.

vjcitn commented 9 months ago

because that piece relates to ensemblVEP which is a) probably going to be deprecated and b) uses REST API in current implementation, the mysql dimension may be ignored for now? i think a meaningful revision of ensemblVEP would eliminate use of mysql

hpages commented 9 months ago

In order to brew uninstall openssl, I had to use the --ignore-dependencies option

Yeah bummer, that's what I suspected. Let's see what happens on the next report for kjohnson1 for ChemmineOB and rawrr and their rev deps:

Note that ChemmineOB is already broken (and has been for a while on kjohnson1. Looks like the package has never built on Mac arm64 as I don't see the binary in BioC 3.18, BioC 3.17, or BioC 3.16. But that's apparently for other reasons (i.e. not related to openssl), so we might not be able to learn much until that one is addressed first.

Thanks!

jwokaty commented 9 months ago

I just wanted to note that there is an "default" installed /usr/bin/openssl. Doing which openssl, points to this path, which we may want to change. Since kjohnson3 started a run today, I set PATH=/opt/R/arm64/bin:$PATH in /etc/profile such that Simon's openssl is selected. I can undo this on the next run if it's not what we want.

It's not clear to me how the configure script knows what openssl to choose. Are these configure scripts made with autoconf?

jwokaty commented 9 months ago

I think something brew installs openssl? I removed it on lconway last week then wanted to check/compare settings with kjohnson1 today when I noticed it in the brew list, so I checked history and it was uninstalled and its files were removed. I'll check again tomorrow to see if it has reappeared.

Note to remind myself what I did today:

lconway:~ biocbuild$ brew list
==> Formulae
boost           gcc         icu4c           libgit2         libunistring        libyaml         open-babel      readline        xorgproto
ca-certificates     gdbm            infernal        libidn2         libx11          lzo         openssl@1.1     salt            xz
cairo           gettext         isl         libmpc          libxau          mono            pcre            six         zeromq
cpanminus       glib            libcbor         libpng          libxcb          mpdecimal       pixman          sqlite          zstd
eigen           gmp         libevent        libpthread-stubs    libxdmcp        mpfr            pstree          tmux
fontconfig      htslib          libffi          libsodium       libxext         mysql-client        python@3.10     utf8proc
freetype        hwloc           libfido2        libssh2         libxrender      ncurses         python@3.9      wget

lconway:~ biocbuild$ brew uninstall openssl@1.1
Uninstalling /usr/local/Cellar/openssl@1.1/1.1.1o... (8,089 files, 18.5MB)

Warning: The following openssl@1.1 configuration files have not been removed!
If desired, remove them manually with `rm -rf`:
  /usr/local/etc/openssl@1.1
  /usr/local/etc/openssl@1.1/cert.pem
  /usr/local/etc/openssl@1.1/ct_log_list.cnf
  /usr/local/etc/openssl@1.1/ct_log_list.cnf.dist
  /usr/local/etc/openssl@1.1/misc
  /usr/local/etc/openssl@1.1/misc/CA.pl
  /usr/local/etc/openssl@1.1/misc/tsget
  /usr/local/etc/openssl@1.1/misc/tsget.default
  /usr/local/etc/openssl@1.1/misc/tsget.pl
  /usr/local/etc/openssl@1.1/misc/tsget.pl.default
  /usr/local/etc/openssl@1.1/openssl.cnf
  /usr/local/etc/openssl@1.1/openssl.cnf.dist

Warning: The following may be openssl@1.1 configuration files and have not been removed!
If desired, remove them manually with `rm -rf`:
  /usr/local/etc/openssl@3
lconway:~ biocbuild$ rm -rf /usr/local/etc/openssl\@3
lconway:~ biocbuild$ rm -rf /usr/local/etc/openssl\@1.1
jwokaty commented 9 months ago

I don't see openssl@1.1 today, which is a good sign.

jwokaty commented 9 months ago

We should also put export PATH=/opt/R/x86_64/bin:$PATH or export PATH=/opt/R/arm64/bin:$PATH in /etc/profile so that it uses Simon's openssl.

hpages commented 9 months ago

/opt/R/x86_64/bin or /opt/R/arm64/bin contains standalone executables. I'm not sure who/what needs them. AFAIK Rhdf5lib needs to be linked to some of the libraries provided by Simon's openssl, like libcrypto.a. These are located in /opt/R/x86_64/lib or /opt/R/arm64/lib.

Adding export PATH=/opt/R/x86_64/bin:$PATH or export PATH=/opt/R/arm64/bin:$PATH in /etc/profile has the potential to mask other executables with consequences that are hard to predict, so I would recommend holding off until we have gathered evidences that some packages actually need that.

jwokaty commented 8 months ago

rawrr and ChemmieOB are building fine (except for ChemmieOB on Mac ARM64, which fail for another reason). I'm adding a note in the documentation that we should skip installing the mysql client on macOSX since ensembl-VEP will be replaced: https://github.com/Bioconductor/BBS/pull/386.