Linking to the `udunits` system lib fails on macOS Catalina

[pat-s]

(This might be an OS version specific issue)

Lately I am getting error linking to the udunits system installation when installing from source (also applies to the udunits2 package).

The same applies to CI builds running on macOS Catalina (10.15.x). I cannot tell the exact time when this started but it used to still work around one week ago.

I am aware that I can set --configure-args= etc. but auto-detection worked before and is very much needed for CI builds.

udunits was installed and linked via brew install udunits and brew link udunits.

[Enchufa2]

I'm not able to find it in the logs... is macOS-latest on GH Catalina? Because here it's working, right? And I don't see any --configure-args being applied in the workflow file. Here we get an error, but it comes from installing udunits2 from source, not from units, and the autodetection in udunits2 configure script has been always flawed.

[pat-s]

is macOS-latest on GH Catalina?

Yes.

R-release uses the macOS (el-capitan) binaries so the source installation issue does not come up there. See the first run where binaries were used. This of course cannot happen on R-devel. However, the issue persists for both (tested this locally). Btw the "using binaries" thing is why you normally do not get bug reports from users (that quickly) since they tend to "just work" and not many devs go the "source install" way.

but it comes from installing udunits2 from source, not from units, and the autodetection in udunits2 configure script has been always flawed.

{units} is installed locally since its the package to be checked and not as a dep via remotes::install_deps(). That's why you only see the issue for {udunits2}.

[Enchufa2]

Btw the "using binaries" thing is why you normally do not get bug reports from users (that quickly) since they tend to "just work" and not many devs go the "source install" way.

Correct me if I'm wrong, but usually issues arise when people source-install without the toolchain recommended by CRAN. It's nice to have CI with another toolchain, but then we may fail with the official one that in turn is required for getting the package on CRAN. So as a result, things get overcomplicated for macOS.

{units} is installed locally since its the package to be checked and not as a dep via remotes::install_deps(). That's why you only see the issue for {udunits2}.

Not sure if I'm following you here. So the source installation via R CMD install works, but the source installation via remotes fails? Maybe the issue has something to do with remotes?

[pat-s]

Correct me if I'm wrong, but usually issues arise when people source-install without the toolchain recommended by CRAN. It's nice to have CI with another toolchain, but then we may fail with the official one that in turn is required for getting the package on CRAN. So as a result, things get overcomplicated for macOS.

Yes. And the macOS toolchain is not transparent and easy. However, things will be come a bit better for R 4.0. Note that {tic} mimics the CRAN toolchain while r-lib/actions does not.

Not sure if I'm following you here. So the source installation via R CMD install works, but the source installation via remotes fails? Maybe the issue has something to do with remotes?

Ah no sorry, I was on the wrong page there. I would expect it to also error during R CMD build.

Interestingly, source install works before covr is run: https://github.com/pat-s/units/runs/567172009?check_suite_focus=true#step:17:55 This really gives me headaches since both runs (devel and release) do the same thing beforehand: brew install udunits.

But: On todays most recent run we're running into another Rcpp SDK10.15 issue most likely: https://github.com/pat-s/units/runs/569670794?check_suite_focus=true#step:15:281 I need to update the R-devel toolchain to finally use the 10.13 SDK. This might take a few days...

Nevertheless and coming back to the linking issue which also exists for me for R-release, I am unable to link against udunits locally. I am almost out of ideas right now.

[pat-s]

The following works

 install.packages("units", configure.args = c("--with-udunits2-include=/usr/local/Cellar/udunits/2.2.27.6/include/", "--with-udunits2-lib=/usr/local/Cellar/udunits/2.2.27.6/lib"))

but this is not a longterm solution. Something must have changed in an update of udunits or maybe pkg-config recently preventing the auto-detection.

[pat-s]

The udunits bottle I have installed cannot be the reasons since its from 2019:

brew info udunits                                                                                                                                                                                                                                                                                                                                                                                                                                                  (6m 37s 873ms)
udunits: stable 2.2.27.6 (bottled)
Unidata unit conversion library
https://www.unidata.ucar.edu/software/udunits/
/usr/local/Cellar/udunits/2.2.27.6 (22 files, 544.8KB)
  Poured from bottle on 2019-08-07 at 15:45:20

On the other hand I have no idea if pkg-config is even involved here. Right now I am out of ideas.

[Enchufa2]

Could it be an issue with security and permissions?

[Enchufa2]

BTW, could you please paste here the output from R CMD config --all for a working installation and a failing one?

[pat-s]

Could it be an issue with security and permissions?

I'm on Catalina since over one month and the issues I am seeing started within the last week. However, I cannot exclude it completely since it might be that there were just no updates/installs of the affected packages during the first weeks on Catalina. However, CI also fails just lately so I suspect that something changed within the last week.

BTW, could you please paste here the output from R CMD config --all for a working installation and a failing one?

There is no working config right now. A successful installation only happens with a manual specification of configure.args as shown above.

Details

R CMD config --all (14s 631ms) CC = ccache /usr/local/opt/llvm/bin/clang CFLAGS = -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include -L/usr/local/opt/libomp/lib -lomp CPICFLAGS = -fPIC CPPFLAGS = -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include -L/usr/local/opt/libomp/lib -lomp CXX = ccache /usr/local/opt/llvm/bin/clang++ CXXFLAGS = -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk -Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include -L/usr/local/opt/libomp/lib -lomp CXXPICFLAGS = -fPIC CXX11 = ccache /usr/local/opt/llvm/bin/clang++ CXX11STD = -std=gnu++11 CXX11FLAGS = -Wall -g -O2 CXX11PICFLAGS = -fPIC CXX14 = ccache /usr/local/opt/llvm/bin/clang++ CXX14STD = -std=gnu++14 CXX14FLAGS = -Wall -g -O2 CXX14PICFLAGS = -fPIC CXX17 = clang++ -mmacosx-version-min=10.13 CXX17STD = -std=gnu++17 CXX17FLAGS = -Wall -g -O2 CXX17PICFLAGS = -fPIC CXX20 = clang++ -mmacosx-version-min=10.13 CXX20STD = -std=gnu++2a CXX20FLAGS = -Wall -g -O2 CXX20PICFLAGS = -fPIC DYLIB_EXT = .dylib DYLIB_LD = ccache /usr/local/opt/llvm/bin/clang DYLIB_LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress FC = ccache /usr/local/gfortran/bin/gfortran FFLAGS = -Wall -g -O2 FPICFLAGS = -fPIC FLIBS = -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm FCFLAGS = -Wall -g -O2 SAFE_FFLAGS = -Wall -g -O2 -msse2 -mfpmath=sse OBJC = clang -mmacosx-version-min=10.13 OBJCFLAGS = -Wall -g -O2 -fobjc-exceptions JAVA = /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/bin/java JAVAC = /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/bin/javac JAVAH = /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/bin/javah JAR = /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/bin/jar JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre JAVA_LIBS = -L/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/server -ljvm JAVA_CPPFLAGS = -I/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/../include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/../include/darwin LDFLAGS = -L/usr/local/lib SHLIB_CFLAGS = SHLIB_CXXFLAGS = SHLIB_CXXLD = ccache /usr/local/opt/llvm/bin/clang++ SHLIB_CXXLDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress SHLIB_CXX11LD = ccache /usr/local/opt/llvm/bin/clang++ -std=gnu++11 SHLIB_CXX11LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress SHLIB_CXX14LD = ccache /usr/local/opt/llvm/bin/clang++ -std=gnu++14 SHLIB_CXX14LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress SHLIB_CXX17LD = clang++ -mmacosx-version-min=10.13 -std=gnu++17 SHLIB_CXX17LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress SHLIB_CXX20LD = clang++ -mmacosx-version-min=10.13 -std=gnu++2a SHLIB_CXX20LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress SHLIB_EXT = .so SHLIB_FFLAGS = SHLIB_LD = ccache /usr/local/opt/llvm/bin/clang SHLIB_LDFLAGS = -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress TCLTK_CPPFLAGS = -I/usr/local/include -I/usr/local/include -I/opt/X11/include TCLTK_LIBS = -L/usr/local/lib -ltcl8.6 -L/usr/local/lib -ltk8.6 -L/opt/X11/lib -lX11 -Wl,-weak-lXss -lXext BLAS_LIBS = -L/Library/Frameworks/R.framework/Resources/lib -lRblas LAPACK_LIBS = -L/Library/Frameworks/R.framework/Resources/lib -lRlapack MAKE = make LIBnn = lib AR = ar RANLIB = ranlib ## The following variables are deprecated CPP = ccache /usr/local/opt/llvm/bin/clang -E CXXCPP = ccache /usr/local/opt/llvm/bin/clang++ -E

[Enchufa2]

Correct me if I'm wrong, but I think that brew installs stuff into Cellar and then creates symlinks, right? If that's the case, maybe the symlinks are missing or they are not being followed due to some new security restriction?

[pat-s]

Installs into /usr/local/Cellar and then symlinks, depending if the formula is being linked or how it is discovered in general.

I just saw https://github.com/r-lib/xml2/pull/297. Maybe this also applies to udunits? Haven't looked into it in detail yet.

[Enchufa2]

Nope, we don't use pkg-config. I think this may be the issue, listed as a "new feature" (ha!).

[Enchufa2]

So we need a reliable way to tell autoconf where the SDK is. Can you test this, please?

[kevinushey]

FWIW a 'cheap' way to accomplish this is to set the SDKROOT environment variable; e.g. I have in my .Renviron:

SDKROOT = /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

[Enchufa2]

Some report that reinstalling the library (in this case, udunits) solved the problem. I don't know, I don't understand the mess that Apple made. This is what we use to find the library:

AC_CHECK_LIB(udunits2, ut_read_xml, 
             LIBS="${LIBS} -ludunits2",
             UD_ERROR="libudunits2.so was not found")

which is independent of the location of the library. It hasn't changed and it should still work. If this doesn't work, it's because the compiler itself is not able to find the library, which is a misconfiguration.

[pat-s]

Even with SDKROOT set I am unable to link it.

Tried with both R-devel and R-release with Apples default clang and the custom clang7 from CRAN.

Also tried with the latest SDK and the 10.13 SDK via -isysroot.

Can you test this, please?

This is my default, pointing to SDK 10.13 lately because 10.15 is having major issues with Rcpp. However, both do not work here (leaving it out completely does also not help).

Maybe there is a need for pkg-config? Seems like the xml2 binding is found with it "easily".

[Enchufa2]

Did you try reinstalling udunits?

Maybe there is a need for pkg-config? Seems like the xml2 binding is found with it "easily".

Nope, upstreams need to provide pkg-config files for that to work, and udunits does not provide those.

[pat-s]

Did you try reinstalling udunits?

Yes reinstalling and relinking.

Nope, upstreams need to provide pkg-config files for that to work, and udunits does not provide those.

OK I see - I do not have any exp with all that linking stuff so far.

[Enchufa2]

All this is just a question of paths. What autoconf does is to try to compile a small useless program that includes a function of the library. If this program compiles ok, it means that the library is present and reachable. The failure means that either the library is not installed (which is not the case) or the paths are not properly set up.

Are you subscribed to r-sig-mac? It may be a good place to ask for help.

[pat-s]

Are you subscribed to r-sig-mac? It may be a good place to ask for help.

Yeah I think it might be time to continue there. Maybe this issue is more of a general one that might affect even more packages regarding linking.

Thanks for taking the time so far!

[pat-s]

Got it! One needs to add -I/usr/local/include to CPPFLAGS. While this works for {units}, I still get the following for {udunits}

checking udunits2.h usability... yes
checking udunits2.h presence... yes
checking for udunits2.h... yes
checking for ut_read_xml in -ludunits2... no

But this might be a problem of the latter. In summary I am not sure if the issues related to the macOS update. I had this -I in my flags before but after changing those for R 4.0 something might have gone lost.

Any thought about the ut_read_xml issue would be highly appreciated :) Try to find a way to fix this though I have no real idea what I'm doing here.

[kevinushey]

FWIW the default CPPFLAGS in CRAN R's own Makevars has:

CPPFLAGS = -I/usr/local/include

So R does normally assume and try to use headers available there.

[Enchufa2]

Got it! One needs to add -I/usr/local/include to CPPFLAGS.

Great! Good to know. This path is a non-standard one, and that's probably why CRAN R's Makevars includes it, as Kevin points out.

While this works for {units}, I still get the following for {udunits}

Yes, as I said in a previous comment, udunits' configure script has been always fundamentally broken for some systems, including Linux systems. But since v0.6-0, {units} directly links against UDUNITS-2 and does not depend on {udunits} anymore.