search

r-spatial / sf

Simple Features for R
https://r-spatial.github.io/sf/
Other
1.34k stars 297 forks source link

Installing sf in R with Apple Silicon Mac #1621

Closed tony352 closed 3 years ago

tony352 commented 3 years ago

Hi, I have a MacBook Pro with the Apple Silicon (M1) chip, and have installed a development version of R and R Studio for ARM64.

I would like to install the sf package but am having trouble getting it to compile. I have installed GDAL, Poppler and libdap onto my Mac via Homebrew. I first got an error about being unable to find the goal-config file, so I added the path as an argument in the install.packages command.
This worked, but I now get another error that I can't make sense of: clang: error: linker command failed with exit code 1 (use -v to see invocation)

Just before the error, I get the warning below which caught my attention--from what I can make out, some of the code is attempting to compile using Intel x86_64, rather than arm64, which I don't want.

ld: warning: ignoring file /opt/homebrew/Cellar/gdal/3.2.1_3/lib/libgdal.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

Undefined symbols for architecture x86_64: "_GDALAllRegister", referenced from: _main in gdal_test-281afe.o ld: symbol(s) not found for architecture x86_64

I also get warnings saying that it's looking for the poppler and libdap directories--both are installed, but in different folders to where it's searching. They are in the folder below: /opt/homebrew/opt/

The full log I get when compiling sf is below. Any guidance that can be provided would be most appreciated!

Tony

` install.packages("sf", configure.args = "--with-gdal-config=/opt/homebrew/Cellar/gdal/3.2.1_3/bin/gdal-config") Installing package into ‘/Users/anthonyperry/Library/R/arm64/4.1/library’ (as ‘lib’ is unspecified) Package which is only available in source form, and may need compilation of C/C++/Fortran: ‘sf’ Do you want to attempt to install these from sources? (Yes/no/cancel) Yes installing the source package ‘sf’

trying URL 'https://cran.rstudio.com/src/contrib/sf_0.9-7.tar.gz' Content type 'application/x-gzip' length 4203095 bytes (4.0 MB)

downloaded 4.0 MB

rsbivand commented 3 years ago

The message:

ld: warning: ignoring file /opt/homebrew/Cellar/gdal/3.2.1_3/lib/libgdal.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

seems rather plain; perhaps waiting a month or two (or three) makes more sense, or simply use rosetta which works (tried) with CRAN binary packages.

tony352 commented 3 years ago

Hi Roger, I'm still learning (and on a steep learning curve!). I'm of course happy to wait but was just trying out the new RStudio for ARM64 Macs. (It's worked wonders already for a project where the speed really does matter).

Can you translate what the message is saying? Which file is it saying is built for macOS-x86_64? I was compiling it on an ARM64 Mac--is it something in my compiler that's set to compile for x86_64? Or is it related to a package itself which will be under further development?

Thanks in advance for any clarification you can provide!

Tony

On Fri, 5 Mar 2021 at 19:32, Roger Bivand notifications@github.com wrote:

The message:

ld: warning: ignoring file /opt/homebrew/Cellar/gdal/3.2.1_3/lib/libgdal.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

seems rather plain; perhaps waiting a month or two (or three) makes more sense, or simply use rosetta which works (tried) with CRAN binary packages.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r-spatial/sf/issues/1621#issuecomment-791633898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQRDRK4HMEX7UUVQOM365LDTCEWUZANCNFSM4YVREMFQ .

rsbivand commented 3 years ago

No idea. ld says that GDAL is built for x86_64, so cannot be linked with arm64, so your upstream GDAL installation does not match what you are doing.

tony352 commented 3 years ago

Hi Roger, that's what I thought but wasn't sure. I'll try to raise a request with GDAL to see if there's anything on their end that they can advise. Thank you!

On Sat, 6 Mar 2021 at 10:08, Roger Bivand notifications@github.com wrote:

No idea. ld says that GDAL is built for x86_64, so cannot be linked with arm64, so your upstream GDAL installation does not match what you are doing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r-spatial/sf/issues/1621#issuecomment-791906543, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQRDRK3GGFSDIZLPZB3REV3TCH5KVANCNFSM4YVREMFQ .

rsbivand commented 3 years ago

I suggest not asking GDAL, but rather installing the other software GDAL needs (GEOS, PROJ) and for drivers you need first, then installing GDAL (maybe from HEAD source or 3.2.2RC1 released yesterday, but I don't see any reference in NEWS to arm64). M1 is so fresh that most larger projects are unprepared. Please keep this issue updated as you try alternatives, as others may benefit from dead-ends you find as well as successes.

jebyrnes commented 3 years ago

I wanted to check in on this. I'm running from the arm64 nightly builds, and have installed gdal via homebrew - which installs an arm64 compiled version.

And yet

ld: warning: ignoring file /usr/local/Cellar/gdal/3.2.2/lib/libgdal.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture arm64:

So - it's ignoring the proper file? I'm a bit confused here, and wonder if there is a config option or something to put into Makevars to fix this.

> sessionInfo()
R version 4.1.0 alpha (2021-04-26 r80229)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Big Sur 11.2.1

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.0 tools_4.1.0    parallel_4.1.0
edzer commented 3 years ago

https://twitter.com/RogerBivand/status/1397157063676022787

lionel- commented 2 years ago

@edzer Should we interpret this link to Roger's tweet as meaning that the configure script in sf doesn't support linking to an arm64 gdal lib installed via homebrew? And instead we should install Simon's versions?

edzer commented 2 years ago

I would hope these scripts do supports linking to properly installed homebrew libraries, but I have very limited means to check or debug (but Roger now has a mac). Many R + macos users run into problems they don't understand and we cannot solve, we advice them to use binary installs. I suspect that the large majority of R + macos users don't know what "linking" actually means, don't know the difference between static and dynamic linking, and for such users the binary installs (not using homebrew) have proven to be more robust.

rsbivand commented 2 years ago

Yes, this is the conclusion we reached looking at user experiences. Jakub @Nowosad and I are revising the spatial task view (still private repo), our draft text for installation is (comments very welcome):

Installation of packages like r pkg("sf") and r pkg("terra") which use external software libraries such as PROJ, GDAL or GEOS requires care. For most users on platforms such as Windows or macOS who are not themselves package developers, it is always better to avoid what are known as source installs, because CRAN binary packages include all of the external software required. Because getOption("pkgType") on these platforms is usually "both", you may be asked to choose to install a source package if it is more recent than the latest binary.

Please do not be tempted to choose a source install for r pkg("sf") or r pkg("terra") or similar; the binary package will be generated within a day or two. To avoid being asked, you may see from ?options under options provided by the utils package that the default behaviour of your installation of R may be controlled by setting options install.packages.check.source and install.packages.compile.from.source, or by setting environment variable R_COMPILE_AND_INSTALL_PACKAGES, see also this helpful comment.

If you are a developer using Windows or macOS or installing from github, the same static-linked binary external software libraries, header files, etc. as those used in building CRAN binary packages are available from: Windows 4.0 and 4.1 downloaded on-the-fly, Windows 4.2 forthcoming rtools42 and macOS both architectures. These external software libraries have been built using the same compile and link settings as R itself, so avoid the risk of possible errors caused by mismatched binaries.

If you are a user (or developer) on systems where getOption("pkgType") is "source", you will need to ensure that the external software is available when installing source packages. Advice for some such systems may be found here. The most common reason for failure is having multiple versions of external software installed on your platform.

On that basis, the text in sf/README.md might be simplified by using similar language.

lionel- commented 2 years ago

Here are some debugging notes for those trying to compile with the homebrew libraries. I was very confused to see this in config.log:

## --------- ##
## Platform. ##
## --------- ##

uname -m = x86_64

Yet running in the terminal I see:

> uname -m
arm64

The reason is that uname running under Rosetta returns x86_64. And indeed the version of R I'm currently using is compiled for x86.

R version 4.1.2 (2021-11-01) -- "Bird Hippie"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

And within R:

system("uname -m")
#> x86_64

To make sure the correct homebrew distribution is exported within R (arm64 is in /opt/homebrew, x86 is in /usr/local/homebrew), I added this to my Rprofile:

# Make sure we are not using arm64 distribution of Homebrew if R is
# running under Rosetta
if (identical(Sys.info()[["machine"]], "x86_64")) {
  local({
    path <- sub(":/opt/homebrew/bin", ":/usr/local/homebrew/bin", Sys.getenv("PATH"))
    Sys.setenv(PATH = path)
  })
}

Then gdal needs to be installed in the x86 distribution.

aourednik commented 2 years ago

Even simpler: directly install the binry gently provided by the developpers:

Run:

install.packages("https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.1/sf_1.0-5.tgz", repos = NULL, type = .Platform$pkgType) instead of install.packages("sf")

instead of

install.packages("sf")

I've filed a feature proposal to RStudio to make this a default behaviour of package manager: https://github.com/rstudio/rstudio/issues/10360

rsbivand commented 2 years ago

Seems sensible, but you would need to check the current version number of R and the sf package, I think, perhaps:

paste0("https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/", R.version$major, ".", substring(R.version$minor, 1, 1), "/sf_", available.packages()["sf","Version"], ".tgz")

which would fail when used if the binary had not yet been built from a recent source release.