EpiNow2 compilation errors on M1 Mac

tony352 commented 3 years ago

Hi,

I've tried compiling EpiNow2 using the latest build of the ARM64 version of R (v4.1.0 2021/02/18, r80027) using a MacBook Pro with Apple Silicon. Unfortunately, the build failed to compile and the library didn't load.

I am using XCode Beta (v12.5) and the latest version of the command line tools (version 12.4.0.0.1.1610135815). It might be an issue on what I've compiled, or perhaps something with the Command Line Tools?

It isn't an urgent issue from my point of view, as I'm able to use the 'normal' EpiNow2 on the non-development version of R. However, it may become more pressing over time as R launches its first public (non-developer) version for Silicon Macs and more people try to use it. I imagine the code will be much faster on Silicon Macs as well, allowing people to do more with the EpiNow2 package.

I've attached a copy of the log for reference.

jhellewell14 commented 3 years ago

Hello,

I'm no expert but from reading through the log you provided it looks like the EpiNow2 build is failing because dependencies are failing to install.

For package V8:

Configuration failed to find the libv8 engine library. Try installing:
 * deb: libv8-dev or libnode-dev (Debian / Ubuntu)
 * rpm: v8-devel (Fedora, EPEL)
 * brew: v8 (OSX)
 * csw: libv8_dev (Solaris)

So I think on OSX you need to run brew install v8 in a terminal (assuming you have homebrew installed).

Then the installation for the package RcppEigen seems to be failing because you do not have a fortran compiler. I don't know the specifics of Apple Silicon but people seem to be discussing the options for fortran compilers here.

Hope this helps. Let me know where you get to.

Cheers, Joel

tony352 commented 3 years ago

Thank you Joel!

I'll take a closer look and see what happens! I thought that the development version had GNU Fortran with it but will also double-check that.

Thanks!

Tony

On Tue, 23 Feb 2021 at 15:35, Joel Hellewell notifications@github.com wrote:

Hello,

I'm no expert but from reading through the log you provided it looks like the EpiNow2 build is failing because dependencies are failing to install.

For package V8:

Configuration failed to find the libv8 engine library. Try installing:

deb: libv8-dev or libnode-dev (Debian / Ubuntu)

rpm: v8-devel (Fedora, EPEL)

brew: v8 (OSX)

csw: libv8_dev (Solaris)

So I think on OSX you need to run brew install v8 in a terminal (assuming you have homebrew installed).

Then the installation for the package RcppEigen seems to be failing because you do not have a fortran compiler. I don't know the specifics of Apple Silicon but people seem to be discussing the options for fortran compilers here https://developer.apple.com/forums/thread/651476.

Hope this helps. Let me know where you get to.

Cheers, Joel

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/epiforecasts/EpiNow2/issues/232#issuecomment-784289208, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQRDRK5SW2MRDQF2GQVTFE3TAPDNVANCNFSM4YCSEB4A .

tony352 commented 3 years ago

I've installed the V8 package via brew and also the GFortran for ARM installer you noted.

I tried to install one of the dependency packages ("RcppParallel") and still got several errors--though none look related to the Fortran compiler.

I tried using brew to install the packages below, in case that might work, but brew could only find the V8 package.

‘RcppParallel’, ‘V8’, ‘rstan’, ‘RcppEigen’, ‘StanHeaders’, ‘EpiNow2’

I've attached the log for RcppParallel in case this is helpful. Is there anything else I should try?

Thanks!

Tony

Log for installing RcppParallel (dependency for EpiNow2).docx

medewitt commented 3 years ago

~~Looks like there is an open issue for RcppParallel on M1/ARM archis, so there might not be a resolution yet.~~

Issue: https://github.com/RcppCore/RcppParallel/issues/137

Shame on me for not reading rest of comments...maybe try pulling the development branch?

install.packages("remotes")
remotes::install_github("RcppCore/RcppParallel@develop", force = TRUE)

Without that dependency the other packages won't build.

tony352 commented 3 years ago

Ah, I missed other comments as well--thank you! The "remotes" packages installed but GitHub couldn't find the development version of RcppParallel. I'll take a closer look at the other comments in case I've missed anything there.

remotes::install_github("RcppCore/RcppParallel@develop", force = TRUE) Error: Failed to install 'unknown package' from GitHub: HTTP error 404. No commit found for the ref develop

Did you spell the repo owner (RcppCore) and repo name (RcppParallel) correctly?

If spelling is correct, check that you have the required permissions to access the repo.

install_github function (repo, ref = "HEAD", subdir = NULL, auth_token = github_pat(quiet), host = "api.github.com", dependencies = NA, upgrade = c("default", "ask", "always", "never"), force = FALSE, quiet = FALSE, build = TRUE, build_opts = c("--no-resave-data", "--no-manual", "--no-build-vignettes"), build_manual = FALSE, build_vignettes = FALSE, repos = getOption("repos"), type = getOption("pkgType"), ...) { remotes <- lapply(repo, github_remote, ref = ref, subdir = subdir, auth_token = auth_token, host = host) install_remotes(remotes, auth_token = auth_token, host = host, dependencies = dependencies, upgrade = upgrade, force = force, quiet = quiet, build = build, build_opts = build_opts, build_manual = build_manual, build_vignettes = build_vignettes, repos = repos, type = type, ...) } <bytecode: 0x13fb95a18>

seabbs commented 3 years ago

Hi Tony,

In general, our major downstream dependency that will cause you issues is rstan. rstan itself has some really great installation documentation so it could be they have updated that for the new Apple silicon.

I just had a quick look around the internet and found this thread but it looks like people largely having problems and so not much help: https://discourse.mc-stan.org/t/anyone-planning-on-getting-an-m1-machine-to-benchmark/19344/15

Sam

seabbs commented 3 years ago

For the RcppParallel issue it looks like that fix has been merged in (https://github.com/RcppCore/RcppParallel/pull/148) and so installing from Github (remotes::install_github("RcppCore/RcppParallel")) will maybe do it?

tony352 commented 3 years ago

That link for RcppParallel worked! :). It installed and loaded.

I'll now try installing the other dependencies, one at a time (noting the limitations we might see with rstan).

Thank you! I can't wait to put EpiNow2 through its paces :).

tony352 commented 3 years ago

I just tried RcppEigen and it looks like it also isn't set up yet to compile on ARM-based Macs. I tried looking through its Github page but may have missed it (learning as I go though!). That said, there is an ARM64 version for Debian, so maybe there's hope? https://ubuntu.pkgs.org/16.04/ubuntu-universe-arm64/r-cran-rcppeigen_0.3.2.7.0-1_arm64.deb.html

RcppEigen log M1.docx

medewitt commented 3 years ago

You could likely build the package in a Docker container (along with the rest of the dependencies).

See https://docs.docker.com/docker-for-mac/apple-m1/ for Docker information.

And then https://github.com/epiforecasts/EpiNow2/wiki/Docker for instructions on running the package on Docker. Using docker you could mount your local volumes to the container in order to share data from the container to your machine (and back and forth).

tony352 commented 3 years ago

I hadn't thought of Docker and now wish I did--it is on my main system now (though using RSwitch to keep the developmental version separate). I'm not sure how Docker will enable these to compile if it can't on the main system, but I am still learning in this area.

I have raised an Issue log on the RcppEigen Github page so I hope that helps as well. I saw that RcppParallel is also produced by the same group so I'm hopeful. I'll keep all posted!

https://github.com/RcppCore/RcppEigen/issues/96

tony352 commented 3 years ago

Hi All, I am pleased to write with good news! I got EpiNow2 to run on my M1 Mac :).

It's not straightforward, as this log shows, but here's basically how I got there:

1) Install GFortran for Silicon (latest version below, as cited by Joel) https://github.com/fxcoudert/gfortran-for-macOS/releases/tag/11-arm-alpha2

2) Install V8 (which I did via brew as suggested above).

3) Install the RcppParallel package in R, which I did as suggested above:

remotes::install_github("RcppCore/RcppParallel")

4) Install RcppEigen in R. This was the trickiest part, and I found that the main stumbling block was that the GFortran app installs in a different path to where the RcppEigen installer expects. So, that installation kept falling over. I took the really crude approach of copying the folder contents for gfortran into where RcppEigen expected--and the installer worked! The library loads without problem. More details on this can be found here: https://github.com/RcppCore/RcppEigen/issues/96

5) Install the remaining dependencies: rstan and StanHeaders These installed successfully but with warnings. The libraries installed and loaded.

6) Install EpiNow2 as a package. This also installed with warnings but loaded successfully. I've attached a log for your reference (apols that it's a Word file--but I wanted to ensure I could send everything here in case needed). The library loaded

7) Moment of truth--run EpiNow2! I had run the query using the terminal, as I couldn't get the R GUI to load when using the development version (it quickly crashes). I pasted a query that I normally use for the Chickasaw Nation into the terminal and it is running. I've attached an overall Rt chart (starting 1 Nov) and county-level estimates (13 US counties).

I didn't time it closely, but the conservative estimate I'd give is 45 minutes to run compared to 90 minutes when running R in Rosetta2 (Intel emulation). There might be other differences, such as running in R GUI (or RStudio when that's available)--but the speed is absolutely noticeable :).

Thank you all for your help! Please let me know if there's anything more I can do as well.

Chickasaw Nation Rt Plot.pdf County-level Rt estimates (Chickasaw Nation).pdf

EpiNow2 installation.docx

seabbs commented 3 years ago

Great news - honestly I am quite surprised this worked. Awesome speed up as well!

Estimates look sensible to me.

We are looking to release a new version in the near future (https://github.com/epiforecasts/EpiNow2/issues/234) I would be very interested to hear any feedback you might have on issues you have run into.

Closing this as looks good to me. Perhaps we should add a link from the install instruction for new Mac users?

Sam

epiforecasts / EpiNow2

EpiNow2 compilation errors on M1 Mac #232