Installation from renv.lock and scratch failed due to OS specific dependencies, linux

[SarahAlidoost]

Besides the fact that installation from both renv.lock and scratch is slow; taking about an hour, it seems that the R packages have OS-specific dependencies, see below. After the error from 'sf' (the last one), I quit the installation from renv.lock file and tried installing from scratch. For that, the readme should mention that the renv.lock file should be removed.

To speed up the installation process, could we group the dependencies based on individual notebooks? This way, we only install the packages required for each specific notebook.

- Errors from renv.lock:

Error: Error installing package 'protolite':
=====================================

* installing *source* package ‘protolite’ ...
** package ‘protolite’ successfully unpacked and MD5 sums checked
....
Failed to run protoc.
Please install the 'libprotobuf-dev' package for your system.

Error: Error installing package 'protolite':
=====================================

* installing *source* package ‘protolite’ ...
** package ‘protolite’ successfully unpacked and MD5 sums checked
....
Failed to run protoc.
Please install the 'protobuf-compiler' package for your system.

- Installing jqr ...                            FAILED
Error: Error installing package 'jqr':
===============================

* installing *source* package ‘jqr’ ...
** package ‘jqr’ successfully unpacked and MD5 sums checked
** using staged installation
Using PKG_CFLAGS=
Using PKG_LIBS=-ljq
--------------------------- [ANTICONF] --------------------------------
Configuration failed because libjq was not found. Try installing:
 * deb: libjq-dev (Debian, Ubuntu).
 * rpm: jq-devel (Fedora, EPEL)
 * csw: libjq_dev (Solaris)
 * brew: jq (OSX)

configure: error: in `/tmp/RtmpiHZIxI/R.INSTALL143c06f73f209/units':
configure: error: 
--------------------------------------------------------------------------------
  Configuration failed because libudunits2.so was not found. Try installing:
    * deb: libudunits2-dev (Debian, Ubuntu, ...)
    * rpm: udunits2-devel (Fedora, EPEL, ...)
    * brew: udunits (OSX)

Error: Error installing package 'sf':
==============================

* installing *source* package ‘sf’ ...
** package ‘sf’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++17
checking for gdal-config... no
no
configure: error: gdal-config not found or not executable.
ERROR: configuration failed for package ‘sf’
* removing ‘/home/sarah/GitHub/prototyping/renv/staging/1/sf’
install of package 'sf' failed [error code 1]

- Errors from scratch:

Error: Error installing package 'textshaping':
=======================================

* installing *source* package ‘textshaping’ ...
** package ‘textshaping’ successfully unpacked and MD5 sums checked
** using staged installation
Package harfbuzz was not found in the pkg-config search path.
Perhaps you should add the directory containing `harfbuzz.pc'
to the PKG_CONFIG_PATH environment variable
No package 'harfbuzz' found
Package fribidi was not found in the pkg-config search path.
Perhaps you should add the directory containing `fribidi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fribidi' found
Using PKG_CFLAGS=
Using PKG_LIBS=-lfreetype -lharfbuzz -lfribidi -lpng
--------------------------- [ANTICONF] --------------------------------
Configuration failed to find the harfbuzz freetype2 fribidi library. Try installing:
 * deb: libharfbuzz-dev libfribidi-dev (Debian, Ubuntu, etc)
 * rpm: harfbuzz-devel fribidi-devel (Fedora, EPEL)
 * csw: libharfbuzz_dev libfribidi_dev (Solaris)
 * brew: harfbuzz fribidi (OSX)
If harfbuzz freetype2 fribidi is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a harfbuzz freetype2 fribidi.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
-------------------------- [ERROR MESSAGE] ---------------------------
<stdin>:1:10: fatal error: hb-ft.h: No such file or directory
compilation terminated.

[vanlankveldthijs]

I've been installing fully inside conda and running into similar issues, but not all.

conda install r installed r-base-4.4.1

After this, I've had to manually install the following packages (for far): conda install protobuf conda install jq conda install udunits2 conda install gdal conda install pkg-config

Rscript -e "renv::restore()" was reasonably quick to continue the installation, but does take a long time to run thought the whole process.

[fnattino]

@SarahAlidoost the reason why this is taking so long could be that your system is building the packages from source instead of downloading available binaries. You are on Ubuntu right? Maybe there are other mirrors with more binaries available for linux? I will have a look. Maybe conda can indeed help for this, but I am not sure if all needed packages are available (have you been also using the r channel @vanlankveldthijs ?).

Will mention that renv.lock should be removed in #10, thanks for the tip !

[vanlankveldthijs]

Maybe conda can indeed help for this, but I am not sure if all needed packages are available (have you been also using the r channel @vanlankveldthijs ?).

I'm using the base channel of miniforge, which is conda-forge.

[fnattino]

I'm using the base channel of miniforge, which is conda-forge.

Aah, sorry, now I got it - so you have installed R and some of the required (system) libraries with conda, but still installing R packages via renv? I guess that this will still be building packages from source then. A faster way could instead also install R packages via conda: conda install r-dbscan r-geojsonio r-leaflet ... with all the dependencies from the DESCRIPTION file (note that R packages in conda all start with r-), maybe trying to add the r channel. If this works, we could also add a corresponding environment file to the repo..

[vanlankveldthijs]

@fnattino is there an easy way to reverse the Rscript installation process?

I wanted to check whether building from source worked after creating a new conda environment and installing conda install r protobuf jq udunits2 gdal pkg-config Unfortunately, this does not affect the Rscript cache, so it will not rebuild the packages.

I know that it may be a lot faster to install any existing binaries, but as long as the readme describes setting up the repo by running Rscript, it should at least also mention the required dependencies.

[fnattino]

I think the following should bypass the cache (at least according to the docs):

Rscript -e "renv::restore(rebuild = TRUE)"

[fnattino]

Just as a reference, I have put together the following environment file, that seems to work to get all (R and Python) packages installed in the conda environment:

name: crisp
channels:
 - conda-forge
 - r
dependencies:
 # Python dependencies
 - python=3.10
 - folium
 - geopandas
 - jupyterlab
 - mapclassify
 - matplotlib
 - momepy
 - numpy<2
 - osmnx
 - pandas
 - scikit-learn
 - shapely
 # R dependencies
 - r
 - r-dbscan
 - r-geojsonio
 - r-leaflet
 - r-lwgeom
 - r-osmdata
 - r-rstac
 - r-sfnetworks
 - r-terra
 - r-tidyverse 

Note that one than needs to remove renv folder (and renv.lock file?) before running R - renv otherwise tries to create a new project-based environment (based on the conda environment).

Will add this to the repository with some extra information in the REAMDE as an alternative installation approach.

[vanlankveldthijs]

In case you want to complete remove the renv cache, the following may help:

"To completely remove renv from a project, call renv::deactivate(clean = TRUE). If you later want to use renv for this project, you’ll need to start from scratch with renv::init()."