Possible problem with gdal in singularity file

[PabRod]

Problem

(Edit: skip this and see below a clearer explanation of the problem).

When running:

singularity run k.sif \
    Rscript -e "rmarkdown::render('vignettes/make_Landsat_cube.Rmd', params = list(starttime = '${STARTTIME}', endtime = '${ENDTIME}'))"

with the proper values of STARTTIME and ENDTIME, and using k.sif directly downloaded from Spider, I crash into this error:

processing file: make_Landsat_cube.Rmd
  |...                                                                   |   4%
   inline R code fragments

  |.....                                                                 |   7%
label: setup (with options) 
List of 1
 $ include: logi FALSE

Quitting from lines 16-25 (make_Landsat_cube.Rmd) 
Error: package or namespace load failed for 'makeDataCube' in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/pablo/R/x86_64-pc-linux-gnu-library/4.0/rgdal/libs/rgdal.so':
  libgdal.so.20: cannot open shared object file: No such file or directory

Execution halted

Solution

I am still working on it. Would you have time for some troubleshooting today or tomorrow, @wandadk?

[wandadk]

Hi Pablo,

Yes, sure! I have sent you a meeting invite.

Op di 11 mei 2021 om 14:45 schreef Pablo Rodríguez-Sánchez < @.***>:

Assigned #41 https://github.com/RETURN-project/makeDataCube/issues/41 to @wandadk https://github.com/wandadk.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/RETURN-project/makeDataCube/issues/41#event-4727021527, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL7IG4UO3SNQUO55ZUP6OOTTNERE5ANCNFSM44VMEQGQ .

[PabRod]

Clearer explanation of the problem

When we run the code below:

singularity run k.sif R

# And, inside R:
> library(rgdal)

The file k.sif, and also its definition file k.def, are available in project/return/Software/containers.

Although we are using the same k.sif in both cases (copied, not regenerated from definition), different things happen if we are working locally or on Spider.

Locally

I receive the error below. Note that, oddly, R tries to find the package among my local files... as if the container is not isolated.

Loading required package: sp
Error: package or namespace load failed for 'rgdal' in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/pablo/R/x86_64-pc-linux-gnu-library/4.0/rgdal/libs/rgdal.so':
  libgdal.so.20: cannot open shared object file: No such file or directory

On Spider

Everything works fine. It prints the message below:

Loading required package: sp
rgdal: version: 1.5-23, (SVN revision 1121)
Geospatial Data Abstraction Library extensions to R successfully loaded
Loaded GDAL runtime: GDAL 3.0.4, released 2020/01/28
Path to GDAL shared files: /usr/share/gdal
GDAL binary built with GEOS: TRUE 
Loaded PROJ runtime: Rel. 7.0.0, March 1st, 2020, [PJ_VERSION: 700]
Path to PROJ shared files: /home/return-prsanchez/.local/share/proj:/usr/share/proj
PROJ CDN enabled: FALSE
Linking to sp version:1.4-5
To mute warnings of possible GDAL/OSR exportToProj4() degradation,
use options("rgdal_show_exportToProj4_warnings"="none") before loading rgdal.

Any clue about what's going on, @oonkjbr?

This is working on Spider, so the priority of this problem is not very high.

[oonkjbr]

@PabRod I am unfortunately not an R expert. However, it seems as if 'rgdal' either (i) does not exist within the image or (ii) the way in which the package is loaded means that it searches in a very specific location that perhaps coincidentally does exist on Spider (as at some point we installed a lot of R packages). I can confirm that it is not (i) as, At least the R version installed on Spider (4.0.4) is different from the R version in the image (4.0.5).

On Spider I get for the native R: $ R

library(rgdal) Error in library(rgdal) : there is no package called 'rgdal'

On Spider I get for the container image R:

library(rgdal) Loading required package: sp ... (successful)

Hence, it seems that something is going wrong on your local laptop. I wonder if perhaps you have some settings setup that may cause a conflict for R. Very likely there are some paths, env. variables or similar that are already set by you for R and that are the root cause. Here it is important to realize that a container is not completely isolated, in that a file system overlay is created between the container image and the host file systems. Hence, one has to be careful here (host properties are discoverable).

If you like to test with then try e.g., adding to your ~/.bashrc the following line export TEST=42

then perform echo $TEST on your regular shell and also in the singularity shell (i.e., singularity shell k.sif). You find that in both cases it returns 42 eventhough your .bashrc file only exists outside the singularity image. What is known to the singularity image from host depends on a lot of settings and hence I can not give you a full list. However, I can imagine that something like this is causing your local issue. You could consider temporarily using a clean env, alias, path setup on your local machine and see if you then still get the issue.

Hope this helps!

[PabRod]

Thanks a lot for taking a look @oonkjbr!