testing installation and setting up a Docker image

[ErinBecker]

The CAC had an in-depth discussion about the benefits and downsides of providing a Docker image (or some other pre-installed instance, for example an AWS image) for learners at a workshop. As Geospatial software is often quite difficult to install, and has a lot of dependencies, there was agreement that it would be useful to:

• test the installation of all of the software to be used in this workshop on Mac OS, Linux, and Windows
• document the installation clearly in the lessons
• document any potential problems in the installation
• set up a Docker image as a backup for learners who have issues with installation
• include instructions for installing Docker and using the Docker image in the lesson

There is more detail about the CAC's discussions of these issues

here here and here

[chris-prener]

thanks @ErinBecker this all seems good. Do you have tips for Linux? I could set up travis on the repo - that would tell us if everything installs but it wouldn't give us 100% clarity on potential pitfalls.

[tyson-swetnam]

I have used the rocker/geospatial container for GIS applications in R and it seems to work just fine.

I would recommend going with the Rocker group's images as they are maintained by a large community already.

[chris-prener]

hey @tyson-swetnam that is the goal - I am waiting until after the bug bbq to firm up what the image looks like incase we are using any packages that aren't already contained in rocker/geospatial. Just want to make sure that package meets our needs. If it doesn't, I'll build an image on top of it that installs rocker/geospatial plus ours.

[chris-prener]

@ErinBecker - quick question - where do we want to keep documentation about set-up and install for these lessons. In the individual SETUP.md files or on a Wiki for one of the repos? Somewhere else?

[juanfung]

Sorry that I'm a bit late to the party. I was lightly tracking the etherpad, and did not realize related issues had been opened.

I'm using Vagrant to test installations on Ubuntu 14.04 and 16.04, 32 and 64 bit. For instance, the following works on 16.04:

# install udunits, gdal, geos, and proj4, geos
sudo add-apt-repository ppa:ubuntugis/ppa
sudo apt-get update
sudo apt-get install libudunits2-dev libgdal-dev libgeos-dev libproj-dev -y

# install R
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
sudo add-apt-repository 'deb [arch=amd64] https://cran.rstudio.com/bin/linux/ubuntu xenial/'
sudo apt-get update
sudo apt-get install r-base -y

# install RStudio
wget https://download1.rstudio.org/rstudio-1.1.442-amd64.deb
sudo apt-get install gdebi-core libxslt1-dev -y
sudo gdebi -n rstudio-1.1.442-amd64.deb

# installing R packages: sf, raster, rgdal, rasterVis
R -e "install.packages(c('rgdal', 'raster', 'rasterVis', 'sf'))"

It is worth noting that the documentation for sf suggests ubuntugis-unstable, but so far this has not been necessary. I don't know if it matters.

I've only tested the Rstudio installation on 16.04. Based on my scouring of stackoverflow, the approach with gdebi should do the trick in most cases (e.g., here.

My hope is that this works on any Debian-based system, but have not verified! Also have not tested on Red Hat or Arch systems.

So, @ErinBecker what is the best way to proceed?

@chris-prener Are you testing OS X and Windows?

[chris-prener]

hey @juanfung this is awesome! Yes, I've got OS X and Windows, and will synthesize all of the Linux notes I get from folks into a single document

[ErinBecker]

Sorry for the late response @chris-prener. We'd like to have all of the setup info on the workshop homepage: https://github.com/datacarpentry/geospatial-workshop/.

[chris-prener]

No worries I know you've been super busy @ErinBecker - should I create an episode for set-up then?

[ErinBecker]

Maybe useful to look at the Genomics workshop homepage for comparison.

There should be a "Getting started box" in the index.md file in the geospatial workshop homepage repo. In that box, we can include information about pre-reqs as well as a link to the setup instructions (which live in the setup.md file).

[juanfung]

This worked on Fedora 27:

# update package manager
sudo dnf upgrade -y

# install gdal, geos, proj4, udunits
sudo dnf -y install gdal gdal-devel gdal-libs
sudo dnf -y install proj proj-devel proj-epsg proj-nad
sudo dnf -y install geos geos-devel udunits2-devel

# install R
sudo dnf -y install R

# install R packages
mkdir r-lib
export R_LIBS_USER = "~/r-lib"
sudo R -e "install.packages('udunits2', configure.args='--with-udunits2-include=/usr/include/udunits2', repos='https://cloud.r-project.org')"
sudo R -e "install.packages(c('rgdal', 'raster', 'rasterVis', 'sf'), repos='https://cloud.r-project.org')"

Something similar should work on CentOS, preceded by yum install -y epel-release to add the EPEL repo. See this SO answer for an example.

I've tried and failed to install sf on CentOS 7. The problem is that the GDAL version in the EPEL repos is not GDAL (>= 2.0.0) as required. Building GDAL from source seems to work, as in gdal-config --version returns the right version, but for some reason R can't find gdal-config during installation of the packages.

Someone more experienced with CentOS may be able to help...

[chris-prener]

thanks @ErinBecker - sorry I've been offline - had a viral infection that kept me largely sidelined from email and work. I'll get on the SETUP.md file with my notes!

[chris-prener]

hey @ErinBecker - quick update - can you just check my markdown and make sure I am putting this together in a way that makes sense?

https://github.com/chris-prener/geospatial-workshop/blob/gh-pages/setup.md

Also @lwasser and @lachlandeer - are we still going to be using rasterVis? I ask because it is not listed in the rocker geospatial image. I'll check and see if it gets installed anyway (the list is not complete). If rasterVis does not get installed anyway, and we are still using it, I'll build us our own Docker image on-top of that!

[lwasser]

i think it's good to show rastervis IF we get to the raster time series lessons. I've never taught a workshop where we actually get to it. normally i demo it at the end :) in short it's probably good to include but don't count on always needing it!

[chris-prener]

OK great @lwasser - the draft setup page is done now - can you take a quick look and let me know how I'm doing here?

https://github.com/chris-prener/geospatial-workshop/blob/gh-pages/setup.md

I'll keep you posted on the Docker side of things.

[lwasser]

you are awesome @chris-prener !! you and @lachlandeer are really kicking butt in the improve all of these lessons arena! just a note i'm heading out of town in just a few hours for a week. So i will be completely unresponsive until mid next week and then it make take me a day or two to catch up. just FYI :)

ok i have a few questions with that said. The install includes gdal and proj 4 etc etc. When i first started teaching this i did have them install that stuff. However, i've taught this material several times without install gdal or proj 4 or any of that and it's been just fine.

So my question is - has it been your experience that all of these libraries are necessary? GDAL in particular is a tricky install. easy on a mac with homebrew but windows can be tricky. and even after it's installed it's not always bound to r properly.

[lwasser]

actually looking closer the only thing i used to have people install was the gdal libraries. i've never installed: UDUNITS, proj4 and geos independently not even on my machines!

[chris-prener]

ok good to know. my understanding is that sf requires it via RTools for Windows... which I've had them install if they are Windows users and it goes totally fine. Am I incorrect there?

• README mentions it - https://github.com/r-spatial/sf
• DESCRIPTION requires it SystemRequirements: GDAL (>= 2.0.0), GEOS (>= 3.4.0), PROJ.4 (>= 4.8.0) - https://github.com/r-spatial/sf/blob/master/DESCRIPTION

I've always had Mac users install the non command line installers from here http://www.kyngchaos.com/software/frameworks and again, have not had issues.

I've never had a student on (nor do I use) Linux, so I can't speak to that.

[chris-prener]

And UDUNITS is a dependency of sf that doesn't install correctly on Linux machines, so you have to install it independently - this isn't an issue for Windows and macOS.

[lwasser]

AHHHH ok and i wouldn't know that as i've been slow to use sf teaching!

[chris-prener]

makes total sense @lwasser - I can be a little clearer about why the libraries are needed on the SETUP page?

[lwasser]

sure thing... well given you've taught sf more than i have i'll trust your decision here and perhaps others have input @jsta @lachlandeer !! it was interesting because i was teaching sp and hdf and for the first year i always fought with these libraries. then i tried it without and the workshop was seamless. It could be however that sf operates differently. A bit of clarification up front is a good idea.

[chris-prener]

ok @lwasser, @ErinBecker, @jsta and @lachlandeer (and @fmichonneau - would love your input here, too) - I wanted to pass on an update on Docker.

There is one package we include in our lessons - rasterVis - that is not available through the rocker/geospatial image. @lwasser indicated that this is only really used in the time series episode, and that she doesn't typically get to that when she teaches this workshop.

There are two options, then:

1. Build and maintain our own image on-top of rocker/geospatial that installs everything there plus rasterVis
2. Provide instructions that rocker/geospatial is adequate for everything but a particular episode, and that if someone is using Docker and wants to do that episode, they'll have to install.packages("rasterVis")

Given that we are one package shy, I would suggest option 2. I don't think the extra overhead of maintaining our own Docker image is worth it for one package when rocker/geospatial is so well maintained and it is easy enough to execute install.packages() inside a Docker session. Thoughts are welcome, either way, though!

[chris-prener]

I've opened a PR over on geospatial-workshop that addresses this Issue. https://github.com/datacarpentry/geospatial-workshop/pull/8

[ErinBecker]

@chris-prener - Is this ready to close?

[chris-prener]

yes @ErinBecker - the install instructions were updated to note that if folks are using Docker, they will need to install rasterVis themselves!