load server software, a la MBON in a Box, using docker, RStudio Server, Shiny...

[bbest]

With virtual machine setup by Qiao Qiao mbon:

ssh ben@mbon.marine.usf.edu

---------- Forwarded message ---------- From: Ben Best bdbest@gmail.com Date: Thu, Feb 23, 2017 at 8:36 PM Subject: USF server access for software installation and serving web applications To: Frank Muller-Karger carib@usf.edu

Hi Frank,

In order to demonstrate SDG14 products that include interactive mapping of dense global data layers, I would like to install and manage a server (preferable Linux Debian or Ubuntu OS) that can have a dedicated public IP address with the following software:

• docker
  • geospatial
  • shiny
  • jupyterhub
  • geoserver

I have been a system administrator for years in the past for a small NGO, an environmental school and a research lab. I'm aware of security concerns, and will only need SSH terminal access to these machines and open up necessary web ports (eg 80, 8080) to serve content. I will require 8 letter passwords and have initially just one account setup. Can work with you all to have others added. I will need at least sudo root access to install the software, plus the occasional additional scientific package in R or Python.

[bbest]

Using the RStudio image · rocker-org/rocker Wiki

simple with default rstudio username and password:

sudo docker run -d -p 8787:8787 rocker/geospatial

custom with root access, user ben, mounted volume /mbon:

sudo docker run -d -p 8787:8787  \
  -e ROOT=TRUE \
  -e USER=ben  -e PASSWORD=`cat ~ben/.passwd` \
  -v /mbon:/mbon \
  rocker/geospatial

# list processes by container-id
docker ps                                                

# kill container
docker kill <container-id>

# enter bash terminal of container
docker exec -it <container-id> bash

# add user within container
adduser <username>

[bbest]

@install vi in RStudio container on mbon.marine.usf.edu:

docker ps    # confirm containerid
containerid=304b27b9411c
docker exec -it $containerid bash
apt-get install vim

[bbest]

Yes!: Linking RStudio and Postgis + Geoserver using docker

Other approach building custom docker image:

• abkapoor/r-studio-postgres - Docker Hub
• dgolicher/rstudio - Docker Hub: Based on rocker/rstudio + support for PostGIS, gdal + many other packages. Latex for pdfs

[bbest]

Hi @7yl4r,

Do you have any experience with docker and/or interest in helping me with getting it going in the desired way?

I need to get Rstudio + Shiny + Postgis + Geoserver docker containers running and linked for interoperating with each other using common volume mounts to share files.

But when I try going through the following steps on my Mac a la Linking RStudio and Postgis using docker | rstudio-pubs, I get an error on docker startup:

# setup postgis container
mkdir -p ~/postgres_data
sudo docker run --name "postgis" -p 25432:5432 -d -v $HOME/postgres_data:/var/lib/postgresql kartoza/postgis

# stop, start postgis
sudo docker stop postgis
sudo docker start postgis

# install additional packages
sudo docker exec -it postgis bash
# inside bash, install s/w
apt-get install postgresql-9.5-plr

# link RStudio server container, swapping for geospatial
sudo mkdir -p ~/rstudio_server
sudo chmod 777 -R ~/rstudio_server
sudo docker stop postgis
sudo docker run --name "rstudio-geospatial" \
  --link postgis:postgis -d -p 8787:8787 \
  -e ROOT=TRUE \
  -v $HOME/rstudio_server:/home/rstudio rocker/geospatial

error:

docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/docker/volumes/c6713658f1d9eb77ff08908efe9f7a146cbe2677bf38b55f57d59bd89828646a/_data\\\" to rootfs \\\"/var/lib/docker/aufs/mnt/d2adcfdbc28357256219f954aa3b0959d45990c029e21a7bdfa9dbf39ee350f5\\\" at \\\"/var/lib/docker/aufs/mnt/d2adcfdbc28357256219f954aa3b0959d45990c029e21a7bdfa9dbf39ee350f5/home/rstudio/kitematic\\\" caused \\\"mkdir /var/lib/docker/aufs/mnt/d2adcfdbc28357256219f954aa3b0959d45990c029e21a7bdfa9dbf39ee350f5/home/rstudio/kitematic: permission denied\\\"\"".

Here are a few references to this problem, which should be resolved using latest Docker so confusion continues:

• Permission denied for directories created automatically by Dockerfile ADD command · Issue #1295 · moby/moby
• Permission denied - /var/jenkins_home/copy_reference_file.log · Issue #177 · jenkinsci/docker
• #749063 - docker makes df horrible - Debian Bug report logs

[bbest]

Apparently a newer preferred alternative to --link is using a network and --net , eg Docker - Wildsong Wiki:

docker network create pg
docker run --name postgis --net=pg -p 5432 -d wildsong/postgis
docker run --name geoserver --net=pg -p 8888:8080 -d wildsong/geoserver
docker network inspect pg

[7yl4r]

I have been waiting for an excuse to play with docker since I started here so I would love to help with this. I don't have experience but I'm familiar with the concepts of containerization.

I'll read some more about linked docker images and those links re the permissions error and see what I can do.

[bbest]

Awesome! Look forward to working this out with you @7yl4r. This is a major sticking point for being able to move forward with the SDG14 products

[7yl4r]

Interestingly, I'm getting a different error at the same step:

me@server:~$ sudo docker run --name "rstudio-geospatial" \
>   --link postgis:postgis -d -p 8787:8787 \
>   -e ROOT=TRUE \
>   -v $HOME/rstudio_server:/home/rstudio rocker/geospatial

docker: Error response from daemon: 
    driver failed programming external connectivity on endpoint rstudio-geospatial (35c5...2efe0): 
        Bind for 0.0.0.0:8787 failed: 
            port is already allocated.

That error is much more straightforward than what you were getting. Looks like the easiest fix for mine is to just let docker map the port automatically by leaving off the client port.

[7yl4r]

looks like it is up?

me@serv:~$ sudo docker start postgis
me@serv:~$ sudo docker run --name "rstudio-geo"   --link postgis:postgis -d -p 8787   -e ROOT=TRUE   -v $HOME/rstudio_server:/home/rstudio rocker/geospatial
me@serv:~$ sudo docker container list
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                     NAMES
95f73edf64bc        rocker/geospatial   "/init"                  2 minutes ago       Up 9 seconds        0.0.0.0:32769->8787/tcp   rstudio-geo
1dd0b9078fac        kartoza/postgis     "/bin/sh -c /start..."   About an hour ago   Up About a minute   0.0.0.0:25432->5432/tcp   postgis
304b27b9411c        rocker/geospatial   "/init"                  2 months ago        Up 2 months         0.0.0.0:8787->8787/tcp    laughing_meninsky

It's not clear to me how to test this. Maybe log in to rstudio-geo and try to connect to postgis?

me@serv:~$ sudo docker exec -it rstudio-geo bash
root@95f73edf64bc:/# ping postgis
PING postgis (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: icmp_seq=0 ttl=64 time=0.341 ms
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.070 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.096 ms
^C--- postgis ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.070/0.169/0.341/0.122 ms

Ready for some test code I guess. :shipit:

[bbest]

Hi @7yl4r,

Good news! Ok, I think my images are out of date. Cleaning up and retrying...

# list and remove previous containers
docker container list -a

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS                              NAMES
4a5bb5ab84aa        rocker/geospatial   "/init"                  3 days ago          Created                       3838/tcp, 0.0.0.0:8787->8787/tcp   rstudio-geospatial
f31cf5746c9c        rocker/rstudio      "/init"                  3 days ago          Created                       3838/tcp, 0.0.0.0:8787->8787/tcp   rstudio
50b2cb700b89        kartoza/postgis     "/bin/sh -c /start..."   3 days ago          Exited (137) 12 minutes ago                                      postgis
d0a8056897b6        hello-world         "/hello"                 4 months ago        Exited (0) 4 months ago                                          cocky_noether
1d2abcb79cc7        rocker/shiny        "/usr/bin/shiny-se..."   4 months ago        Exited (0) 4 months ago                                          rocker_shiny
df1e93ec1cfd        hello-world         "/hello"                 4 months ago        Exited (0) 4 months ago                                          nostalgic_bassi
deb40d224b96        nginx               "nginx -g 'daemon ..."   6 months ago        Exited (0) 4 months ago                                          webserver

# remove previous containers
docker container rm rstudio-geospatial rstudio postgis rocker_shiny

# check image versions
docker image list -a

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
kartoza/postgis     latest              24fc1d18ba47        3 weeks ago         762 MB
rocker/geospatial   latest              0568f47b9b44        4 weeks ago         2.78 GB
rocker/rstudio      latest              7a807646f0be        4 weeks ago         993 MB
rocker/shiny        latest              8fcb2d14a88d        4 months ago        1.32 GB
hello-world         latest              48b5124b2768        4 months ago        1.84 kB
nginx               latest              abf312888d13        6 months ago        182 MB

# update images
docker pull kartoza/postgis
docker pull rocker/geospatial
docker pull rocker/rstudio
docker pull rocker/shiny

[bbest]

Ok, it's working :) Guess I just had outdated / incompatible images before. Thanks for helping @7yl4r.

You can at least login to your RStudio instance http://localhost:8787 and per Using the RStudio image · rocker-org/rocker Wiki.

# check and cleanup images after pull
docker image list -a
docker image rm 24fc1d18ba47 0568f47b9b44 7a807646f0be 8fcb2d14a88d
docker image list -a

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
rocker/rstudio      latest              53a4701567fe        2 hours ago         999 MB
rocker/geospatial   latest              f6400b80d053        2 days ago          2.93 GB
kartoza/postgis     latest              90fd507b8ac1        3 days ago          762 MB
rocker/shiny        latest              682eb5fda1f3        3 weeks ago         1.23 GB
hello-world         latest              48b5124b2768        4 months ago        1.84 kB
nginx               latest              abf312888d13        6 months ago        182 MB

[bbest]

shiny+rstudio+postgis on mbon.marine.usf.edu

# review containers & images
docker container ls -a
docker container ls -a --format "{{.Names}}"

# archive existing rstudio-geo and rename
docker commit rstudio-geo rstudio-geo:2017-06-15
docker rename rstudio-geo rstudio-geo_2017-06-15
docker stop rstudio-geo_2017-06-15

# cleanup old
docker stop postgis
docker container rm rstudio-geospatial postgis laughing_meninsky epic_easley \
  wonderful_wing nervous_edison agitated_jennings docker-nginx

# setup volumes to mount
# mkdir -p /mbon         # rstudio-data
mkdir -p /mbon/shiny     # shiny-data
mkdir -p /mbon/shiny-log # shiny-log
mkdir -p /mbon/postgres  # postgres-data

# update images
docker image ls
docker pull kartoza/postgis
docker pull rocker/geospatial
docker pull rocker/rstudio
docker pull rocker/shiny
docker image ls

# run postgis on port 5432
sudo docker run --name "postgis" \
  -d -p 5432:5432 \
  -e ROOT=TRUE \
  -v /mbon:/mbon \
  -v /mbon/postgres:/var/lib/postgresql \
  kartoza/postgis

# run shiny on port 80
docker run --name "shiny" \
  -d -p 80:3838 \
  -e ROOT=TRUE \
  -v /mbon:/mbon \
  -v /mbon/shiny:/srv/shiny-server \
  -v /mbon/shiny-log:/var/log/shiny-server \
  rocker/shiny

# run rstudio-geo on port 8787
# docker container stop rstudio-geo; docker container rm rstudio-geo
sudo docker run --name "rstudio-geo" \
  -d -p 8787:8787  \
  --link postgis:postgis \
  --link shiny:shiny \
  -e ROOT=TRUE \
  -e USER=mbon -e PASSWORD=`cat ~/.mbon_passwd` \
  -v /mbon:/mbon \
  rocker/geospatial

Working at:

• RStudio: http://mbon.marine.usf.edu:8787
• Shiny app: http://mbon.marine.usf.edu/test

[bbest]

postgis+geoserver+shiny-rstudio+www on mbon.marine.usf.edu

Want spatial capabilities, especially rgdal, in Shiny server too, so going with recommendation to compose a Dockerfile and docker build Shiny into the rocker/geospatial Dockerful by:

• adding the line RUN export ADD=shiny && bash /etc/cont-init.d/add to rocker-org/geospatial/3.4.0/Dockerfile into new Dockerfile in github.com/marinebon/docker-rstudio-shiny

• docker build image on mbon.marine.usf.edu, which takes ~ 3 hours:

  docker build -t rstudio-shiny github.com/marinebon/docker-rstudio-shiny

Reference: Deploying rocker/geospatial app to rocker/shiny? · Issue #235 · rocker-org/rocker

prep docker images, containers and dirs

# review containers & images
docker ps ls -a
docker ps ls -a --format "{{.Names}}"

# cleanup old
docker stop postgis
docker container rm rstudio-geospatial postgis

# update images
docker image ls
docker pull kartoza/postgis
docker pull kartoza/geoserver
docker pull rocker/geospatial
docker pull rocker/shiny
docker pull nginx

# setup volumes to mount
# mkdir -p /mbon         # rstudio-data
mkdir -p /mbon/shiny     # shiny-data
mkdir -p /mbon/shiny-log # shiny-log
mkdir -p /mbon/postgres  # postgres-data
mkdir -p /mbon/geoserver # geoserver-data
mkdir -p /mbon/www       # www-data
mkdir -p /mbon/www-conf  # www-conf

populate initial www files

# docker stop www; docker container rm www
docker run --name "www" \
  -v /mbon:/mbon:ro \
  -p 80:80 \
  -d -t nginx

docker exec -it www bash

Now in www container, copy html and conf files:

cd /usr/share/nginx/html
cp -R * /mbon/www/.
cd /etc/nginx
cp -R * /mbon/www-conf/.
exit

Kill www container now html and conf files copied:

docker stop www
docker container rm www

set permissions on dirs

sudo chgrp -R docker shiny shiny-log geoserver www www-conf
sudo chmod -R 775 shiny shiny-log geoserver www www-conf

docker compose & build image rstudio-geospatial+shiny: rstudio-shiny

Deploying rocker/geospatial app to rocker/shiny? · Issue #235 · rocker-org/rocker

• add to rocker/geospatial Docker: RUN export ADD=shiny && bash /etc/cont-init.d/add
• new Docker at github.com/marinebon/docker-rstudio-shiny
• docker build: takes ~ 3 hours

docker build -t rstudio-shiny github.com/marinebon/docker-rstudio-shiny

push image

docker login # registed as bdbest (bdbest@gmail.com)
docker tag rstudio-shiny bdbest/rstudio-shiny:R-3.4-geospatial
docker push bdbest/rstudio-shiny:3.4-geospatial

clear docker to free disk

Cleanup and reset docker on Jenkins workers / slaves

docker ps -q | xargs --no-run-if-empty docker stop
docker ps -q -a | xargs --no-run-if-empty docker rm --force --volumes
docker volume ls -q | xargs --no-run-if-empty docker volume rm
docker images -a -q | xargs --no-run-if-empty docker rmi -f

docker run final containers

docker run --name "postgis" \
  -p 5432:5432 \
  -v /mbon:/mbon \
  -v /mbon-postgres:/var/lib/postgresql \
  -d -t kartoza/postgis

docker run --name "geoserver" \
  -p 8080:8080 \
  --link postgis:postgis \
  -v /mbon:/mbon \
  -v /mbon/geoserver:/opt/geoserver/data_dir \
  -d -t kartoza/geoserver

# docker stop rstudio-shiny; docker container rm rstudio-shiny
docker run --name "rstudio-shiny" \
  -p 8787:8787 -p 3838:3838 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  -v /mbon:/mbon \
  -v /mbon/shiny:/srv/shiny-server \
  -v /mbon/shiny-log:/var/log/shiny-server \
  -e ROOT=TRUE \
  -e USER=mbon -e PASSWORD=`cat ~/.mbon_passwd` \
  -d -t bdbest/rstudio-shiny:R-3.4-geospatial

# docker stop www; docker container rm www
docker run --name "www" \
  -v /mbon:/mbon:ro \
  -v /mbon/www:/usr/share/nginx/html:ro \
  -v /mbon/www-conf:/etc/nginx:ro \
  -p 80:80 \
  -d -t nginx

human readable disk free

df -h

Filesystem                   Size  Used Avail Use% Mounted on
udev                         3.9G     0  3.9G   0% /dev
tmpfs                        799M   82M  718M  11% /run
/dev/mapper/mbon--vg-root     19G  7.7G  9.6G  45% /
tmpfs                        3.9G  608K  3.9G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                    472M  105M  343M  24% /boot
corals.marine.usf.edu:/mbon   15T  5.0T  9.6T  35% /mbon
tmpfs                        799M     0  799M   0% /run/user/1000

PROBLEM!: too little disk

Run out of disk space when loading 9G obis_occ.csv into postgres database, which can't store in /mbon, presumably because it's a cross-mounted system corals.marine.usf.edu:/mbon.

test postgis db connection

# mbon server
psql -h localhost -U docker -p 5432 -l

# postgis container
docker exec -it postgis bash
psql -h localhost -U docker -p 5432 -l

# if not running, try starting to get error message
service postgresql start

# fix permissions
sudo chmod -R 0700 /mbon-postgres/9.5/main

# rstudio-shiny container
docker exec -it rstudio-shiny bash
# install psql client
apt-get update -y
sudo apt install postgresql-client
# connect using environment variable added by docker --link
psql -h $POSTGIS_PORT_5432_TCP_ADDR -U docker -p 5432 -l

TODO

• db bkup: psql -h localhost -p 25432 -U docker elections < dump.sql

• postgresql - Working with PostGIS data in R? - Geographic Information Systems Stack Exchange

require(rgdal)
dsn="PG:dbname='gis'"

# list tables in db
ogrListLayers(dsn)

# fetch polygon
polys = readOGR(dsn="PG:dbname='gis'","ccsm_polygons")

• RPubs - Working with PostGIS from R

• Using PostgreSQL in R: A quick how-to | R-bloggers

• https://github.com/kartoza/docker-geoserver

• Post-installation steps for Linux - Configure Docker to start on boot | Docker Documentation

• http://nginx.org/en/docs/: www rewrite rules and serving sdg14 and other repo sites

References

• RPubs - Working with PostGIS from R
• rstudio-pubs-static.s3.amazonaws.com/203532_28f752f6b0d84cc7b05bb08055ad48a8.html
• RPubs - Working with PostGIS from R
• Getting started with PostgreSQL in R | R-bloggers
• Linking RStudio and Postgis using docker | rstudio-pubs

[bbest]

Hi @7yl4r,

Would you be able to help me with making more disk space available to the virtual mbon server?

I'm running out of disk space when loading the 9G OBIS observation CSV (obis_occ.csv) into postgres database (using SQL from obis-postgis-connect.Rmd), which is related to being unable to assign the /var/lib/postgresql path to /mbon (initdb throws errors about not being a cluster file system), which is presumably because /mbon is a cross-mounted file system corals.marine.usf.edu:/mbon. Details are above in issue.

Since this will be storage for the spatial database, fast locally connected storage is ideal. And would be great to get at least a couple hundred GB.

Thanks so much for your help!

[7yl4r]

Yes I think I can do that. I was able to expand one of our VM's partitions using libguestfs-tools's virt-resize, but found it to be a bit of a clunky process and had to try a few times.

I can add a local volume more easily. I believe that will look like a 2nd local hard disk to your VM and thus should work. Do you think that will work for you?

[7yl4r]

In terms of making it fast and local... I'll have to take the server offline to install additional physical disks. There isn't much else running on this one so just let me know when you want to schedule the downtime.

[bbest]

Hi @7yl4r,

Yeah, a second volume would work great! No need to expand root volume. Thanks a bunch

[7yl4r]

When would you like me to schedule downtime so I can add the drive?

[bbest]

Hi Tyler,

Please go for it. I don't think anyone else is using it. Would love to load OBIS data and work on this over the weekend.

Thanks very much, Ben

On Jun 16, 2017 11:33 AM, "Tylar" notifications@github.com wrote:

When would you like me to schedule downtime so I can add the drive?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14/issues/4#issuecomment-309101725, or mute the thread https://github.com/notifications/unsubscribe-auth/ACtLCRBJn2aSLsAIGZO96p8fugyTLc6mks5sEsp2gaJpZM4MinbW .

[7yl4r]

900 gb disk now available at /dev/vdb. Below are the commands I would typically format with from mbon but feel free to do something different if you like:

fdisk /dev/vdb
# this brings up interactive CLI... do:
#    p # show partitions
#        # this list should be empty, if not then you might want to double check for typos
#    n # new
#    p # primary
#    allow defaults or set prefs
#    w # write and exit
#    formatted drive now should be @ /dev/vdb1

/sbin/mkfs.ext3 -L /my/filesystem/label /dev/vdb1
mkdir /mnt/my_mount_name
mount /dev/vdb1 /mnt/my_mount_name

# if this works
vi /etc/fstab  # and add a line similar to:
/dev/vdb1 /mnt/my_mount_name ext3 defaults 0 0

[bbest]

Ok, excited to get this going, but I don't see it...

ben@mbon:~$ sudo fdisk /dev/vdb
[sudo] password for ben: 

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

fdisk: cannot open /dev/vdb: No such file or directory

ben@mbon:~$ ls /dev
autofs           loop5               rfkill    tty26  tty59      ttyS4
block            loop6               rtc       tty27  tty6       ttyS5
bsg              loop7               rtc0      tty28  tty60      ttyS6
btrfs-control    loop-control        sda       tty29  tty61      ttyS7
bus              mapper              sda1      tty3   tty62      ttyS8
cdrom            mbon-vg             sda2      tty30  tty63      ttyS9
char             mcelog              sda5      tty31  tty7       uhid
console          mem                 sg0       tty32  tty8       uinput
core             memory_bandwidth    sg1       tty33  tty9       urandom
cpu              mqueue              shm       tty34  ttyprintk  userio
cpu_dma_latency  net                 snapshot  tty35  ttyS0      vcs
cuse             network_latency     snd       tty36  ttyS1      vcs1
disk             network_throughput  sr0       tty37  ttyS10     vcs2
dm-0             null                stderr    tty38  ttyS11     vcs3
dm-1             port                stdin     tty39  ttyS12     vcs4
dri              ppp                 stdout    tty4   ttyS13     vcs5
dvd              psaux               tty       tty40  ttyS14     vcs6
ecryptfs         ptmx                tty0      tty41  ttyS15     vcsa
fb0              pts                 tty1      tty42  ttyS16     vcsa1
fd               ram0                tty10     tty43  ttyS17     vcsa2
full             ram1                tty11     tty44  ttyS18     vcsa3
fuse             ram10               tty12     tty45  ttyS19     vcsa4
hpet             ram11               tty13     tty46  ttyS2      vcsa5
hugepages        ram12               tty14     tty47  ttyS20     vcsa6
hwrng            ram13               tty15     tty48  ttyS21     vda
i2c-0            ram14               tty16     tty49  ttyS22     vfio
initctl          ram15               tty17     tty5   ttyS23     vga_arbiter
input            ram2                tty18     tty50  ttyS24     vhci
kmsg             ram3                tty19     tty51  ttyS25     vhost-net
lightnvm         ram4                tty2      tty52  ttyS26     virtio-ports
log              ram5                tty20     tty53  ttyS27     vport0p1
loop0            ram6                tty21     tty54  ttyS28     zero
loop1            ram7                tty22     tty55  ttyS29
loop2            ram8                tty23     tty56  ttyS3
loop3            ram9                tty24     tty57  ttyS30
loop4            random              tty25     tty58  ttyS31

ben@mbon:~$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                         3.9G     0  3.9G   0% /dev
tmpfs                        799M  8.9M  790M   2% /run
/dev/mapper/mbon--vg-root     19G  7.8G  9.5G  45% /
tmpfs                        3.9G     0  3.9G   0% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                    472M  105M  343M  24% /boot
corals.marine.usf.edu:/mbon   15T  5.0T  9.6T  35% /mbon
tmpfs                        799M     0  799M   0% /run/user/538
tmpfs                        799M     0  799M   0% /run/user/1000

Please feel free to format and mount to /mbon-local

[7yl4r]

sorry! it's /dev/vda

[7yl4r]

I haven't really looked into the details of my formatting checklist to be sure they are the best way to do it; those are just my quick-fix notes. I read good things about lvm and fancier things, but I need to read up some more.

[bbest]

Ok, formatting drive now. Your instructions seem good so far...

sudo fdisk /dev/vda
# this brings up interactive CLI... do:
#    p # show partitions
#        # this list should be empty, if not then you might want to double check for typos
#    n # new
#    p # primary
#    allow defaults or set prefs
#    w # write and exit
#    formatted drive now should be @ /dev/vda1

sudo /sbin/mkfs.ext3 -L /my/filesystem/label /dev/vda1

[bbest]

Woohoo, obis_occ.csv loaded and new drive working!

sudo su - root

fdisk /dev/vda
# this brings up interactive CLI... do:
#    p # show partitions
#        # this list should be empty, if not then you might want to double check for typos
#    n # new
#    p # primary
#    allow defaults or set prefs
#    w # write and exit
#    formatted drive now should be @ /dev/vda1

# format
mkfs -t ext4 /dev/vda1

# temporarily mount
mkdir -p /mnt/mbon-local
mount -o defaults /dev/vda1 /mnt/mbon-local
umount /dev/vda1

# mount on boot
vi /etc/fstab 

added line:

/dev/vda1        /mbon-local     ext4    defaults        0       2

mount -a

mount: can't find UUID=7fb746f3-677f-43a1-adf5-b7f8ae005d7f

Doh! Commented out:

#UUID=7fb746f3-677f-43a1-adf5-b7f8ae005d7f /boot           ext2    defaults        0       2

Tried to get /boot by uncommenting:

/dev/sda1        /boot           ext2    defaults        0       2

but got:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

So commented both lines for /boot

# /boot was on /dev/sda1 during installation
#UUID=7fb746f3-677f-43a1-adf5-b7f8ae005d7f /boot           ext2    defaults        0       2
#/dev/sda1       /boot           ext2    defaults        0       2

and then mounting works:

mount -a

ben@mbon:~$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                         3.9G     0  3.9G   0% /dev
tmpfs                        799M  9.1M  790M   2% /run
/dev/mapper/mbon--vg-root     19G  7.7G  9.5G  45% /
tmpfs                        3.9G  640K  3.9G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
corals.marine.usf.edu:/mbon   15T  5.0T  9.6T  35% /mbon
tmpfs                        799M     0  799M   0% /run/user/1000
/dev/vda1                    886G   11G  831G   2% /mbon-local

[bbest]

Next: use docker run --restart unless-stopped per Start containers automatically | Docker Documentation

[bbest]

Install PostgreSQL Studio: Web-based PostgreSQL Development

ben@mbon:$ docker exec -it geoserver bash

cd /usr/local/tomcat/webapps
wget http://downloads.postgresqlstudio.org/2.0/pgstudio_2.0.zip
unzip pgstudio_2.0.zip

env | grep -i postgis
# POSTGIS_PORT_5432_TCP=tcp://172.17.0.2:5432

http://mbon.marine.usf.edu:8080/pgstudio

[bbest]

Updating since @7yl4r expanded disk, CPU and memory resources per https://github.com/marinebon/sdg14/issues/20...

configure autostart

• Configure Docker to start on boot | Docker Documentation

  sudo systemctl enable docker

• Start containers automatically | Docker Documentation

  docker run --restart unless-stopped ...

clear docker to free disk

Cleanup and reset docker on Jenkins workers / slaves

docker ps -q | xargs --no-run-if-empty docker stop
docker ps -q -a | xargs --no-run-if-empty docker rm --force --volumes
docker volume ls -q | xargs --no-run-if-empty docker volume rm
docker images -a -q | xargs --no-run-if-empty docker rmi -f

docker run final containers with restart

docker run --name "postgis" \
  --restart unless-stopped \
  -p 5432:5432 \
  -v /mbon:/mbon \
  -v /mbon-postgres:/var/lib/postgresql \
  -d -t kartoza/postgis

docker run --name "geoserver" \
  --restart unless-stopped \
  -p 8080:8080 \
  --link postgis:postgis \
  -v /mbon:/mbon \
  -v /mbon/geoserver:/opt/geoserver/data_dir \
  -d -t kartoza/geoserver

docker run --name "rstudio-shiny" \
  --restart unless-stopped \
  -p 8787:8787 -p 3838:3838 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  -v /mbon:/mbon \
  -v /mbon/shiny:/srv/shiny-server \
  -v /mbon/shiny-log:/var/log/shiny-server \
  -e ROOT=TRUE \
  -e USER=mbon -e PASSWORD=`cat ~/.mbon_passwd` \
  -d -t bdbest/rstudio-shiny:R-3.4-geospatial

docker run --name "www" \
  --restart unless-stopped \
  -p 80:80 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  --link rstudio-shiny:rstudio-shiny \
  -v /mbon:/mbon:ro \
  -v /mbon/www:/usr/share/nginx/html:ro \
  -v /mbon/www-conf:/etc/nginx:ro \
  -d -t nginx

install PostgreSQL Studio

docker exec geoserver bash -c 'cd /usr/local/tomcat/webapps; wget http://downloads.postgresqlstudio.org/2.0/pgstudio_2.0.zip; unzip pgstudio_2.0.zip'

Visit http://mbon.marine.usf.edu:8080/pgstudio

confirm docker running

docker ps -a

CONTAINER ID        IMAGE                                   COMMAND                  CREATED              STATUS              PORTS                                            NAMES
5c54dfebd408        nginx                                   "nginx -g 'daemon ..."   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp                               www
83b74df0366c        bdbest/rstudio-shiny:R-3.4-geospatial   "/init"                  2 minutes ago        Up 2 minutes        0.0.0.0:3838->3838/tcp, 0.0.0.0:8787->8787/tcp   rstudio-shiny
719e5ca7cd7b        kartoza/geoserver                       "catalina.sh run"        4 minutes ago        Up 4 minutes        0.0.0.0:8080->8080/tcp                           geoserver
426f948c56b3        kartoza/postgis                         "/bin/sh -c /start..."   9 minutes ago        Up 9 minutes        0.0.0.0:5432->5432/tcp                           postgis

confirm enough disk space

human readable disk free

df -h

Filesystem                   Size  Used Avail Use% Mounted on
udev                          16G     0   16G   0% /dev
tmpfs                        3.2G  314M  2.9G  10% /run
/dev/mapper/mbon--vg-root     19G  7.1G   11G  42% /
tmpfs                         16G  576K   16G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                         16G     0   16G   0% /sys/fs/cgroup
/dev/sda1                    472M  105M  343M  24% /boot
corals.marine.usf.edu:/mbon   15T  4.8T  9.9T  33% /mbon
/dev/vda1                    886G   40G  802G   5% /mbon-postgres
tmpfs                        3.2G     0  3.2G   0% /run/user/1000

[bbest]

psql rename user docker -> mbon

docker exec -it postgis bash
psql -h localhost -U postgres -p 5432 postgres

ALTER USER "postgres" WITH PASSWORD '****';
\q

exit # back to mbon
psql -h localhost -U postgres -p 5432 postgres

ALTER USER docker RENAME TO mbon;
ALTER USER mbon WITH PASSWORD '****';
\q

[bbest]

Moved content to /mbon/sdg14 on www:

docker rm www
docker run --name "www" \
  --restart unless-stopped \
  -p 80:80 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  --link rstudio-shiny:rstudio-shiny \
  -v /mbon:/mbon:ro \
  -v /mbon/sdg14:/usr/share/nginx/html:ro \
  -v /mbon/www-conf:/etc/nginx:ro \
  -d -t nginx

[bbest]

Changed back to /mbon/www, which contains git clone https://github.com/marinebon/sdg14-www.git .

docker stop www
docker rm www

docker run --name "www" \
  --restart unless-stopped \
  -p 80:80 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  --link rstudio-shiny:rstudio-shiny \
  -v /mbon:/mbon:ro \
  -v /mbon/www:/usr/share/nginx/html:ro \
  -v /mbon/www-conf:/etc/nginx:ro \
  -d -t nginx

[bbest]

redo rstudio-shiny with /mbon-local volume

old + -v /mnt/mbon-supplement:/mbon-local

• How can I add a volume to an existing Docker container? - Stack Overflow
• load server software, a la MBON in a Box, using docker, RStudio Server, Shiny... · Issue #4 · marinebon/sdg14

archive old

docker ps -a
docker commit rstudio-shiny rstudio-shiny:2017-10-16
docker stop rstudio-shiny
docker rename rstudio-shiny rstudio-shiny_2017-10-16

create new

docker run --name "rstudio-shiny" \
  --restart unless-stopped \
  -p 8787:8787 -p 3838:3838 \
  --link postgis:postgis \
  --link geoserver:geoserver \
  -v /mbon:/mbon \
  -v /mbon/shiny:/srv/shiny-server \
  -v /mbon/shiny-log:/var/log/shiny-server \
  -v /mnt/mbon-supplement:/mbon-local \
  -e ROOT=TRUE \
  -e USER=mbon -e PASSWORD=`cat ~/.mbon_password` \
  -d -t bdbest/rstudio-shiny:R-3.4-geospatial

[bbest]

reinstall pgstudio

docker exec geoserver bash -c '\
  cd /usr/local/tomcat/webapps; \
  wget http://downloads.postgresqlstudio.org/2.0/pgstudio_2.0.zip; \
  unzip pgstudio_2.0.zip'

ALTER USER docker WITH PASSWORD 'M*****#*';

[bbest]

after apt-get install gdal-bin, aptitude install gdal-bin came into dependency upgrade hell with postgis server:

ERROR:  could not access file "$libdir/postgis-2.3": No such file or directory

Couldn't even run basic SQL:

SELECT postgis_full_version();

So redoing...

docker stop geoserver
# docker stop postgis
docker pull kartoza/postgis
docker rename postgis postgis-funk
docker rm postgis
docker run --name "postgis" \
  --restart unless-stopped \
  -p 5432:5432 \
  -v /mbon:/mbon \
  -v /mnt/mbon-supplement:/mbon-local \
  -v /mnt/mbon-supplement/postgresql:/var/lib/postgresql \
  -d -t kartoza/postgis
docker start postgis

docker exec -it postgis bash
psql -h localhost -p 5432 -U docker -W mbon_gis

SELECT postgis_full_version();
# POSTGIS="2.3.2 r15302" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="0.11.99" RASTER

docker rename geoserver geoserver-funk
docker run --name "geoserver" \
  --restart unless-stopped \
  -p 8080:8080 \
  --link postgis:postgis \
  -v /mbon:/mbon \
  -v /mnt/mbon-supplement:/mbon-local \
  -v /mnt/mbon-supplement/geoserver:/opt/geoserver/data_dir \
  -d -t kartoza/geoserver

docker start geoserver

docker exec geoserver bash -c '\
  cd /usr/local/tomcat/webapps; \
  wget http://downloads.postgresqlstudio.org/2.0/pgstudio_2.0.zip; \
  unzip pgstudio_2.0.zip'

[bbest]

GeoServer: Install Map Vector Tiles

Installing the Vector Tiles Extension — GeoServer 2.13.x User Manual

cd /usr/local/tomcat/webapps/geoserver/WEB-INF/lib
wget https://downloads.sourceforge.net/project/geoserver/GeoServer/2.12.0/extensions/geoserver-2.12.0-vectortiles-plugin.zip
unzip geoserver-2.12.0-vectortiles-plugin.zip
#inflating: gson-2.3.1.jar          
#inflating: gs-vectortiles-2.12.0.jar  
#inflating: java-vector-tile-1.0.9.jar  
#inflating: protobuf-javanano-3.0.0-alpha-2.jar  

GeoServer: Enable CORS in GeoServer

Used for consuming MapBox Vector Tiles in apps.

• Cross-Origin Resource Sharing (CORS) - HTTP | MDN
• Enabling CORS in GeoServer (jetty)? - Geographic Information Systems Stack Exchange
  • determine version jetty-server-9.2.13.v20150730.jar from https://fossies.org/linux/www/geoserver-2.12.0-bin.zip/
  • download: http://central.maven.org/maven2/org/eclipse/jetty/jetty-servlets/9.2.13.v20150730/jetty-servlets-9.2.13.v20150730.jar
• Container Considerations : Enable CORS — GeoServer 2.12.x User Manual

Install jetty-servlets & jetty-util

cd /usr/local/tomcat/webapps/geoserver/WEB-INF/lib/
wget http://central.maven.org/maven2/org/eclipse/jetty/jetty-servlets/9.2.13.v20150730/jetty-servlets-9.2.13.v20150730.jar
wget http://central.maven.org/maven2/org/eclipse/jetty/jetty-util/9.2.13.v20150730/jetty-util-9.2.13.v20150730.jar

Enable CORS in web.xml

cd /usr/local/tomcat/webapps/geoserver/WEB-INF
apt-get install vim # install vi
vi web.xml

Uncomment following sections in web.xml:

  <!-- Uncomment following filter to enable CORS -->
  <filter>
    <filter-name>cross-origin</filter-name>
    <filter-class>org.eclipse.jetty.servlets.CrossOriginFilter</filter-class>
  </filter>

  <!-- Uncomment following filter to enable CORS -->
  <filter-mapping>
    <filter-name>cross-origin</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

Restart

exit
docker stop geoserver
docker start geoserver

Logs of Tomcat for Debugging

cd /usr/local/tomcat/logs
grep -ri jetty *