M. Edward (Ed) Borasky znmeb@znmeb.net, 2018-04-04
Data Science Pet Containers comprise a collection of open-source software for all phases of the data science workflow, from ingestion of raw data through visualization, exploration, analysis and reporting. We provide the following tools:
As the name implies, the software is distributed via Docker. The user
simply clones a Git repository and uses the command docker-compose up
to bring up the services.
Why do it this way?
I've coded up some examples of how I've used this toolset for Hack
Oregon. They're in data-science-pet-containers/examples. Those that
require host-side operations mostly use Bash for scripting, and they
work on a Linux host, including Windows 10 Pro Windows Subsystem for
Linux (WSL) Ubuntu.
Clone this repository and
cd data-science-pet-containers/containers.
Copy sample.env to .env. Edit .env and change the
POSTGRES_PASSWORD. You don't need to change the other values.
Copy any PostgreSQL database backups you want restored to
data-science-pet-containers/containers/Backups. Copy any raw data
files you want on the image to
data-science-pet-containers/containers/Raw.
docker-compose -f postgis.yml up -d --build. The first time you
run this, it will take some time. Once the image is built and the
databases restored, it will be faster.
When it's done you'll see
Successfully tagged postgis:latest
Creating containers_postgis_1 ... doneType docker logs -f containers_postgis_1 to verify that the
restores worked and the service is listening.
PostgreSQL init process complete; ready for start up.
LOG: database system was shut down at 2018-03-18 05:19:22 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher startedType CTL-C to stop following the container log.
Connect to the container from the host: user name is postgres,
host is localhost, port is the value of HOST_POSTGRES_PORT,
usually 5439, and password is the value of POSTGRES_PASSWORD. You
can connect with any client that uses the PostgreSQL protocol
including pgAdmin and QGIS. You can also connect with Jupyter
notebooks and RStudio Desktop.
To stop the service, type docker-compose -f postgis.yml stop. To start
it back up again, docker-compose -f postgis.yml start.
The container and its filesystem will persist across host reboots. To
destroy them, type docker-compose -f postgis.yml down.
cd data-science-pet-containers/containers.sample.env to .env. For security reasons,
.env is listed in .gitignore, so it won't be checked
into version control..env. The variables you need to define are
HOST_POSTGRES_PORT: If you have PostgreSQL installed on
your host, it's probably listening on port 5432. The
postgis service listens on port 5432 inside the Docker
network, so you'll need to map its port 5432 to another
port. Set HOST_POSTGRES_PORT to the value you want; 5439
is what I use.POSTGRES_PASSWORD: To connect to the postgis service,
you need a user name and a password. The user name is the
default, the database superuser postgres. Docker will set
the password for the postgres user in the postgis
service to the value of POSTGRES_PASSWORD.DB_USERS_TO_CREATE: When the postgis service first comes
up, the users in this list are created in the database. If
you're working on the 2018 Hack Oregon projects, there's no
reason to change this.Here's sample.env:
# postgis container
HOST_POSTGRES_PORT=5439
POSTGRES_PASSWORD=some.string.you.can.remember.that.nobody.else.can.guess
DB_USERS_TO_CREATE=disaster-resilience housing-affordability local-elections transportation-systems urban-developmentDB_USERS_TO_CREATEI've provided these to facilitate database ownership wrangling and
backup-restore testing. As given in sample.env, these are the accounts
Hack Oregon's PostgreSQL server will have. Some notes:
In the postgis image, they are also Linux users. For example, you
can
docker-exec -it -u disaster-resilience containers_postgis_1 /bin/bash
and you'll be at a command prompt as disaster-resilience.
In the postgis image, they are database superusers. For example,
you can do createuser to create a database user and createdb to
create a database. The "home database" for each of these users - a
database with the same name as the Linux and database user - is
created the first time the container comes up.
amazon image, they are not Linux users, nor are they
database superusers. They are created with the same permissions they
have on the Hack Oregon PostgreSQL server:
--no-createdb --no-createrole --no-superuser --no-replication.ALTER DATABASE disaster OWNER TO disaster-resilience;ALTER DATABASE disaster OWNER TO "disaster-resilience";See the demo in exanples/reowning_a_database for an example of how
these users can be exploited in database wrangling.
Choose your version:
postgis.yml: PostGIS only. If you're doing all the analysis on
the host and just want the PostGIS service, choose this. If
you're an experienced Linux command-line user, this image has a
comprehensive collection of extract-transform-load (ETL) and GIS
tools.jupyter.yml: PostGIS and Jupyter Choose this if you want to
run a Jupyter notebook server inside the Docker network.rstats.yml: PostGIS and RStudio Server. Choose this if you
want an RStudio Server inside the Docker network.amazon.yml: PostGIS and an Amazon Linux 2 server running
PostgreSQL. This is a specialized configuration for testing
database backups for AWS server readiness. Most users won't need
to use this.Type docker-compose -f <version> up -d --build. Docker will
build/rebuild the images and start the services.
Note that if you want to bring up all the services in one
shot, just type docker-compose up -d --build. This takes quite a
bit of time - from 45 minutes to an hour the first time, depending
on download bandwidth and disk I/O speed.
The postgis service is based on the official PostgreSQL image from the
Docker Store: https://store.docker.com/images/postgres. It is running
jessie PostgreSQL server.All the images except amazon acquire PostgreSQL and its accomplices
from the official PostgreSQL Global Development Group (PGDG) Debian
repositories: https://www.postgresql.org/download/linux/debian/.
I've tried to provide a comprehensive command line experience. Git,
curl, wget, lynx, nano, emacs and vim are there, as is most
of the command-line GIS stack (gdal, proj, spatialite,
rasterlite, geotiff, osm2pgsql and osm2pgrouting), and of course
psql.
I've also included python3-csvkit for Excel, CSV and other text files,
unixodbc for ODBC connections and mdbtools for Microsoft Access
files. If you want to extend this image further, it is based on Debian
jessie.
You can log in as the Linux superuser root with
docker exec -it -u root containers_postgis_1 /bin/bash.
I've added a database superuser called dbsuper. This should be your
preferred login, rather than using the system database superuser
postgres. Log in with
docker exec -it -u dbsuper -w /home/dbsuper containers_postgis_1 /bin/bash.
For Linux hosts, including Windows 10 Pro Windows Subsystem for Linux Ubuntu (https://github.com/hackoregon/data-science-pet-containers/blob/master/win10pro-wsl-ubuntu-tools/README.md), I've created some convenience scripts:
login2postgis.bash: Type ./login2postgis.bash and you'll be
logged into the containers_postgis_1 container as dbsuper. You
can also log in as one of the users in DB_USERS_TO_CREATE, for
example, ./login2postgis.bash local-elections.login2amazon.bash: Log in to containers_amazon_1 as dbsuper.pull-postgis-raw.bash: This script does
docker cp containers_postgis_1:/home/dbsuper/Raw. That is, it
copies all the files from /home/dbsuper/Raw into Raw in your
current directory, creating Raw if it doesn't exist.
I use this for transferring backup files for testing; I'll create a
database in the PostGIS container, create a backup in
/home/dbsuper/Raw and copy it out to the host with this script. I
create the backups in Raw instead of Backups so they don't get
automatically restored.
push-amazon-raw.bash: This is the next step - it copies Raw to
/home/dbsuper/Raw in containers_amazon_1 for restore testing.You can use the Python virtualenvwrapper utility. See
https://virtualenvwrapper.readthedocs.io/en/latest/# for the
documentation.
To activate, enter
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh.
gitdocker exec as dbsuper as described above.cd /home/dbsuper.configure-git.bash. You'll need to supply your email address
and name../configure-git.bash.To clone a repository, use its https URL. For a private repository,
you'll need to authenticate when you clone. For a public one, you'll
only have to authenticate if you want to push.
In either case, once you've authenticated, git will cache your
credentials for an hour. As you probably noticed, this timeout is
adjustable in configure-git.bash.
Cloning this repository:
docker exec as dbsuper as described above.cd /home/dbsuper../clone-me.bash.You will find the repository in
$HOME/Projects/data-science-pet-containers
localhost, port HOST_POSTGRES_PORT.postgis, port 5432.postgres
and the password is POSTGRES_PASSWORD.postgres, you do not need a password to connect.If you've installed the EnterpriseDB PostgreSQL distribution, you probably already have pgAdmin, although it may not be the latest version. If you want to install pgAdmin without PostgreSQL:
To connect to the postgis service on localhost:HOST_POSTGRES_PORT
with pgAdmin:
Servers and create a server. Give it any name you
want.Connection tab, set the host to localhost, the port to
HOST_POSTGRES_PORT, the maintenance database to postgres, the
user name to postgres and the password to the value you set for
POSTGRES_PASSWORD.Save password box and press the Save button. pgAdmin
will add the tree for the postgis service.When the postgis service first starts, it initializes the database
cluster. After that, it looks in a directory called
/docker-entrypoint-initdb.d/ and restores any .sql or sql.gz files
it finds. Then it looks for .sh scripts and runs them. We use this to
restore databases automatically at startup.
To use this feature:
postgres user will always exist, as will dbsuper and all users
listed in DB_USERS_TO_CREATE.For each database you want restored, create a backup file. For
documentation / repeatability, do this with pg_dump on the command
line or in a script:
pg_dump -Fp -v -C -c --if-exists -d <database> \
| gzip -c > <database>.sql.gzwhere <database> is the name of the database.
At restore time, a new database will be created (-C -c). This is
done by DROPping existing objects; the --if-exists keeps the DROPs
from failing if the objects don't exist.
data-science-pet-containers/containers/Backups. Note that
.gitignore is set for the common backup file extensions -
*.sql.gz, *.sql and *.backup - so these backup files won't be
version-controlled.docker-compose -f postgis.yml build.Docker will copy the backup files into /home/dbsuper/Backups on the
postgis image, and place a script restore-all.sh in
/docker-entrypoint-initdb.d/. The first time the image runs,
restore-all.sh will
DB_USERS_TO_CREATE,
and then.backup, .sql.gz and .sql files it finds in
/home/dbsuper/Backups. Note that .backup files will be restored
to a freshly-created database owned by postgres.Raw directoryIf you want to load raw data onto the postgis image, copy the files to
the data-science-pet-containers/containers/Raw directory. The next
time the image is built they will be copied to /home/dbsuper/Raw.
You can put backup files in Raw but they will not be restored
automatically. See
data-science-pet-containers/examples/reowning_a_database for an
example. Like the backups, these files are not version-controlled.
This service is based on the Anaconda, Inc. (formerly Continuum)
miniconda3 image: https://hub.docker.com/r/continuumio/miniconda3/.
I've added a non-root user jupyter to avoid the security issues
associated with running Jupyter notebooks as "root".
The jupyter user has a Conda environment, also called jupyter. To
keep the image size manageable, I have not installed any data
science tools up front.
To install a (large) data science stack, run the script
/home/jupyter/kitchen-sink.bash as the jupyter user. This will
install
By default the Jupyter notebook server starts when Docker brings up the
service. Type docker logs conatiners_jupyter_1. You'll see something
like this:
$ docker logs conatiners_jupyter_1
[I 08:00:22.931 NotebookApp] Writing notebook server cookie secret to /home/jupyter/.local/share/jupyter/runtime/notebook_cookie_secret
[I 08:00:23.238 NotebookApp] Serving notebooks from local directory: /home/jupyter
[I 08:00:23.238 NotebookApp] 0 active kernels
[I 08:00:23.238 NotebookApp] The Jupyter Notebook is running at:
[I 08:00:23.238 NotebookApp] http://0.0.0.0:8888/?token=d90b23c9368933095c9fd8e25f29d2ba48f7ce67247e216d
[I 08:00:23.238 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 08:00:23.238 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://0.0.0.0:8888/?token=d90b23c9368933095c9fd8e25f29d2ba48f7ce67247e216dBrowse to localhost:8888 and copy/paste the token when the server asks
for it.
gitconfigure-git.bash with the Jupyter notebook file editor.
You'll need to supply your email address and name.New -> Terminal dropdown at the
upper right of the Home tab../configure-git.bash.To clone a repository, use its https URL. For a private repository,
you'll need to authenticate when you clone. For a public one, you'll
only have to authenticate if you want to push.
In either case, once you've authenticated, git will cache your
credentials for an hour. As you probably noticed, this timeout is
adjustable in configure-git.bash.
Cloning this repository:
New -> Terminal dropdown at the
upper right of the Home tab../clone-me.bash.You will find the repository in
$HOME/Projects/data-science-pet-containers
To install packages:
New -> Terminal dropdown at the
upper right of the Home tab.bash. The terminal comes up initially in the sh shell,
which is missing many command-line conveniences.source activate jupyter.conda search to find packages in the Conda ecosystem, then
install them with conda install. You can also install packages
with pip if they're not in the Conda repositories.postgis serviceTo connect to the postgis service, use the user name and maintenance
database name postgres. The host is postgis, the port is 5432 and
the password is the value of POSTGRES_PASSWORD.
Reference: https://drivendata.github.io/cookiecutter-data-science/
New -> Terminal dropdown at the
upper right of the Home tab../make-cookiecutter-project.The script will install cookiecutter in the jupyter environment if
necessary. Then it will launch the Cookiecutter data science interactive
setup to create a new project in /home/jupyter/Projects.
Follow the instructions to set up the project.
This service is based on the rocker/rstudio image from Docker Hub:
https://hub.docker.com/r/rocker/rstudio/. I've added header files so
that the R packages RPostgres, odbc, sf and devtools will
install from source, but there are no R packages on the image besides
those that ship with rocker/rstudio.
Browse to localhost:8787. The user name and password are both
rstudio. Note that if you're using Firefox, you may have to adjust
a setting to use the terminal feature.
Tools -> Global Options -> Terminal.Connect with WebSockets option.gitconfigure-git.bash. You'll need to supply your email address
and name../configure-git.bash.To clone a repository, use its https URL. For a private repository,
you'll need to authenticate when you clone. For a public one, you'll
only have to authenticate if you want to push.
In either case, once you've authenticated, git will cache your
credentials for an hour. As you probably noticed, this timeout is
adjustable in configure-git.bash.
Cloning this repository:
./clone-me.bash.You will find the repository in
$HOME/Projects/data-science-pet-containers
As noted above, to keep the image size down, I've only installed header
files so that the R packages RPostgres, odbc, sf and devtools
will install. That covers the majority of use cases.
However, if you find an R package that won't install because of missing header or other Linux dependency, open an issue at https://github.com/hackoregon/data-science-pet-containers/issues/new.
Most packages that have missing dependencies will list the name of the
Debian packages you need to install. If that's the case, open a root
console with docker exec -it -u root containers_rstudio_1 /bin/bash.
Then type apt install <package-name>. After the Debian package is
installed, you should be able to install the R package.
postgis serviceTo connect to the postgis service, use the user name and maintenance
database name postgres. The host is postgis, the port is 5432 and
the password is the value of POSTGRES_PASSWORD.
This image is based on the Amazon Linux 2 "2-with-sources" Docker image at https://hub.docker.com/_/amazonlinux/. The main reason it's in this collection is to provide a means of restore-testing backup files before handing them off to the DevOps engineers for deployment on AWS.
data-science-pet-containers/containers/Backups.docker-compose -f amazon.yml up -d --build. The backup files will
be copied to /home/dbsuper/Backups on both the postgis and
amazon images.docker logs -f containers_postgis_1. The backup files should be
automatically restored. If there are errors, you'll need to fix your
backup files. When the restores are done, type CTL-C to stop
following the log.amazon container -
docker exec -it -u dbsuper -w /home/dbsuper containers_amazon_1 /bin/bash.cd Backups; ls. You'll see the backup files. For example:
$ cd Backups; ls
odot_crash_data.sql.gz passenger_census.sql.gz restore-all.shThose are the same backup files you just successfully restored in
the postgis image.
./restore-all.sh. This is the same script that did the
automatic restores on postgis and it should have the same result.
If there are no errors in the automatic restore on postgis and the
restore you just did in amazon the backup files are good. To bring
it up, type docker-compose -f amazon,yml up -d --build.This all started with an infamous "cattle, not pets" blog post. For some history, see http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/. In the Red Hat / Kubernetes / OpenShift universe, it's common for people to have a workstation that's essentially a Docker / Kubernetes host with all the actual work being done in containers. See https://rhelblog.redhat.com/2016/06/08/in-defense-of-the-pet-container-part-1-prelude-the-only-constant-is-complexity/ and https://www.projectatomic.io/blog/2018/02/fedora-atomic-workstation/.
So - pet containers for data scientists.