create ckan Dockerfile

[senderle]

Our first goal is to create a Dockerfile capable of building an image compatible with the existing Postgres and SOLR servers. To that end, we've created a digest of the configuration history for the serve (see below). This is not by any means a clean configuration recipe. But it will serve, we hope, as the basis for a clean recipe, at least for people who know CKAN well enough.

It may be that most of what we do here is already done in the standard built-in CKAN Dockerfiles. Or it may be that some of these steps are unnecessary or redundant or wrong. Please feel free to correct, but do understand that our first priority is compatibility with the existing database and index.

Also, we did our best to scrub passwords and secrets. If we missed something, please let us know immediately.

sudo apt-get update
sudo apt-get install -y nginx apache2 libapache2-mod-wsgi libpq5 redis-server git-core
wget http://packaging.ckan.org/python-ckan_2.6-trusty_amd64.deb
sudo dpkg -i python-ckan_2.6-trusty_amd64.deb
service apache2 reload
sudo aptitude install postgresql-client
sudo vi /etc/ckan/default/production.ini
createuser -S -D -R -P -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin ckan_default
psql -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin ckan_default
sudo ckan db init
sudo ckan datastore set-permissions
sudo mkdir -p /var/lib/ckan/default
ckan.storage_path = /var/lib/ckan/default
sudo chown www-data /var/lib/ckan/default
sudo chmod u+rwx /var/lib/ckan/default
createdb -O ckan_default datastore_default -E utf-8 -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com
createuser -S -D -R -P -l -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin datastore_default
createdb -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin -O ckan_default datastore_default -E utf-8 
createdb -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin datastore_default -E utf-8 
psql -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin datastore_default
sudo ckan datastore set-permissions | psql -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com:5432 -U datarefuge_admin --set ON_ERROR_STOP=1
sudo ckan datastore set-permissions | psql -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin --set ON_ERROR_STOP=1
sudo ckan datastore set-permissions
sudo ckan datastore set-permissions | psql -h datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com -U datarefuge_admin datastore_default --set ON_ERROR_STOP=1

curl -O https://github.com/DataShades/ckanext-cloudconnector/archive/master.zip
file master.zip 
git clone https://github.com/DataShades/ckanext-cloudconnector.git
cd ckanext-cloudconnector/
. /usr/lib/ckan/default/bin/activate
python setup.py install 
deactivate 

psql --host=datarefuge.ckd3w35yzltq.us-east-1.rds.amazonaws.com --port=5432 --username=ckan_default --password=[something] --dbname=datarefuge -i datarefugeckan0.pem
paster search-index rebuild --config=/etc/ckan/default/production.ini

paster sysadmin add [user] -c /etc/ckan/default/production.ini

cd /usr/lib/ckan/default/src
paster --plugin=ckan create -t ckanext datarefuge_theme
paster --plugin=ckan create -t ckanext ckanext-datarefuge-theme
sudo paster --plugin=ckan create -t ckanext ckanext-datarefuge-theme

cd /usr/lib/ckan/default/src/ckan
/usr/lib/ckan/default/bin/paster db clean --config=/etc/ckan/default/production.ini
/usr/lib/ckan/default/bin/paster db init --config=/etc/ckan/default/production.ini
/usr/lib/ckan/default/bin/paster sysadmin --config=/etc/ckan/default/production.ini
/usr/lib/ckan/default/bin/paster sysadmin add sharp --config=/etc/ckan/default/production.ini

vi /etc/nginx/sites-enabled/
vi /etc/nginx/sites-enabled/ckan
vi /etc/nginx/sites-enabled/ckan_secure 

git clone git@github.com:DataShades/ckanext-cloudconnector.git
git clone https://github.com/DataShades/ckanext-cloudconnector.git
cd ckanext-cloudconnector/
python setup.py  install 
vi /etc/ckan/default/production.ini 
service apache2 restart

paster cloudstorage initdb -c /etc/ckan/default/production.ini
pip install polib

cd lib/python2.7/site-packages/ckanext_cloudstorage-0.1.0-py2.7.egg/
cd src/ckan
paster cloudstorage initdb -c /etc/ckan/default/production.ini

pip install docopt
pip install ckanapi
paster cloudstorage -c /etc/ckan/default/production.ini 
paster cloudstorage initdb -c /etc/ckan/default/production.ini 

sudo wget https://github.com/azavea/ckanext-odp_theme/archive/develop.zip

[dcwalk]

Hey @senderle this all looks good, I'm just wondering what the status is on Dockerfile creation? Would love to start getting interested people helping with DataRefuge.org but not sure what the status is. I've been tracking steps for coordinating here: https://github.com/edgi-govdata-archiving/overview/issues/45

Can you let me know if you have specific needs etc, and I can mention them at our standup? Alternatively would love to have you attend and speak directly to any issues! (https://github.com/edgi-govdata-archiving/overview#development)

[senderle]

Hi @dcwalk, thanks! I set this up as a starting point for containerization but I believe the task has been taken up in a different workflow since then. I don't want to lead anybody to duplicate work unnecessarily.

Assuming they're doing exactly this, I would expect that we could just fork a repo from them here. But I'm out on the periphery of this project and am just taking up odd jobs when I have a moment, so I'm not 100% knowledgeable.

Let me check on the details and get back to you.

[senderle]

OK, @dcwalk the word is that they're just getting started, and we're going to hear more from them about their plans at the end of the week. I'll ping this thread again when we know details. Hopefully we'll be able to synch up everyone's efforts on this front here, but if not we'll figure out an alternative.

Thanks for checking in! Feel free to ping me again if you haven't heard anything by next Monday.

[dcwalk]

Thanks for the update!

[senderle]

Most recent word is that they are going with something other than Docker. (Unfortunately!) Still short on details. I will continue to post updates here as they come.