migrate genome context django app onto legumeinfo servers for improved coordination of sites

[adf-ncgr]

we'll need:
python (2.7.5)
with packages
Django (1.5.3)
psycopg2 (2.5.1)
mod_wsgi (4.4.6) for apache

and then something along the lines of this in the apache conf (with the appropriate modifications for the setup on the legumeinfo servers, of course; note that everything rooted beneath /lis_gene_families is from the git repository):

1. Django site
   Alias /chado/static/ /user/legs/web/comparative-legumes.org/lis_gene_families/django/chadotest/static/
   
   Order deny,allow
   Allow from all
   

WSGIScriptAlias /chado /user/legs/web/comparative-legumes.org/lis_gene_families/django/chadotest/apache/wsgi.py
WSGIPythonPath /user/legs/web/comparative-legumes.org/lis_gene_families/django/chadotest
#WSGIScriptAlias /chadotest /user/legs/web/comparative-legumes.org/lis_gene_families/django/chadotest/apache/test.py

Order deny,allow
Allow from all

feel free to delegate any of this back to me that you want, or I could go through a trial install on lis-adf using my root access there. whatever seems like the most expedient path forward...

[LEGUME-320] created by adf_ncgr

[adf-ncgr]

Andrew,

For Django, would version 1.7.7 work? Alternatively, packages exist for 1.4.20 and 1.6.11, but not 1.5.x. I'd prefer to use a packaged version if possible for easy upgrades.

Same for mod_wsgi: a package exists for 3.5, but not 4.4.x.

by nathanweeks

[adf-ncgr]

I imagine those versions will be fine, but I guess we won't know until we try. Let's give it a shot...

by adf_ncgr

[adf-ncgr]

I installed the following packages on lis-dev:

pkg install py27-psycopg2 ap24-mod_wsgi3 py27-django py27-simplejson # see below for why simplejson was installed

And added the following Apache config (note that /chado is already taken, so I used /lis_gene_families for the URL; also, django/chadotest/apache doesn't exist in the git repository; rather, it is django/chadotest/chadotest):

$ cat /usr/local/etc/apache24/Includes/lis_gene_families.conf 
WSGIScriptAlias /lis_gene_families /usr/local/www/lis_gene_families/django/chadotest/chadotest/wsgi.py


  Require all granted


Alias /lis_gene_families/static /usr/local/www/lis_gene_families/django/chadotest/static

static>
  Require all granted


WSGIPythonPath /usr/local/www/lis_gene_families/django/chadotest

Navigating to http://lis-dev.agron.iastate.edu/lis_gene_families resulted in an internal server error, and produced the following error in /var/log/httpd-error.log :

File "/usr/local/www/lis_gene_families/django/chadotest/chado/models.py", line 23, in 
from django.utils import simplejson # for preparing json objects
ImportError: cannot import name simplejson

It looks like simplejson was originally included in a subdirectory. I updated the code to use the "global" version installed via package:

diff --git a/django/chadotest/chado/models.py b/django/chadotest/chado/models.py
index 9da71a1..16524f3 100644
--- a/django/chadotest/chado/models.py
+++ b/django/chadotest/chado/models.py
@@ -20,7 +20,7 @@
 from django.db import models
 from django.db.models import Count
 # import extra stuff
-from django.utils import simplejson # for preparing json objects
+import simplejson # for preparing json objects

 class Db(models.Model):

This no longer produces an internal server error, but rather a (Django-produced) 404. If someone could test this with a "real" URL, that would be great.

I think it will be preferable to run mod_wsgi in daemon mode, but let's get it working with the the simpler "embedded" mode works first.

by nathanweeks

[adf-ncgr]

Thanks Nathan-
under your setup
/lis_gene_families/chado/ is the simplest URL that should work.
I had to make another import simplejson change, but now I'm stuck with
a db connection error:
OperationalError at /chado/

could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

I made changes to django/chadotest/chadotest/settings.py that I think should be appropriate
for your postgres setup, but either I botched it or what seems more likely is that the
change isn't getting picked up without a server restart. If only we were running in daemon mode already!

in any case, when you get a chance can you either restart the apache server and/or see if you can
fix what might be ailing the db connection specification?

thanks!

adf

by adf_ncgr

[adf-ncgr]

I had to set the PGPORT environment variable for Apache:

# cat /usr/local/etc/apache24/envvars.d/postgresql.env 
export PGPORT=$(ifconfig -a | awk '$1 == "inet" { split($2,A,/\./); print (A[3]*2^8 + A[4]); exit }')

This resulted in an error previously seen when importing simplejson, fixed thus:

diff --git a/django/chadotest/d3viz_force_directed/views.py b/django/chadotest/d3viz_force_directed/views.py
index 70a7b6c..616f762 100644
--- a/django/chadotest/d3viz_force_directed/views.py
+++ b/django/chadotest/d3viz_force_directed/views.py
@@ -2,7 +2,7 @@ from chado.models import Organism, Cvterm, Phylotree, Phylonode
 from chado.models import Feature, FeatureCvterm, FeatureRelationship
 from django.shortcuts import render, get_object_or_404, redirect
 from django.http import HttpResponse
-from django.utils import simplejson
+import simplejson
 from django.core.urlresolvers import reverse

 #

Now the "welcome" page displays at http://lis-dev.agron.iastate.edu/lis_gene_families/chado/

Next, I'll work on getting daemon mode configured to make testing changes less cumbersome.

by nathanweeks

[adf-ncgr]

daemon mode seems to be working---at least, I replaced the previous WSGIPythonPath directive with:

WSGIDaemonProcess lis_gene_families python-path=/usr/local/www/lis_gene_families/django/chadotest
WSGIProcessGroup lis_gene_families

then restarted apache, made requests to http://lis-dev.agron.iastate.edu/lis_gene_families/chado/ , noted which httpd process apparently responded to the request (via top & ps), then executed:

touch /usr/local/www/lis_gene_families/django/chadotest/chadotest/wsgi.py

and noted that the aforementioned httpd process no longer existed (indicating that a new process was started to reload the Django application).

I think some paths to static files need to be adjusted in the code somewhere. After everything is working, we can port the code and configuration to lis-stage.

by nathanweeks

[adf-ncgr]

thanks, this sounds good; yes, there are definitely some problems with getting css and js, but I think we're close. Unless you're already poking into it, I will try to find some time to get this resolved today.
Presumably, it will be a good additional test of the daemon mode, to see if these changes "just work" (assuming they are the correct changes!)

by adf_ncgr

[adf-ncgr]

Andrew: I'm not poking around in the code right now, so feel free to have at it. Per my crude understanding, the WSGI application file (wsgi.py) needs to be touch'ed to trigger a reload.

by nathanweeks

[adf-ncgr]

well, that was easy:
chadotest/settings.py
-STATIC_URL = '/static/'
+STATIC_URL = '/lis_gene_families/static/'

working better, but still not quite there; the actual context viewer now giving the infamous:
Fatal error: Unsupported operand types in /usr/local/www/drupal7/includes/theme.inc on line 1088

but I think I know more or less what's going on here (the linked URL isn't quite right, and we're getting kicked into drupal land).

by adf_ncgr

[adf-ncgr]

OK. I verified that the aforementioned Apache configuration allows access to the contents of the static/ directory; e.g.:

http://lis-dev.agron.iastate.edu/lis_gene_families/static/gif/close.png

by nathanweeks

[adf-ncgr]

I've made some additional changes, and as far as I can tell, the lis-dev/lis_gene_families site
is now functioning well. I've committed all the changes to the git repository, and made a branch
legumeinfo-settings to hold the configuration in settings.py that are appropriate for legumeinfo sites;
not sure this is the best way of handling it, maybe you'll have other ideas.

I think the next step will be for you to replicate the setup on lis-stage and then we can
switch to the legumeinfo-settings branch when it's been cloned. Thanks for getting this done, I think
it will make life easier (for some of us, at least!)

by adf_ncgr

[adf-ncgr]

Andrew,

I copied the configuration over to lis-stage; however, while editing a command-line I accidentally reversed the arguments and overwrote the settings.py on lis-dev with the one from lis-stage. Could you please fix the settings.py and verify that it works on lis-stage? Sorry for the extra trouble!

by nathanweeks

[adf-ncgr]

no problem- makes me feel better about having made it live under version control on its own little branch. I've fixed on lis-dev and lis-stage and minimally tested that it seems to be working fine in
both places. That probably means the issue is close-able, but it might be worth getting someone detail-oriented to review it...

by adf_ncgr