Document the actual production deploy method

[pndsagora]

Hello,

I have a problem using openfisca-web-api with Apache.

I followed setup instructions

We are using Apache 2.2.29, python 2.7.10 with mod_wsgi 4.5.6.

We use the following apache configuration :

<VirtualHost *:16080>
    <Directory "/soft/openfisca/">
        Options -Indexes -Multiviews +ExecCGI
        Order allow,deny
        Allow from all
    </Directory>
    ErrorLog "/logtmp/log/apache/error_openfisca_log"
    CustomLog "/logtmp/log/apache/access_openfisca_log" common
    LogLevel debug
    WSGIApplicationGroup %{GLOBAL}
    WSGIProcessGroup OpenFisca
    WSGIScriptAlias /openfisca /soft/openfisca/openfisca-web-api/openfisca_web_api/application.py
</VirtualHost>

And the following OpenFisca modules version :

• openfisca-core 4.1.5
• openfisca-france 10.0.0
• openfisca-parsers 1.0.0
• openfisca-web-api 3.0.0

When we try to access http://localhost:16080/openfisca we always get an http 500 error with the following message in logs :

[Tue Feb 07 15:23:41 2017] [info] [client localhost] mod_wsgi (pid=26058, process='OpenFisca', application=''): Loading WSGI script '/soft/openfisca/openfisca-web-api/openfisca_web_api/application.py'.
[Tue Feb 07 15:23:41 2017] [error] [client localhost] mod_wsgi (pid=26058): Target WSGI script '/soft/openfisca/openfisca-web-api/openfisca_web_api/application.py' cannot be loaded as Python module.
[Tue Feb 07 15:23:41 2017] [error] [client localhost] mod_wsgi (pid=26058): Exception occurred processing WSGI script '/soft/openfisca/openfisca-web-api/openfisca_web_api/application.py'.
[Tue Feb 07 15:23:41 2017] [error] [client localhost] Traceback (most recent call last):
[Tue Feb 07 15:23:41 2017] [error] [client localhost]   File "/soft/openfisca/openfisca-web-api/openfisca_web_api/application.py", line 18, in <module>
[Tue Feb 07 15:23:41 2017] [error] [client localhost]     from . import conf, controllers, environment, urls
[Tue Feb 07 15:23:41 2017] [error] [client localhost] ValueError: Attempted relative import in non-package

Is there something wrong with our configuration or is it an openfisca bug ?

[cbenz]

Hello @pndsagora

Thanks for your interest in OpenFisca!

At first sight I would say it's a configuration issue.

In this line:

WSGIScriptAlias /openfisca /soft/openfisca/openfisca-web-api/openfisca_web_api/application.py

you are referencing the wrong application.py file.

This mistake is due to the fact that 2 files are named application.py and poor explanations.

Thanks to your report we're going to update the repository to rename the configuration application.py to wsgi_entry_point.py or something like that.

Please feel free to close the issue if you find this reply satisfying, or answer back :-)

[cbenz]

The right config/application.py file is not committed. It's a generic file like this:

# -*- coding: utf-8 -*-

# Very first: activate the openfisca virtualenv
activate_env = '/home/openfisca/virtualenvs/openfisca/bin/activate_this.py'
execfile(activate_env, dict(__file__=activate_env))

from logging.config import fileConfig
import os

os.environ['PYTHON_EGG_CACHE'] = '/tmp/python-eggs'

from paste.deploy import loadapp

base_dir = os.path.dirname(__file__)
conf_file_path = os.path.join(base_dir, 'paste.ini')
fileConfig(conf_file_path)
application = loadapp('config:{}'.format(conf_file_path))

Please adapt it to your needs.

But today the api.openfisca.fr instance is deployed using a reverse-proxy and gunicorn Python WSGI server.

Sorry, the README.md is not up-to-date.

I created an issue #83

[pndsagora]

Thanks for your quick answer, it's really helpful.

I'm kind of a python noob, I presume you create the file

/home/openfisca/virtualenvs/openfisca/bin/activate_this.py

using Python virtualenv, but is there a way to init app without virtualenv ? We are on a host without Internet connection ... 😭

[cbenz]

We are on a host without Internet connection ... 😭

Are you in a public adminstration? Sorry I couldn't resist ^^

Indeed the virtualenv is just an option.

Just remove those lines:

# Very first: activate the openfisca virtualenv
activate_env = '/home/openfisca/virtualenvs/openfisca/bin/activate_this.py'
execfile(activate_env, dict(__file__=activate_env))

Be sure to name your INI file paste.ini and place it in the same directory.

Whatever, if you're not in a hurry, you can wait for the #83 to be resolved.

[pndsagora]

Are you in a public adminstration? Sorry I couldn't resist ^^

How did you guess 😄

I've removed the 2 lines, then added the paste.ini (just copied development-france.ini) and now I get another error :

[Tue Feb 07 17:33:53 2017] [info] [client localhost] mod_wsgi (pid=27924, process='OpenFisca', application=''): Loading WSGI script '/soft/openfisca/openfisca.wsgi'.
[Tue Feb 07 17:33:53 2017] [error] [client localhost] mod_wsgi (pid=27924): Target WSGI script '/soft/openfisca/openfisca.wsgi' cannot be loaded as Python module.
[Tue Feb 07 17:33:53 2017] [error] [client localhost] mod_wsgi (pid=27924): Exception occurred processing WSGI script '/soft/openfisca/openfisca.wsgi'.
[Tue Feb 07 17:33:53 2017] [error] [client localhost] Traceback (most recent call last):
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/openfisca.wsgi", line 13, in <module>
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     application = loadapp('config:{}'.format(conf_file_path))
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     return loadobj(APP, uri, name=name, **kw)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     global_conf=global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     global_conf=global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     return loader.get_context(object_type, name, global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     section)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     object_type, name=use, global_conf=global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     global_conf=global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     global_conf=global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     return loader.get_context(object_type, name, global_conf)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     object_type, name=name)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_point
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     pkg_resources.require(self.spec)
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 968, in require
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     needed = self.resolve(parse_requirements(requirements))
[Tue Feb 07 17:33:53 2017] [error] [client localhost]   File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 854, in resolve
[Tue Feb 07 17:33:53 2017] [error] [client localhost]     raise DistributionNotFound(req, requirers)
[Tue Feb 07 17:33:53 2017] [error] [client localhost] DistributionNotFound: The 'OpenFisca-Web-API' distribution was not found and is required by the application

I've the following directory tree :

$ ls -al /soft/openfisca/
total 36
drwxr-xr-x 7 admtsof admtsof 4096 Feb  7 17:41 .
drwxr-xr-x 6 root    root    4096 Oct 14 16:50 ..
drwxrwxr-x 5 admtsof admtsof 4096 Feb  7 16:13 openfisca-core
drwxrwxr-x 4 admtsof admtsof 4096 Jan 31 12:32 openfisca-france
drwxrwxr-x 6 admtsof admtsof 4096 Feb  7 16:12 openfisca-parsers
drwxrwxr-x 9 admtsof admtsof 4096 Feb  7 17:41 openfisca-web-api
-rw-r--r-- 1 admtsof admtsof  337 Feb  7 17:31 openfisca.wsgi
-rw-rw-r-- 1 admtsof admtsof 1630 Feb  7 17:41 paste.ini

I've also updated the httpd.conf as following :

WSGIScriptAlias /openfisca /soft/openfisca/openfisca.wsgi

[cbenz]

OK that's getting complicated, given I don't have the full operations log you did.

But I'm working on #83 and you'll just have to follow the instructions.

[scenaristeur]

C'est marrant deux devs français qui se parlent en anglais😉. Dès que ça devient technique, on oublie sa langue maternelle ? Bon OK, ça ne fait pas avancer le pb... Je sors 😖

[pndsagora]

@scenaristeur : Jacques Toubon sors de ce corps !

After RTFM, I've made some progress.

I've installed each modules to site-packages using this command :

for m in openfisca-*; do cd "$m" && /soft/openfisca/python2.7/bin/python setup.py install && cd ..; done;

I had to comment some references to translation files in setup.py otherwise it fails :

# ('share/locale/fr/LC_MESSAGES', ['./openfisca_france/i18n/fr/LC_MESSAGES/openfisca-france.mo']),

My openfisca.wsgi looks like this :

from logging.config import fileConfig
import os

os.environ['PYTHON_EGG_CACHE'] = '/tmp/python-eggs'

from paste.deploy import loadapp

base_dir = os.path.dirname(__file__)
conf_file_path = os.path.join(base_dir, 'paste.ini')
fileConfig(conf_file_path)
application = loadapp('config:{}'.format(conf_file_path))

Now I have the following error :

[Wed Feb 08 10:13:55 2017] [info] mod_wsgi (pid=31270): Attach interpreter ''.
[Wed Feb 08 10:13:55 2017] [info] mod_wsgi (pid=31270): Imported 'mod_wsgi'.
[Wed Feb 08 10:13:59 2017] [error] 10:13:59,948 ERROR [openfisca_core.taxbenefitsystems] Unable to load openfisca variables from file "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_France-10.0.0-py2.7.egg/openfisca_france/scenarios.py"
[Wed Feb 08 10:13:59 2017] [error] [client localhost] mod_wsgi (pid=31004): Target WSGI script '/soft/openfisca/openfisca.wsgi' cannot be loaded as Python module.
[Wed Feb 08 10:13:59 2017] [error] [client localhost] mod_wsgi (pid=31004): Exception occurred processing WSGI script '/soft/openfisca/openfisca.wsgi'.
[Wed Feb 08 10:13:59 2017] [error] [client localhost] Traceback (most recent call last):
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/openfisca.wsgi", line 11, in <module>
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     application = loadapp('config:{}'.format(conf_file_path))
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     return loadobj(APP, uri, name=name, **kw)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     return context.create()
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     return self.object_type.invoke(self)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     return fix_call(context.object, context.global_conf, **context.local_conf)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     val = callable(*args, **kw)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_Web_API-3.0.0-py2.7.egg/openfisca_web_api/application.py", line 71, in make_app
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     environment.load_environment(global_conf, app_conf)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_Web_API-3.0.0-py2.7.egg/openfisca_web_api/environment.py", line 108, in load_environment
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     tax_benefit_system.load_extension(extension)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_Core-4.1.5-py2.7.egg/openfisca_core/taxbenefitsystems.py", line 184, in load_extension
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     self.add_variables_from_directory(extension_directory)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_Core-4.1.5-py2.7.egg/openfisca_core/taxbenefitsystems.py", line 158, in add_variables_from_directory
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     self.add_variables_from_file(py_file)
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_Core-4.1.5-py2.7.egg/openfisca_core/taxbenefitsystems.py", line 143, in add_variables_from_file
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     module = load_module(module_name, *find_module(module_name, [module_directory]))
[Wed Feb 08 10:13:59 2017] [error] [client localhost]   File "/soft/openfisca/python2.7/lib/python2.7/site-packages/OpenFisca_France-10.0.0-py2.7.egg/openfisca_france/scenarios.py", line 11, in <module>
[Wed Feb 08 10:13:59 2017] [error] [client localhost]     from entities import Individu, Famille, FoyerFiscal, Menage
[Wed Feb 08 10:13:59 2017] [error] [client localhost] ImportError: No module named entities

@cbenz : I'll wait till you finish #83

Thanks for your help.

[cbenz]

@pndsagora #85 is merged, may I close this issue? Did you find the documentation useful? Cf https://github.com/openfisca/openfisca-web-api/blob/master/production-config/README.md

[cbenz]

I close it, but you may reopen if you feel the need to :)