Run script supporting Django

[b-long]

Currently, I can't deploy the master branch (PR #18 changes) as I'm afraid it will break Django. Hoping @dtenenba or @jimhester can review this with me.

[jimhester]

I unfortunately don't have any experience with Django so probably won't be much help. :disappointed:

[b-long]

Ok, in that case I'm going to hold off on https://trello.com/c/GUjlYkBr/37-deploy-spb-master-branch-to-production . There's only a limited amount of time to deploy https://trello.com/c/n44gksrO/35-deploy-latest-bbs-changes-to-all-build-nodes-dev-and-release

[dtenenba]

Can you be more specific about what you think will break? Did it actually break in your testing?

[b-long]

When I attempt to run the manager locally, it fails. Here are two examples :

1) Using the syntax as in biocadmin's crontab on staging.bioconductor.org ( python spb_history/manage.py runserver 0.0.0.0:8000 > manager.log 2>&1 &), my log file contains :

Unhandled exception in thread started by <function wrapper at 0x7fc1799020c8>
Performing system checks...

System check identified no issues (0 silenced).
Traceback (most recent call last):
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 112, in inner_run
    self.check_migrations()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 164, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 19, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 47, in __init__
    self.build_graph()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 182, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
    self.ensure_schema()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 49, in ensure_schema
    if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 162, in cursor
    cursor = self.make_debug_cursor(self._cursor())
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor
    self.ensure_connection()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 119, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 204, in get_new_connection
    conn = Database.connect(**conn_params)
django.db.utils.OperationalError: unable to open database file

2) Using the module syntax (python -m spb_history.manage runserver 0.0.0.0:8000 > manager.log 2>&1 &) , my log file contains :

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/spb_history/manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 303, in execute
    settings.INSTALLED_APPS
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named settings

I did attempt to use the spb_history/settings.py.sqlite and spb_history/spb_history.sqlite files from staging, however there were still errors.

[dtenenba]

I got it to work on my local machine. Make sure settings.py.sqlite is renamed to settings.py and note that it has a hardcoded path to the database (.sqlite file) that needs to be corrected.

[dtenenba]

@b-long Actually, though, when I go to http://localhost:8000 I see that there is still an error, it can't find the module bioconductor.config. Which maybe makes sense since the command was started as python spb_history/manage.py runserver 0.0.0.0:8000 and maybe from there it can't 'see' the bioconductor directory/module. I tried it with the -m syntax:

python -m spb_history.manage. runserver 0.0.0.0:8000

and got ImportError: No module named settings

Not sure exactly how to fix this at the moment.

[b-long]

@dtenenba What happens (in your environment), when you run :

cd spb_history
python -m manage runserver 0.0.0.0:8000

?

[dtenenba]

Aha, good thought. Unfortunately I get the same error. But that makes sense too, because bioconductor is one level up and not really visible to python when you start it like this....

[b-long]

Part of the problem is that in production, we only declare portions of the spb_history tree as modules :

biocadmin@staging:~/packagebuilder (patch/debugging-archiver)$ pwd
/home/biocadmin/packagebuilder
biocadmin@staging:~/packagebuilder (patch/debugging-archiver)$ find . -name "__init__.py"
./spb_history/viewhistory/migrations.old/__init__.py
./spb_history/viewhistory/migrations/__init__.py
./spb_history/viewhistory/__init__.py
./spb_history/__init__.py

This means that (at preset, on production) modules can only be within spb_history or it's subdirectories. My sense is that we have two options:

1. Adjust all modules inside spb_history to use packagebuilder.spb_history.xyz when importing a module
2. Move spb_history to it's own repository and appropriately adjust configuration there.

I think 1 could be simpler in the short-term, but I'm not having success with that. In attempting 2, I've got to the point that Django finds the sqlite file, but then fails with OperationalError: no such table: viewhistory_package (which I presume means that something isn't quite right with my sqlite file). Of course, I could try and bind to the production DB, which may be better. As I assume the team would favor Option 1, I'm going to give that one more try. Here's my attempt at 2 if anyone is interested: https://github.com/b-long/spb_history

[dtenenba]

@b-long Another argument against 1) is that most of the files under spb_history are generated by django, so changing them could break compatibility with future versions.

[b-long]

True, but I think we have that concern to begin with, until we develop more understanding of Django and it's recommended usage (in terms of adding modules, upgrading versions, etc.).

Are you saying you'd prefer me to pursue option 2 ? I'm happy to do that, just want to clarify. Right now, my option 1 implementation is here. I'm running it from the packagebuilder directory :

python -m spb_history.manage runserver 0.0.0.0:8000 > manager.log 2>&1

However, it results in the following error :

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/spb_history/manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
    django.setup()
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/home/blong/Documents/Work/REPOS__git/b-long/packagebuilder/env/local/lib/python2.7/site-packages/django/apps/config.py", line 86, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named viewhistory

[dtenenba]

No, just pointing it out. I'm in favor of whatever option works.

[b-long]

I see, seems reasonable.

[b-long]

@dtenenba I've managed to get everything working, in separate modules :

• spb_history
• package_builder

Do you have time to give these a try? Usage is :

# Ensure you're on the spb_history branch referenced above
cd spb_history
./run-archiver.sh
./run-django.sh
# Ensure you're on the packagebuilder branch referenced above
cd ../packagebuilder
./run-build-server.sh
cd ../spb_history
./rerun-test-build.sh