search

comses / miracle

Repeatable data analysis workflows for computational models
1 stars 3 forks source link

fabfile not using virtualenv #3

Closed cpritcha closed 9 years ago

cpritcha commented 9 years ago

When I tried to setup the applications with fab initdb (using the virtualenv) I got import errors. When I changed

@task(aliases=['idb', 'initdb'])
def initialize_database_schema():
    """
    Creates the Django DB schema by running makemigrations and then a migrate.
    """
    local('python manage.py makemigrations')
    local('python manage.py migrate')

to


@task(aliases=['idb', 'initdb'])
def initialize_database_schema():
    """
    Creates the Django DB schema by running makemigrations and then a migrate.
    """
    dj('makemigrations')
    dj('migrate')

it made it a bit further. Maybe some of the local should be replaced with dj calls (using the virtualenv). For instance,

@roles('localhost')
@task(alias='te')
def test(name=None, coverage=False):
    if name is not None:
        env.apps = name
    else:
        env.apps = ' '.join(miracle_settings.MIRACLE_APPS)
    if coverage:
        ignored = ['*{0}*'.format(ignored_pkg) for ignored_pkg in env.ignored_coverage]
        env.python = "coverage run --source='.' --omit=" + ','.join(ignored)
    local('%(python)s manage.py test %(apps)s' % env)

does not seem to be using the virtualenv for testing.

alee commented 9 years ago

That's odd, can you include the error messages reported after running fab initdb? You may run into issues if you're using a system fabric instead of the virtualenv fabric, a which fab should report something like /home/<username>/.virtualenvs/miracle/bin/fab

cpritcha commented 9 years ago

I was using the absolute path to fab instead of activating which seems to have caused problems. Output is shown below

vagrant@webserver:/vagrant$ /home/vagrant/.virtualenvs/miracle/bin/fab initdb
[localhost] local: python manage.py makemigrations
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management

Fatal error: local() encountered an error (return code 1) while executing 'python manage.py makemigrations'

Aborting.
local() encountered an error (return code 1) while executing 'python manage.py makemigrations'

When I use the workon command first I get:

vagrant@webserver:~/.virtualenvs$ cd ~/.virtualenvs
vagrant@webserver:~/.virtualenvs$ workon miracle
(miracle)vagrant@webserver:~/.virtualenvs$ cd /vagrant
(miracle)vagrant@webserver:/vagrant$ fab setup
[localhost] Executing task 'setup'
[localhost] Executing task 'setup_postgres'
[localhost] local: createuser miracle -e --createdb -U postgres
createuser: could not connect to database postgres: FATAL:  Peer authentication failed for user "postgres"

Fatal error: local() encountered an error (return code 1) while executing 'createuser miracle -e --createdb -U postgres'

Aborting.
local() encountered an error (return code 1) while executing 'createuser miracle -e --createdb -U postgres'
(miracle)vagrant@webserver:/vagrant$ fab initdb
[localhost] local: python manage.py makemigrations
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
    django.setup()
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/vagrant/miracle/core/models.py", line 35, in <module>
    class DatasetConnectionMixin(object):
  File "/vagrant/miracle/core/models.py", line 38, in DatasetConnectionMixin
    connection = connections['datasets']
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/db/utils.py", line 237, in __getitem__
    self.ensure_defaults(alias)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/db/utils.py", line 170, in ensure_defaults
    raise ConnectionDoesNotExist("The connection %s doesn't exist" % alias)
django.db.utils.ConnectionDoesNotExist: The connection datasets doesn't exist

Fatal error: local() encountered an error (return code 1) while executing 'python manage.py makemigrations'

Aborting.
local() encountered an error (return code 1) while executing 'python manage.py make migrations'

When I ran fab setup originally I got

(miracle)vagrant@webserver:/vagrant$ fab setup
[localhost] Executing task 'setup'
[localhost] Executing task 'setup_postgres'
[localhost] local: createuser miracle -e --createdb -U postgres
createuser: could not connect to database postgres: FATAL:  Peer authentication failed for user "postgres"

Fatal error: local() encountered an error (return code 1) while executing 'createuser miracle -e --createdb -U postgres'

Aborting.
local() encountered an error (return code 1) while executing 'createuser miracle -e --createdb -U postgres'

So I changed my pg_hba.conf file from peer to trust for local connections and restarted postgres. Then I got

(miracle)vagrant@webserver:/vagrant$ fab setup
[localhost] Executing task 'setup'
[localhost] Executing task 'setup_postgres'
[localhost] local: createuser miracle -e --createdb -U postgres
CREATE ROLE miracle NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;
[localhost] local: createdb miracle_metadata -U miracle
[localhost] local: createdb miracle_data -U miracle
[localhost] local: python manage.py makemigrations
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
    django.setup()
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/vagrant/miracle/core/models.py", line 35, in <module>
    class DatasetConnectionMixin(object):
  File "/vagrant/miracle/core/models.py", line 38, in DatasetConnectionMixin
    connection = connections['datasets']
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/db/utils.py", line 237, in __getitem__
    self.ensure_defaults(alias)
  File "/home/vagrant/.virtualenvs/miracle/local/lib/python2.7/site-packages/django/db/utils.py", line 170, in ensure_defaults
    raise ConnectionDoesNotExist("The connection %s doesn't exist" % alias)
django.db.utils.ConnectionDoesNotExist: The connection datasets doesn't exist

Fatal error: local() encountered an error (return code 1) while executing 'python manage.py makemigrations'

Aborting.
local() encountered an error (return code 1) while executing 'python manage.py makemigrations'

Any suggestions?

alee commented 9 years ago

Ah, you need to create a settings/local.py file (same as the .travis.yml file)

cp miracle/settings/local.py.example miracle/settings/local.py

cpritcha commented 9 years ago

I forgot about that. Sorry. Here's the output for fab setup now. It works! Except for "/opt/miracle/data/" not being created. Do I need to change any permissions?

vagrant@webserver:/vagrant$ cp miracle/settings/local.py.example miracle/settings/local.py
(miracle)vagrant@webserver:/vagrant$ fab setup
MEDIA_ROOT path '/opt/miracle/data/' does not exist, trying to create it now.
Unable to create path /opt/miracle/data/, user uploads will not work properly.
[localhost] Executing task 'setup'
[localhost] Executing task 'setup_postgres'
[localhost] local: createuser miracle -e --createdb -U postgres
CREATE ROLE miracle NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;
[localhost] local: createdb miracle_metadata -U miracle
[localhost] local: createdb miracle_data -U miracle
[localhost] local: python manage.py makemigrations
MEDIA_ROOT path '/opt/miracle/data/' does not exist, trying to create it now.
Unable to create path /opt/miracle/data/, user uploads will not work properly.
Migrations for 'core':
  0001_initial.py:
    - Create model Analysis
    - Create model Author
    - Create model BookmarkedProject
    - Create model Dataset
    - Create model DataTable
    - Create model DataTableColumn
    - Create model MiracleUser
    - Create model Project
    - Add field project to dataset
    - Add field published_by to dataset
    - Add field project to bookmarkedproject
    - Add field user to bookmarkedproject
    - Add field authors to analysis
    - Add field project to analysis
    - Alter unique_together for bookmarkedproject (1 constraint(s))
[localhost] local: python manage.py migrate
MEDIA_ROOT path '/opt/miracle/data/' does not exist, trying to create it now.
Unable to create path /opt/miracle/data/, user uploads will not work properly.
Operations to perform:
  Synchronize unmigrated apps: staticfiles, django_extensions, messages, bootstrap3
  Apply all migrations: core, sessions, admin, auth, default, contenttypes
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying core.0001_initial... OK
  Applying default.0001_initial... OK
  Applying default.0002_add_related_name... OK
  Applying default.0003_alter_email_max_length... OK
  Applying sessions.0001_initial... OK

Done.
cpritcha commented 9 years ago

How about the setup for pg_hba.conf? How are connections setup are your machine?

alee commented 9 years ago

On my local dev box I use trust permissions in pg_hba.conf

I can alter the fabfile to sudo create /opt/miracle/data but that's not really needed yet until I wire up dataset uploading, so you should be able to run tests, the django devserver, and so on now via fab test or fab server. Let me know if you have any other issues and please feel free to adjust the instructions in README.md to properly set up the dev environment.