search

yeti / manticore-django

Utility functionality for a Manticore Django project
MIT License
5 stars 4 forks source link

readme

manticore-django

Utility functionality for a Manticore Django project

Fabric Script

Fabric script

Manticore-django comes with a fabric script for creating, cloning, and deploying Python Django source code. The script is inspired by Mezzanine's deployment script with additional features. These features allow the script to deploy an application and database server.

Setup for:

Tech Stack

Out of the box tech stack:

Requirements

Prerequisities for running the script:

These requirements should be listed in your requirements/requirements.txt file:

Setup

  1. A valid installation of Python 2.6 or higher must be installed on your computer.

  2. Fabric must be installed: pip install fabric

  3. Create fabric_settings.py in the same directory as manage.py. See next section for details.

  4. If this is a first time installation, the project has to be setup: a. remove Mezzanine's fabric script; b. rename copies of deploy/local_settings.py.template as:

    • deploy/development_settings.py,
    • deploy/production_settings.py,
    • deploy/staging_settings.py d. add deploy/celeryd.conf. See the next section for details.

Your directory structure should look like:

  1. If you make major configuration changes, you should run fab deploy.create:True deploy.deploy.

  2. If you are changing only source code, you should run fab deploy.deploy or fab deploy.deployapp

Usage

After the script is configured properly (see next section), then you can run fabric tasks to setup and redeploy your app.

Before running any of these commands, you'll have to setup your server with a Debian-based operating system and have username+password SSH access.

Here's the command to setup your server the first time:

fab -f path/to/fabfile deploy.all

Subsequently, you can reploy using:

fab -f path/to/fabfile deploy.deploy

Switching Targets

  1. fab deploy.development ...
  2. fab deploy.staging ...
  3. fab deploy.production ...

The development environment (1) is default.

If something isn't working right and you want to know why, pass in a parameter True to these commands.

  1. fab deploy.development:True
  2. fab deploy.staging:True
  3. fab deploy.production:True

Script Configuration

fabric_settings.py

In your fabric_settings.py file:

    FABRIC = {
        "development": {
             "SSH_USER": "", # SSH username
             "SSH_PASS":  "", # SSH password (consider key-based authentication)
             "SSH_KEY_PATH":  "", # Local path to SSH key file, for key-based auth
             "DEPLOY_SSH_KEY_PATH": "", # Local path to an SSH private key that should be installed on the server for accessing remote repositories (optional)
             "DEPLOY_MY_PUBLIC_KEY": "~/.ssh/id_rsa.pub", # Local path to your SSH public key so you can access the server (optional)
             "DEPLOY_DB_CLUSTER_SSH_KEY_PATH":"", # Local path to an SSH private key that will be installed on DATABASE_HOSTS to install with the postgres user
             "APPLICATION_HOSTS": [], # List of hosts to deploy to, public IP addresses
             "DATABASE_HOSTS": [], # List of hosts to deploy to, public IP addresses for SSH access. First entry is master, others are slaves.
             "CRON_HOSTS": [], # Optional list of hosts to run the cron job, public IP addresses, defaults to APPLICATION_HOSTS if not specified
             "PRIVATE_APPLICATION_HOSTS": [], # List of private IP addresses which APPLICATION_HOSTS and CRON_HOSTS communicate with the database, defaults to 127.0.0.1
             "PRIVATE_DATABASE_HOSTS": [], # List of private IP addresses that DATABASE_HOSTS receives connections from the application server, defaults to 127.0.0.1. First entry is master, others are slaves.
             "PRIVATE_CRON_HOSTS": [], # Optional list of private IP addresses for CRON_HOSTS to communicate with the database, default is the same as PRIVATE_APPLICATION_HOSTS
             "VIRTUALENV_HOME":  "", # Absolute remote path for virtualenvs
             "PROJECT_NAME": "", # Unique identifier for project
             "REQUIREMENTS_PATH": "requirements/requirements.txt", # Path to pip requirements, relative to project
             "APT_REQUIREMENTS": ["",""], # Optional list of Debian apt-get packages that are prerequisities for pip packages
             "GUNICORN_PORT": 8000, # Port gunicorn will listen on
             "LOCALE": "en_US.UTF-8", # Should end with ".UTF-8"
             "LIVE_HOSTNAME": "www.example.com", # Host for public site.
             "SITENAME": "Default", # Registered sitename in Django.
             "REPO_URL": "", # Git or Mercurial remote repo URL for the project
             "DB_PASS": "", # Live database password
             "ADMIN_PASS": "", # Live admin user password
             "LINUX_DISTRO": "squeeze", # Linux distribution such as Debian 6.0 (squeeze), 7.0 (wheezy), Ubuntu 10.04 (lucid), Ubuntu 12.04 (precise)
        },
        "staging": {
             ...
        },
        "production" : {
            ...
        }
    }

deploy/development_settings.py, deploy/staging_settings.py, and deploy/production_settings.py

Create server-specific settings for each of the specified targets:

deploy/celeryd.conf

Add deploy/celeryd.conf to your project.

    ; ==============================================
    ;  celery worker supervisor example for Django
    ; ==============================================

    [program:celery_%(proj_name)s]
    command=%(manage)s celery worker --loglevel=INFO
    directory=%(proj_path)s
    environment=PYTHONPATH='%(proj_path)s'
    user=nobody
    numprocs=1
    stdout_logfile=/var/log/%(proj_name)s.celeryd.log
    stderr_logfile=/var/log/%(proj_name)s.celeryd.log
    autostart=true
    autorestart=true
    startsecs=10

    ; Need to wait for currently executing tasks to finish at shutdown.
    ; Increase this if you have very long running tasks.
    stopwaitsecs = 600

    ; if rabbitmq is supervised, set its priority higher
    ; so it starts first
    priority=998

Creating and Cloning Projects with Vagrant

Usage scenario: you can use Pycharm to remotely connect to the Vagrant box for development.

TODO: we need to document this feature of the script

Setting up PyCharm

Prerequisites:

local_settings.py

If you are running the working (non-committed) version of Django from /vagrant/, then you'll have to configure local_settings.py:

    ...

    FABRIC = {
        "vagrant": {
            "SSH_USER": "vagrant", # SSH username
            "SSH_PASS":  "vagrant", # SSH password (consider key-based authentication)
            "SSH_KEY_PATH":  "", # Local path to SSH key file, for key-based auth

            # deployment SSH key
            #"DEPLOY_MY_PUBLIC_KEY": "~/.ssh/id_rsa.pub", # not needed because we are authenticating with username & password
            "DEPLOY_SSH_KEY_PATH": "", # local-file.pub
            #"DEPLOY_DB_CLUSTER_SSH_KEY_PATH": "", # local-file.pem, but not needed for Vagrant

            # Vagrant local box
            "APPLICATION_HOSTS": ['127.0.0.1:2222'], # SSH port for Vagrant is 2222
            "PRIVATE_APPLICATION_HOSTS": ['127.0.0.1'],
            "DATABASE_HOSTS": ['127.0.0.1:2222'],  # SSH port for Vagrant is 2222
            "PRIVATE_DATABASE_HOSTS":['127.0.0.1'],
            "LIVE_HOSTNAME": "127.0.0.1",
            "DB_PASS": "vagrant", # Live database password
            "ADMIN_PASS": "vagrant", # Live admin user password

            # application settings
            "SITENAME": "Default", # your sitename goes here
            "VIRTUALENV_HOME":  "/home/vagrant", # Absolute remote path for virtualenvs
            "PROJECT_NAME": "", # Unique identifier for project
            "REQUIREMENTS_PATH": "requirements/requirements.txt", # Path to pip requirements, relative to project
            "GUNICORN_PORT": 8001, # Port gunicorn will listen on
            "LOCALE": "en_US.UTF-8", # Should end with ".UTF-8"
            "REPO_URL": "", # Git or Mercurial remote repo URL for the project
            "LINUX_DISTRO": "squeeze", # Debian 6
        },
        ... other deployment settings ...
    }

deploy/vagrant_settings.py

This is a copy of local_settings.py that will be pushed to the Vagrant instance. The configuration for this file is:

    DATABASES = {
        "default": {
            # Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
            "ENGINE": "django.db.backends.postgresql_psycopg2",
            # DB name or path to database file if using sqlite3.
            "NAME": "%(proj_name)s",
            # Not used with sqlite3.
            "USER": "%(proj_name)s",
            # Not used with sqlite3.
            "PASSWORD": "%(db_pass)s",
            # Set to empty string for localhost. Not used with sqlite3.
            "HOST": "%(primary_database_host)s",
            # Set to empty string for default. Not used with sqlite3.
            "PORT": "",
        }
    }

    # Mezzanine settings
    SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTOCOL", "https")

    CACHE_MIDDLEWARE_SECONDS = 60

    CACHE_MIDDLEWARE_KEY_PREFIX = "%(proj_name)s"

    # Caches are disabled for vagrant.
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
        }
    }

    # Since Vagrant is running on your computer, you can turn on debugging.
    DEBUG = True

    SESSION_ENGINE = "django.contrib.sessions.backends.cache"

    # Django 1.5+ requires a set of allowed hosts
    ALLOWED_HOSTS = [%(allowed_hosts)s]

    # Celery configuration (if django-celery is installed in requirements/requirements.txt)
    BROKER_URL = 'amqp://%(proj_name)s:%(admin_pass)s@127.0.0.1:5672/%(proj_name)s'

    # We don't need to report any crashes to an outside server.
    RAVEN_CONFIG = {}

    ...

Release Notes

Tested Environments

I tested the script with the following configurations:

Configuration Issues

Deploy Issues

Vagrant Issues