deployment process / instructions improvements

[alexlist]

Some feedback regarding the deployment instructions

deploy.py create-config makes invalid assumptions regarding directories...

python active/www/app/deploy.py create_config     -HTTP_Config active/www/app/httpd_config/nodedb_ffgraz_net__443.config -input_encoding=utf-8 > nodedb.ffgraz.net-https.conf

The resulting config contains the homedir of the running user hardcoded in several places.

#!/bin/sh
export PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr
exec /usr/bin/python -O /srv/nodedb/staging/active/www/app/./Command.py fcgi -config
<VirtualHost *:80>
  ServerName None
  DocumentRoot /home/nodedb-deploy/active/www

  AddDefaultCharset     utf-8
  AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript

  Alias /media/GTW/ /home/nodedb-deploy/active/tapyr/_GTW/media/
  Alias /media/v/   /home/nodedb-deploy/active/www/app/media/v/
  Alias /media/     /home/nodedb-deploy/active/www/media/

  <Directory /home/nodedb-deploy/active/tapyr/_GTW/media>
    Order    deny,allow
    Allow    from all
    FileETag all
  </Directory>
[...]

I suggest to make the basedir a config option.

NB: /home is a bad choice, usually services are deployed underneath /srv, and individual users or special purpose non-service accounts like shared deployment users live in /home.

In our case, the basedir is /srv/nodedb/staging

As a workaround, I manually searched and replaced the path.

Though it is nice to have this type of helper script, I wonder why it is necessary to have such a complex virtual host config for a web app, with different permissions and expiry settings for different subdirs.

The apache config is not supposed to change on a regular basis, and almost identical configs could be used for staging vs production. This can be achieved by templates that make this distinction.

Also, it is good practice to separate code owner and code runner. In our case, code is deployed as nodedb-deploy, but run as www-data or another user which has no write access anywhere in the basedir or below.

fcgi is a choice rarely made these days - please add instructions on how to run the app using wsgi

[alexlist]

Also, the file is named something like 443 but sets <VirtualHost *:80>

Let's stick to Apache vhost templates where only very few things need to be changed. I will provide you with the configfiles once they're working.

[alexlist]

Please set the DocumentRoot to somewhere that is not part of the deployed app, but maybe a path that is used for static assets. Separating static assets also allows for separating them on a frontend and e.g. caching them, while traffic to the app is not cached.

[alexlist]

Please note that we did not use virtualenv: the packages that are not available as Debian packages were installed with pip system-wide.

[tanzer]

The config file you specify for deploy.py create_config can (and should) define the right pathes.

httpd_config/nodedb_graz_funkfeuer_at__443.config in this repository contains among others:

config_path     = "~/fcgi/nodedb_graz_funkfeuer_at__443.config"
script_path     = "~/fcgi/nodedb_graz_funkfeuer_at__443.fcgi"

This is just an example and needs to be adapted to your requirements! If you specify a config-file without config_path and script_path, create_config will write to standard output and the pathes will most likely be wrong.

The resulting config contains the homedir of the running user hardcoded in several places. I suggest to make the basedir a config option.

Its is: define root_dir in the file specified for -HTTP_Config or pass -root_dir=/srv/nodedb/staging on the command line

fcgi is a choice rarely made these days - please add instructions on how to run the app using wsgi

If the fcgi script looks like this:

  exec /usr/bin/python -O /srv/nodedb/staging/active/www/app/./Command.py fcgi ...

you need to replace the Command.py fcgi by Command.py wsgi.

[tanzer]

Also, the file is named something like 443 but sets <VirtualHost *:80>

Please show the file you specified for -HTTP_Config. The parameter port defines what goes into VirtualHost.

For the example files, this looks like:

$ grep port httpd_config/*.config

httpd_config/nodedb_graz_funkfeuer_at__443.config:port            = "443"
httpd_config/nodedb_graz_funkfeuer_at__80.config:port            = "80"

[tanzer]

Please set the DocumentRoot to somewhere that is not part of the deployed app, but maybe a path that is used for static assets. Separating static assets also allows for separating them on a frontend and e.g. caching them, while traffic to the app is not cached.

You can do this by moving the appropriate directories and changing the apache config correspondingly.

Start with::

Alias /media/GTW/ /home/nodedb-deploy/active/tapyr/_GTW/media/ Alias /media/v/ /home/nodedb-deploy/active/www/app/media/v/ Alias /media/ /home/nodedb-deploy/active/www/media/

[aaronkaplan]

@tanzer : is there a way to do this right from the beginning (as much automatically) as @alexlist requested? How much effort would this be for you? Ideally the user follows a copy & paste instruction and needs to do nothing else anymore.

[alexlist]

If instructions are copypasta-able, they are most likely also scriptable...

[aaronkaplan]

On Mar 11, 2015, at 4:05 PM, Alexander List notifications@github.com wrote:

If instructions are copypasta-able, they are most likely also scriptable...

that's the idea :)

But @tanzer had a good point: it seems that for a general *nix system, it's not 100% clear where what should be where. Are we using Apache, nginx, etc? This might vary.

However, if we can define a clear and clean Ubuntu / Debian setup, I'd already be happy. Even if it is only copy&paste right now.

[alexlist]

Config for deploy is in file /srv/nodedb/staging/active/www/app/httpd_config/nodedb_ffgraz_net__443.config

root_dir        = "/srv/nodedb/staging"
config_path     = "/srv/nodedb/staging/active/www/app/httpd_config"
host_macro      = "gtw_host_ssl"
port            = "443"
script_path     = "/srv/nodedb/staging/fcgi/nodedb_ffgraz_net__443.fcgi"
server_admin    = "admin@graz.funkfeuer.at"
server_name     = "nodedb.ffgraz.net"
ssl_key_name    = "nodedb.ffgraz.net.key"

Deploy command:

PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr python active/www/app/deploy.py create_config -HTTP_Config active/www/app/httpd_config/nodedb_ffgraz_net__443.config -input_encoding=utf-8

The result is:

nodedb-deploy@nodedb:/srv/nodedb/staging$ PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr python active/www/app/deploy.py create_config -HTTP_Config active/www/app/httpd_config/nodedb_ffgraz_net__443.config -input_encoding=utf-8
#!/bin/sh
export PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr
exec /usr/bin/python -O /srv/nodedb/staging/active/www/app/./Command.py fcgi -config 
<VirtualHost *:80>
  ServerName None
  DocumentRoot /home/nodedb-deploy/active/www

  AddDefaultCharset     utf-8
  AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript

  Alias /media/GTW/ /home/nodedb-deploy/active/tapyr/_GTW/media/
  Alias /media/v/   /home/nodedb-deploy/active/www/app/media/v/
  Alias /media/     /home/nodedb-deploy/active/www/media/

  <Directory /home/nodedb-deploy/active/tapyr/_GTW/media>
    Order    deny,allow
    Allow    from all
    FileETag all
  </Directory>

  <Directory /home/nodedb-deploy/active/www/app/media/v>
    Order    deny,allow
    Allow    from all
  </Directory>

  <Directory /home/nodedb-deploy/active/www/app>
    Order    deny,allow
    Deny     from all
  </Directory>

  <Directory /home/nodedb-deploy/active/www/media>
    Order    deny,allow
    Allow    from all
    FileETag all
  </Directory>

  <Directory /home/nodedb-deploy/active/www>
    DirectoryIndex index.html
  </Directory>

    ExpiresActive On
  <FilesMatch "\.(gif|jpe?g|png)$">
    ExpiresDefault "access plus 1 year"
  </FilesMatch>

  <Directory /home/nodedb-deploy/active/tapyr/_GTW/media/js>
    ExpiresActive On
    ExpiresDefault "access plus 1 week"
  </Directory>

  <Directory /home/nodedb-deploy/active/tapyr/_GTW/media/css>
    ExpiresActive On
    ExpiresDefault "access plus 1 week"
  </Directory>

  <Directory /home/nodedb-deploy/active/www/media/v>
    ExpiresActive On
    ExpiresDefault "access plus 1 year"
  </Directory>

  <Directory /home/nodedb-deploy/active/www/media>
    ExpiresActive On
    ExpiresDefault "access plus 1 day"
  </Directory>

  AddHandler fcgid-script .fcgi
  Options    +ExecCGI

  DefaultMaxClassProcessCount 1
  DefaultMinClassProcessCount 0

  ScriptAliasMatch .* None

  CustomLog ${APACHE_LOG_DIR}/None-80-access.log combined
  ErrorLog  ${APACHE_LOG_DIR}/None-80-error.log
  LogLevel  warn

  UseCanonicalName On
</VirtualHost>

And no fcgi file.

I hate being ignored...

[alexlist]

@aaronkaplan @tanzer

It is a convention at least among DevOps folks I work with to not use /home for services, but rather /srv. Separate code-owner from code-runner. Make only accessible via WWW what needs to be accessible.

Have a shared user as code-owner that is used to deploy new code.

I suggest a layout like this:

/srv/<FQDN>/<deploymenttype>/...

that would look like

/srv/nodedb.ffgraz.net/staging/
/srv/nodedb.ffgraz.net/staging-logs/
/srv/nodedb.ffgraz.net/scripts (maintenance scripts)
/srv/nodedb.ffgraz.net/www (for static content)

under /srv/nodedb.ffgraz.net/staging/ have dirs like nodedb-revnoX, with a symlink from nodedb to the current revno, so it's easy to switch back. You can also separate code and static assets this way. On a large site, you may want to serve the static content from a different vhost, and/or cache it...

Just food for thought, I don't claim this is the only way to do it...

[alexlist]

I now used the "help" function of deploy.py as suggested to discover the parameters and create the fcgi script:

nodedb-deploy@nodedb:/srv/nodedb/staging$ PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr python active/www/app/deploy.py -config local_config/deploy.config fcgi_script > fcgi/nodedb_ffgraz.net__443.fcgi
nodedb-deploy@nodedb:/srv/nodedb/staging$ cat fcgi/nodedb_ffgraz.net__443.fcgi 
#!/bin/sh
export PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr
exec /usr/bin/python -O /srv/nodedb/staging/active/www/app/./Command.py fcgi -config 
nodedb-deploy@nodedb:/srv/nodedb/staging$ chmod +x fcgi/nodedb_ffgraz.net__443.fcgi 

The create_config command creates a httpd config, which I don't want - my httpd config is fine.

The documentation claims it creates the fcgi script as well - that is wrong, fcgi_script has to be used for that.

[alexlist]

nodedb-deploy@nodedb:~$ export PYTHONPATH=/srv/nodedb/staging/active/cndb:/srv/nodedb/staging/active/tapyr
nodedb-deploy@nodedb:~$ /usr/bin/python -O /srv/nodedb/staging/active/www/app/./Command.py fcgi -config /srv/nodedb/staging/local_config/ffg.config 
unexpected indent (<string>, line 2) 

Usage :
    /srv/nodedb/staging/active/www/app/./Command.py [@bundle] [command] ...
        Manage database, run server or WSGI app.

        Possible bundles: @mig1, @mig2

        command                : Cmd_Choice
            Possible values: auth_mig, collect_garbage, create, delete, fcgi, info, load, load_auth_mig, migrate, readonly, run_server, script, setup_cache, shell, version_hash, wsgi

            auth_mig              : Create a migration of the authorization objects
            collect_garbage       : Collect freed IP addresses out of cooldown
            create                : Create database specified by `-db_url`.
            delete                : Delete database specified by `-db_url`.
            fcgi                  : Run as a FastCGI server.
            info                  : Display info about database specified by `-db_url`.
            load                  : Load database specified by `-db_url`.
            load_auth_mig         : Load a migration of the authorization objects
            migrate               : Migrate database specified by `-db_url` to `-target_db_url`.
            readonly              : Change readonly-state of database.
            run_server            : Run as application server.
            script                : Run one or more scripts.
            setup_cache           : Setup the cache of the application.
            shell                 : Open interactive python shell.
            version_hash          : Show version-hashof program or database or both.
            wsgi                  : Run as wsgi application.

        -Auth_Migrate          : Bool
            Migrate authorization objects
        -Engine_Echo           : Bool
            Set the echo flag of the database engine, if appropriate for the backend
        -Pdb_on_Exception      : Bool
            Start python debugger pdb on exception
        -SA_WE_Opt             : SA_WE_Opt
            Turn SA warnings into errors
        -auth_required         : Bool
            Is authorization required?
        -config                : Config
            File(s) specifying defaults for options.
            Config directories: u'/home/nodedb-deploy/', u'/srv/nodedb/staging/active/www/app/../..', u'/home/nodedb-deploy/httpd_config', u'/srv/nodedb/staging/active/www/app/httpd_config', '/srv/nodedb/staging/active/www/app'
        -copyright_start       : Int
            Start of copyright for this application
        -db_name               : Abs_Path
            Default name of database
        -db_url                : Str
            Database url (form: `dialect://user:password@host:port/db_name`)
        -help                  : Help
            Display help about command
        -mig_auth_file         : Abs_Path
            Default name of auth migration file
        -sha                   : SHA
            Name of secure hash algorithm to use.
            Possible values: sha1, sha224, sha256, sha384, sha512
        -verbose               : Bool

        Argument/option bundles of /srv/nodedb/staging/active/www/app/./Command.py
            @mig1
                Migrate from `db_url` to `target_db_url`
                command               : migrate
                target_db_url         : hps:///migration
            @mig2
                Migrate from `target_db_url` to `db_url`
                Auth_Migrate          : yes
                command               : migrate
                db_url                : hps:///migration

        Actual option and argument values of /srv/nodedb/staging/active/www/app/./Command.py
            -Auth_Migrate           = False
            -Engine_Echo            = False
            -Pdb_on_Exception       = False
            -SA_WE_Opt              = False
            -auth_required          = True
            -config                 = .ffg.config
                []
            -copyright_start        = 2012
            -db_name                = /srv/nodedb/staging/active/www/app/ffg
            -db_url                 = hps://
            -help                   = []
            -mig_auth_file          = ~/.AM_ffg.sam
                /home/nodedb-deploy/.AM_ffg.sam
            -sha                    = <Secure_Hash object sha224>
            -verbose                = False
            command                 = None

[alexlist]

fixed leading indent in app config (do we have to be so picky about this?)