= Introduction = This is the full MiG project code released at the MiGrid project at Google Code:
http://code.google.com/p/migrid/
MiG is Free Software and it is developed by the MiG Project lead by Brian Vinter (brian DOT vinter AT nbi DOT ku DOT dk).
Please refer to the COPYING file in this directory for further information about the GPL v2 license under which MiG is distributed.
= Getting Started = Please refer to the information available at the aforementioned URL especially the wiki pages including:
https://code.google.com/p/migrid/wiki/GettingStarted
= Requirements = A MiG server basically requires an Apache web server, the OpenSSH client tools and a Python interpreter with a few external modules. -Apache 1.3.x or 2.x (http://httpd.apache.org/) -Apache SSL module (http://httpd.apache.org/docs/current/mod/mod_ssl.html) -Apache proxy module (http://httpd.apache.org/docs/current/mod/mod_proxy.html) -Apache Rewrite module (http://httpd.apache.org/docs/current/mod/mod_rewrite.html) -OpenSSH clients (http://www.openssh.org/) -Python 2.6 or later but not 3.x (https://www.python.org/) -Python Enchant module (https://pypi.python.org/pypi/pyenchant).
Optional WSGI interface, OpenID login, instant messaging service, efficient file access services, event handler service, spell checking, Wiki, SCM and tracker VGrid features require: -Apache WSGI module (https://code.google.com/p/modwsgi/) -Apache OpenID auth module (http://findingscience.com/mod_auth_openid/) -Python OpenID module (https://github.com/openid/python-openid) -Python pbkdf2 module (https://pypi.python.org/pypi/pbkdf2) -Python irclib module (https://pypi.python.org/pypi/python-irclib/) -Python Paramiko module (https://pypi.python.org/pypi/paramiko/) -Python FTPD library (https://pypi.python.org/pypi/pyftpdlib) -Python WSGI WebDAV module (http://wsgidav.readthedocs.org) -Python watchdog module (https://pypi.python.org/pypi/watchdog) -Python Enchant module (https://pypi.python.org/pypi/pyenchant). -Mercurial (http://mercurial.selenic.com/) -Trac (http://trac.edgewall.org/) -Trac plugins (e.g. http://trac-hacks.org/)
On Debian/Ubuntu servers the corresponding basic packages can be installed with: sudo aptitude install apache2-mpm-prefork openssh-client python \ python-pip python-setuptools
and most of the optional dependencies similarly with: sudo aptitude install libapache2-mod-wsgi libapache2-mod-auth-openid \ python-pbkdf2 python-irclib python-paramiko \ python-enchant mercurial trac trac-mercurial
We highly recommend installing the optional wsgidav and openid modules directly from upstream, because packaged versions are generally outdated even on pypi. The packaged openid module lacks some security fixes and the wsgidav one lacks a fix for upload/write access for OSX clients. They can easily be installed with pip: sudo pip install https://github.com/openid/python-openid/archive/master.zip sudo pip install https://github.com/mar10/wsgidav/archive/master.zip
Additional packaged Trac extensions can be installed with: sudo aptitude install trac-customfieldadmin trac-graphviz \ trac-mastertickets trac-wikiprint trac-wikirename trac-wysiwyg
and the unpackaged ones can be grabbed from trac-hacks.org with pip and easy_install: sudo easy_install http://trac-hacks.org/svn/wikicssplugin sudo easy_install http://trac-hacks.org/svn/fullblogplugin sudo easy_install http://trac-hacks.org/svn/discussionplugin/0.11 sudo easy_install http://trac-hacks.org/svn/tracpasteplugin sudo easy_install http://trac-hacks.org/svn/downloadsplugin sudo pip install TracStats
where 0.11 in the URLs may need to be changed to fit your particular version of Trac. Please note that the source code stats in the TracStats plugin do not currently work for Mercurial repositories! Additional plugins are available from http://trac-hacks.org/
The downloads plugin currently needs patching to work. Please refer to the notes in the [downloads] section of the generated MiG trac.ini file.
Please note that there may be subtle internal plugin dependencies and conflicts that e.g. can cause problems if plugins are not loaded in the right order. We have seen database upgrade problems if FullBlog, Watchlist and Discussion are enabled but not loaded in an order where Watchlist is loaded in between the other two. Similar issues appeared when we enabled the Pastebin and Downloads plugins in one step. It was necessary to either patch tracdownloads/db/db1.py to ignore errors for existing tables or to enable one plugin and upgrade all Trac environments before enabling the other plugin and upgrading again. Thus you may have to experiment with the installed plugins in a conservative way.
With the inclusion of Trac we no longer rely on the MoinMoin software for stand-alone wikis.
The optional grid_ftps daemon requires the pyftpdlib module in a recent version, so it is easiest to install it with pip: sudo pip install pyftpdlib
The optional grid_webdavs daemon requires the wsgidav module in a recent version, so it is easiest to install it with pip as mentioned above.
The now deprecated optional grid_davs daemon requires the pywebdav module in a recent version, so it is easiest to install it with pip: sudo pip install pywebdav
The optional grid_openid daemon requires the openid module in a recent version, so it is easiest to install it with pip as mentioned above.
The optional grid_events daemon requires the watchdog module which may be installed with: sudo pip install watchdog
= Installing MiG = If you want to run your own MiG server for your own grid or to develop MiG you should download and unpack the source code (including this file) on a UNIX compatible computer as described below.
The MiG core services are provided by the MiG daemons from the mig/server directory and they can simply be run directly from the unpacked source code directory when a suitable server configuration is added.
For the web interfaces to work you will need to run an apache server as described in the mig/install directory. Grid job handout relies on OpenSSH client commands like ssh and scp. MiG does not include the actual Apache web server or OpenSSH clients, so you will need to install those using either packages provided by your distribution or install it from source.
MiG is tested on Debian/Ubuntu and Redhat Linux using Apache 1.3 or 2.X with mod-ssl respectively but other distribution and apache combinations should also work. MiG relies on apache's mod-ssl for automatic certificate validation and access control. Furthermore quite a bit of rewrite rules are used for access and convenience so the mod-rewrite apache module is required too.
You can read more about the apache configuration in the provided mig/install/README.Debian file.
This server documentation expects the MiG code to run as a separate 'mig' user on the UNIX system, but this is not a requirement. Just modify your apache and MiG configurations appropriately if you want to run MiG as a different user or with other paths. It is important to configure apache so that the MiG web interfaces can read and write the files created by the MiG daemons and vice versa. This may require extra care if the MiG installation and apache runs as different system users. If you use the default setup you do not need to worry about this.
As root you can create an ordinary user, mig, for running the MiG server:
Login as the new user:
To avoid other processes from tampering it is a good idea to set either the permissions on the entire mig user home very restrictively:
or at least set the umask tight enough to avoid unauthorized access to the MiG server files. If you run MiG with different apache and mig users, you will most likely need to provide both users write access to the mig user home, though.
Download and unpack the MiG source or make a checkout from svn as described on: https://code.google.com/p/migrid/source/checkout
At this point it may be comfortable to copy some of the basic account configuration files from mig/install/mig-user to ~/ but this is not required.
Now you are ready to actually configure your installation. The easiest way to do that is to use the configuration generator in mig/install/generateconfs.py to create configurations that match your setup. For the default settings it could just be done as: cd mig/install/ ./generateconfs.py
If your setup uses custom paths or settings just provide them on the commandline like the command help indicates: ~/mig/install > ./generateconfs.py -h Usage: ./generateconfs.py [OPTIONS] Where supported options include -h/--help for this help or the conf settings: --source=SOURCE --destination=DESTINATION --public_fqdn=PUBLIC_FQDN --cert_fqdn=CERT_FQDN --oid_fqdn=OID_FQDN --sid_fqdn=SID_FQDN --user=USER --group=GROUP --apache_etc=APACHE_ETC --apache_run=APACHE_RUN --apache_lock=APACHE_LOCK --apache_log=APACHE_LOG --mig_code=MIG_CODE --mig_state=MIG_STATE --mig_certs=MIG_CERTS --enable_sftp=ENABLE_SFTP --enable_davs=ENABLE_DAVS --enable_ftps=ENABLE_FTPS --enable_wsgi=ENABLE_WSGI --enable_sandboxes=ENABLE_SANDBOXES --enable_vmachines=ENABLE_VMACHINES --enable_freeze=ENABLE_FREEZE --enable_hsts=ENABLE_HSTS --enable_vhost_certs=ENABLE_VHOST_CERTS --enable_openid=ENABLE_OPENID --openid_providers=OPENID_PROVIDER --daemon_keycert=DAEMON_KEYCERT --alias_field=ALIAS_FIELD --hg_path=HG_PATH --hgweb_scripts=HGWEB_SCRIPTS --trac_admin_path=TRAC_ADMIN_PATH --trac_ini_path=TRAC_INI_PATH --public_port=PUBLIC_PORT --cert_port=CERT_PORT --oid_port=OID_PORT --sid_port=SID_PORT --user_clause=USER_CLAUSE --group_clause=GROUP_CLAUSE --listen_clause=LISTEN_CLAUSE --serveralias_clause=SERVERALIAS_CLAUSE --distro=DISTRO
For one of our servers running MiG as the 'mig' user with the code checked out directly in the home directory and with Debian apache 1.3.x defaults, this could be: ./generateconfs.py --source=. --destination=generated-confs \ --public_fqdn=mig-1.imada.sdu.dk \ --cert_fqdn=mig-1.imada.sdu.dk \ --sid_fqdn=mig-1.imada.sdu.dk --user=mig \ --group=mig --apache_etc=/etc/apache \ --apache_run=/var/run/apache \ --apache_lock=/var/lock/apache \ --apache_log=/var/log/apache \ --mig_code=/home/mig/mig \ --mig_state=/home/mig/state \ --mig_certs=/home/mig/MiG-certificates \ --hg_path=/usr/bin/hg \ --hgweb_scripts=/usr/share/doc/mercurial/examples \ --trac_admin_path=/usr/bin/trac-admin \ --trac_ini_path=/home/mig/mig/server/trac.ini \ --public_port=80 --cert_port=443 --sid_port=8092 \ --user_clause=User --group_clause=Group \ --listen_clause='Listen' \ --serveralias_clause='#ServerAlias'
or similarly with Debian apache 2.x without OpenID: ./generateconfs.py --source=. --destination=generated-confs \ --base_fqdn=migrid.org \ --public_fqdn=www.migrid.org \ --cert_fqdn=dk-cert.migrid.org \ --sid_fqdn=dk-sid.migrid.org --user=mig --group=mig \ --apache_etc=/etc/apache2 \ --apache_run=/var/run/apache2 \ --apache_lock=/var/lock/apache2 \ --apache_log=/var/log/apache2 \ --mig_code=/home/mig/mig \ --mig_state=/home/mig/state \ --mig_certs=/etc/apache2/MiG-certificates \ --hg_path=/usr/bin/hg \ --hgweb_scripts=/usr/share/doc/mercurial-common/examples \ --trac_admin_path=/usr/bin/trac-admin \ --trac_ini_path=/home/mig/mig/server/trac.ini \ --public_port=80 --cert_port=443 --sid_port=443 \ --enable_openid=False --enable_wsgi=True --enable_sftp=True \ --enable_sandboxes=True --enable_vmachines=True \ --user_clause=User --group_clause=Group \ --listen_clause='#Listen' \ --serveralias_clause='ServerAlias' or the same with OpenID, HSTS, WSGI and vhost-specific certificates: ./generateconfs.py --source=. --destination=generated-confs \ --base_fqdn=migrid.org \ --public_fqdn=www.migrid.org \ --cert_fqdn=dk-cert.migrid.org \ --sid_fqdn=dk-sid.migrid.org \ --oid_fqdn=dk-oid.migrid.org \ --user=mig --group=mig \ --apache_etc=/etc/apache2 \ --apache_run=/var/run/apache2 \ --apache_lock=/var/lock/apache2 \ --apache_log=/var/log/apache2 \ --mig_code=/home/mig/mig \ --mig_state=/home/mig/state \ --mig_certs=/etc/apache2/MiG-certificates \ --hg_path=/usr/bin/hg \ --hgweb_scripts=/usr/share/doc/mercurial-common/examples \ --trac_admin_path=/usr/bin/trac-admin \ --trac_ini_path=/home/mig/mig/server/trac.ini \ --public_port=80 --cert_port=443 --oid_port=443 --sid_port=443 \ --enable_openid=True \ --openid_providers='https://openid.ku.dk/ https://dk-oid.migrid.org:8443/openid/' \ --enable_wsgi=True --enable_sftp=True \ --enable_davs=True --enable_ftps=True \ --enable_sandboxes=True --enable_vmachines=True \ --enable_vhost_certs=True \ --enable_hsts=True --user_clause=User --group_clause=Group \ --listen_clause='#Listen' \ --serveralias_clause='ServerAlias' --alias_field=email \ --daemon_keycert=~/certs/combined.pem \ --daemon_pubkey=~/certs/combined.pub \ --landing_page='/wsgi-bin/dashboard.py' \ --skin=migrid-basic
and a storage-only setup with CentOS 6.x, apache 2.x and OpenID login: ./generateconfs.py --source=. --destination=generated-confs \ --base_fqdn=erda.dk \ --public_fqdn=www.erda.dk \ --cert_fqdn=cert.erda.dk \ --oid_fqdn=erda.dk \ --sid_fqdn=sid.erda.dk --user=mig --group=mig \ --apache_etc=/etc/httpd \ --apache_run=/var/run/httpd \ --apache_lock=/var/lock/subsys/httpd \ --apache_log=/var/log/httpd \ --mig_code=/home/mig/mig \ --mig_state=/home/mig/state \ --mig_certs=/etc/httpd/MiG-certificates \ --hg_path=/usr/bin/hg \ --hgweb_scripts=/usr/share/doc/mercurial-1.4 \ --trac_admin_path='' --trac_ini_path='' \ --public_port=80 --cert_port=443 --oid_port=443 --sid_port=443 \ --openid_providers='https://openid.ku.dk/ https://erda.dk:8000/openid/' \ --enable_openid=True --enable_wsgi=True --enable_sftp=True \ --enable_davs=True --enable_ftps=True --enable_hsts=True \ --enable_sandboxes=False --enable_vmachines=False \ --enable_freeze=True --enable_vhost_certs=True \ --user_clause=User --group_clause=Group \ --listen_clause='#Listen' --serveralias_clause='#ServerAlias' \ --alias_field=email --daemon_keycert=~/certs/combined.pem \ --daemon_pubkey=~/certs/combined.pub \ --distro=centos --landing_page='/wsgi-bin/fileman.py' \ --skin=erda-basic
Most of the arguments should be relatively straight forward, but you need to provide the MIG_CERTS path where your apache server key and certificates are available along with optional MiG x509 server certificates (used for MiG server to server communication). The actual keys and certificates can be added later, so you can just choose a suitable directory path at first.
The hg and trac path pairs are optional and can be set to the empty string if mercurial/trac is not available or if VGrid wikis, SCMs and trackers should simply not be enabled. If you want VGrid trackers including mercurial integration, but don't want the direct VGrid SCM links, you can set the trac_X and hg_X options but leave out the scm entry in the ordered list of vgrid_links in the SITE section. The same procedure applies for visibility of the other VGrid components.
Similarly the mercurial package provides all required components for VGrid SCMs on Debian/Ubuntu. The same applies for the trac + trac-mercurial packages. Paramiko is required for the optional grid_sftp daemon to work and the python-paramiko package provides all required components for it on Debian/Ubuntu. Python FTP server library (pyftpdlib) in a recent version (1.x) is required for the optional grid_ftps daemon to work and the python-pyftpdlib package provides all required components for it on recent Debian/Ubuntu. If no recent version is available, it can still easily be installed with pip instead. WsgiDAV is required for the optional grid_webdavs daemon to work and a recent version is needed for full OSX client support. Thus it is recommended to install directly from github or with pip. PyWebDAV is required for the now deprecated optional grid_davs daemon to work and the python-webdav package in a recent version may provide all required components for it on Debian/Ubuntu. If no recent version is available, it can still easily be installed with pip instead. All optional file server services like sftp, ftps and davs rely on the python pbkdf2 module for password auth support and the python-pbkdf2 package provides all required components for it on Debian/Ubuntu.
The four CLAUSE arguments can be used to comment out the explicit setting of user, group, serveralias and ports in the apache conf by providing a '# User', '# Group', '# ServerAlias' and '# Listen'. This is mostly relevant if using apache2 with WSGI.
The generator will inform you about the steps to install your configuration files in the right locations.
== Running a MiG Server == Before you run the MiG daemons you need to have a working configuration for your daemons in mig/server/MiGserver.conf or another location you can specify in the MIG_CONF environment variable. Please note that if you want to use this environment variable, it must be available to all MiG components to work. You can use the generator as mentioned above or manually modify e.g. the localhost example configuration in MiGserver-localhost.conf. At any time you can verify the validity of your configuration with the checkconf.py script in the same directory.
The central daemon is grid_script.py which takes care of all job management on the server. If you want to include grid monitor web pages you should additionally run the grid_monitor.py daemon. The optional job notifications and ssh multiplexing daemons are available as grid_imnotify.py / grid_imnotify_stdout.py and grid_sshmux.py in the same location. In case you don't know what they do, you can most likely safely ignore them and just run the grid_script.py daemon.
All the daemons can be launched from inside the mig/server directory: cd /path/to/unpacked/mig/source/mig/server python grid_script.py
Alternatively they can be launched from other locations as long as the configuration path is provided in the environment: export MIG_CONF="/path/to/MiGserver.conf" python /path/to/unpacked/mig/source/mig/server/grid_script.py
Each daemon will keep running until you actively stop it, so you need individual shell sessions for each daemon.
For testing purposes this interactive execution is fine, but in more permanent setups you will probably want to run the MiG daemons as true daemons so that you can disconnect from the server and leave them running. The easiest solution to that problem is to run the daemons inside a GNU Screen session: screen -S MiG cd /path/to/unpacked/mig/source/mig/server python grid_script.py [ctrl-a d to disconnect]
Then you can disconnect from the server and resume the session any time later by reattaching the screen session: screen -S MiG -R
Please refer to 'man screen' or other screen documentation for further details.
== Adding users == You need a MiG certificate+key to fully interact with any MiG server as a user. If you don't want to set up your on Certificate Authority (CA) you can use e.g. our certificates. Please use the certificate request link from http://www.migrid.org if you haven't got a certificate yet.
If you run your own CA you can simply use the certificate request mechanism included in MiG (https://server/cgi-sid/reqcert.py) to combine certificate and user creation. Certificate requests will automatically result in an email with full certificate and MiG user creation instructions to the configured MiG admins.
Otherwise you can use the external certificate sign up request mechanism included in MiG (https://server/cgi-sid/extcert.py) or simply run the MiG user creation commands directly as described below.
To manually add a user to your MiG server you need to look up the certificate fields and run the createuser script:
cd ~/mig/server ./createuser.py
You will be prompted for user details one by one before the user is added to the local MiG user database. Any user added to this database can access your MiG server and manage his/her MiG jobs and files. The user must present a MiG certificate with the exact same Distinguished Name to get access, however. If you do not use the MiG CA or another CA with the same Distinguished Name format (/C=./ST=./O=./CN=./emailAddress=.*) you have to supply the -i DN option for the user to work.
Example: adding myself as a user on a MiG server:
Please enter the details for the new user: Full Name: Jonas Bardino Organization: DIKU State: 2-letter Country Code: DK Email: bardino@diku.dk Comment: This is my own MiG user Password: using user dict: {'comment': 'This is my own MiG user', 'country': 'DK', 'state': '', 'full_name': 'Jonas Bardino', 'organization': 'DIKU', 'password': '*****==', 'email': 'bardino@diku.dk'} logging to: server.log ; level: info Creating dirs and files for new user: Jonas Bardino User name without spaces: Jonas_Bardino
User Jonas Bardino was successfully added to user DB! DB entry and dirs for Jonas Bardino were created or updated
My MiG certificate with Full Name Jonas Bardino and so on will now give me access to this development server.
You can find the field details using openssl or by viewing the certificate imported in a browser.
== Adding resources == When you have added yourself as a user on your MiG server, you can open your personal Resources page on the corresponding web interface and add resources of all kinds there. Please refer to the wiki pages online for explanations on each kind of resource and some examples of setups.
== Stopping a MiG server == All the daemons can be stopped with ctrl-c and most also support a SHUTDOWN message through the named input pipe defined in the configuration file: echo SHUTDOWN >> /path/to/server.stdin
To completely stop MiG you need to stop all the MiG daemons and the apache server.
= Uninstalling MiG = The default server configuration template keeps all MiG files installed under the single directory where the MiG source code is unpacked, so uninstalling is simply a matter of deleting that directory. If you change your server configuration to save e.g. state files outside this directory you will have to manually clean up those directories as well to completely uninstall MiG.