kubitron / redmine_git_hosting

A ChiliProject/Redmine plugin which makes configuring your own git hosting easy.
78 stars 15 forks source link

Redmine Git Hosting Plugin (v0.5.1x)

A ChiliProject / Redmine plugin which makes configuring your own git hosting easy. This plugin allows straightforward management of gitolite and associated public keys, the git daemon, and integrates code from Scott Schacon's "grack" utility to provide Git Smart HTTP access. Git repositories are automatically created when the repository is created in Redmine. There is also an option to automatically create a git repository for a project, when the project is created. Caching functionality is also implemented to speed page-load times for viewing git repositories.

Release Notes are available off the wiki: https://github.com/kubitron/redmine_git_hosting/wiki/Release-Notes.

Compatibility

The current version of this plugin is not compatible with all versions of Redmine and ChiliProject. As of this writing, it should be compatible with the following versions:

This plugin is compatible with most versions of gitolite with versions < 3.x. This plugin is currently not fully functional with 3.x versions of gitolite, so it is best to avoid them for now. Gitolite versions >= 2.0.3 provide slightly better resilience to recover from configuration problems that prevent administrative access (the plugin will exploit gl-admin-push to fix the access key).

Configuration Strategy

One major problem when configuring ChiliProject/Redmine + Git is how the git repositories are accessed, and in particular setting permissions properly. This plugin solves this problem by allowing the web server/rails user to run git via sudo. A previous version of this plugin (see ssh branch) requires the configuration of git to run through ssh, but using sudo to allow the web server user to run commands as the git user is much faster. To configure your sudoers file to allow your web server user to run commands as git, and your git user to run commands as the web server user add these two lines to your sudoers file (run visudo to edit this file):

www-data        ALL=(git)       NOPASSWD:ALL
git             ALL=(www-data)  NOPASSWD:ALL

This assumes www-data is your web server/rails user, and git is your git user (the user gitolite is installed under). This will allow www-data to execute commands as git without prompting for a password and git to execute commands as www-data without prompting for a password. No other access (e.g. no root access, permissions to run commands as other users) are granted by these lines. These lines are only there to facilitate communication between the web server user and the git user, no other users, keeping the system secure.

Also, the requiretty sudo setting can prevent the plugin from working correctly. Several users have reported this problem on CentOS. Check the Defaults directive in the sudoers file to see if this setting has been set. You address the problem by either removing requiretty from the Defaults directive, or by adding the following lines below the original Defaults directive to remove this requirement for only the two necessary users:

Defaults:git      !requiretty
Defaults:www-data !requiretty

Again, this assumes www-data is your web server/rails user, and git is the user gitolite is installed under.

Note that this guide refers to the "web server user" as the user under which Rails is being, which is usually (but not always) the same as the user that runs the main web server. If you are running Rails under a different user, follow these instructions using that user, not the one for the main web server.

As of the most recent set of patches, this plugin is compatible with running multiple Redmine installations on the same server, each with the same or different gitolite users/repositories. The later configuration (multiple Redmine installations, each with a different gitolite installation) is particularly useful for a web-hosting scenario with independent developers.

Step-By-Step configuration instructions

(1) Install gitolite. If possible, use gitolite version >= 2.0.3, so that the plugin will be able to recover from a variety of failures that mess up administrative access to the gitolite repository (the plugin will utilize gl-admin-push to fixup admin access in these circumstances). Note that the plugin will work with earlier versions of gitolite; however, it will not be able to recover from as many faults (it will attempt to use gl-setup which doesn't work quite as well).

The fine details of gitolite configuration are beyond the scope of these instructions, and there are plenty of guides elsewhere on configuration options. The following provides a simple (working) configuration. We will assume that gitolite has been installed for user "git" (adjust accordingly if gitolite has been installed with some other username). Generate an administrator key and initialize the gitolite repository in the following fashion as root:

su git
ssh-keygen -N "" -f ~/.ssh/id_rsa
gl-setup ~/.ssh/id_rsa.pub

Simply take any defaults offered by gl-setup (i.e. quit out of offered edit of the configuration file without making any changes). These commands will generate a new key in the .ssh directory of the "git" user, and use this key as the gitolite administrative key during the initialization of the gitolite repository.

You may choose to generate your administrative key in another way. However you generate your key, it is important that this key not have a passphrase set. Further, it is very important that this ssh key be unique and not used for anything other than administrative access, i.e. should not be used by any user as their access key.

(2) If you want to enable anonymous access to your repositories via the git:// protocol you will need to install the git-daemon. The details of this are beyond the scope of the tutorial as well (and again there are lots of available guides out there). This step is optional -- if you don't want to enable anonymous access you can skip it.

(3) Run visudo (you will need root permissions to run this) and set the necessary lines in your sudoers file, listed above. Assuming your web server is run as www-data and gitolite is installed as git, you need to add this to your sudoers file:

www-data        ALL=(git)       NOPASSWD:ALL
git             ALL=(www-data)  NOPASSWD:ALL

If you have the requiretty set in the Defaults directive of your sudoers file (it is there by default in CentOS) either remove it or add the following lines below the original directive:

Defaults:git      !requiretty
Defaults:www-data !requiretty

Note: with at least some versions of Ubuntu, you must place any additions to the sudoers file at the end, otherwise the line starting with "admin ..." ends up negating these additions -- probably to your great frustration.

(4) In the root of your Redmine/ChiliProject rails directory -- the Redmine/ChiliProject root, not the plugin root, and not the public directory -- create a directory called .ssh, and copy your gitolite administrative public/private key pair into this directory using "gitolite_admin_id_rsa" as the base name. Then change the owner of these files to www-data and the permissions to 600 for the private key and 644 for the public key. Assuming that you followed the key-generation step from (1) above, these instructions translate to performing the following as root:

cd REDMINE_ROOT
mkdir .ssh
cp ~git/.ssh/id_rsa     .ssh/gitolite_admin_id_rsa
cp ~git/.ssh/id_rsa.pub .ssh/gitolite_admin_id_rsa.pub

chown www-data -R .ssh
chmod 700 .ssh
chmod 600 .ssh/gitolite_admin_id_rsa
chmod 644 .ssh/gitolite_admin_id_rsa.pub

(5) Clone the redmine_git_hosting plugin into the vendor/plugins subdirectory of your Redmine/ChiliProject root directory:

cd REDMINE_ROOT/vendor/plugins
git clone https://github.com/kubitron/redmine_git_hosting.git
rm -rf redmine_git_hosting/.git
cd ../..

(6) It is best to set several plugin variables BEFORE you run the db:migrate_plugins task in step (7). In particular it is important that the gitScriptDir, gitUser, httpServer, gitServer, gitoliteIdentityFile and gitoliteIdentityPublicKeyFile variables are set correctly. Others that could be set include gitConfigFile, gitConfigHasAdminKey, gitRepositoryBasePath, gitRedmineSubdir, gitRepositoryHierarchy, and gitRepositoryIdentUnique (for Redmine 1.4+); however, the default values for these variables should be sufficient for most installations.

To perform initial adjustment of settings, open an editor and edit 'REDMINE_ROOT/vendor/plugins/redmine_git_hosting/init.rb' file. Starting on line 17, you will see the settings definitions you should edit.

If you followed the above directions you will not need to modify the gitoliteIdentityFile or gitoliteIdentityPublicKeyFile variables -- these specify the path to the private/public key files for accessing the gitolite admin repository. Further, although you can change the following three variables, their default values provide for a very reasonable installation:

All of these variables can be modified at a later time in the Administration => Plugins => Redmine Git Hosting Plugin configuration page. However to ensure that the database migration from your existing repositories goes smoothly it is best to modify these variables now.

As an example of the significance of the previous three variables, suppose that project-3 is a child of project-2 which is a child of project-1. Assume gitRepositoryBasePath == "repository/" and gitRedmineSubdir == "projects". When gitRepositoryHierarchy is 'true', project-3.git will be stored in repository/projects/project-1/project-2/project-3.git, which will further be reflected in the ssh access URL of repository/projects/project-1/project-2/project-3.git. In contrast, when gitRepositoryHierarchy is 'false', project-3.git will be stored directly under repository/projects -- regardless of the number and identity of any parents that it might have. In Redmine 1.4+, if gitRepositoryIdentUnique is set to 'false', then secondary repositories for project-3 will additionally have paths of the form repository/projects/project-1/project-2/project-3/repo-1.git (when gitRepositoryHierarchy is 'true'). Note that the top of the settings page (Administration => Plugins => Redmine Git Hosting Plugin configuration page) provides information about how your chosen configuration affects the storage locations and URLs for accessing projects.

(7) Run the rake db:migrate_plugins task to update the database. You will need to do this once for every rails environment you have configured (e.g. production, development, testing). For the production environment, run the following:

rake db:migrate_plugins RAILS_ENV=production

Note that you should perform a migrate_plugins operation whenever you update the plugin.

(8) Make a decision about whether or not you want the plugin to be able to modify scripts in the scripts directory on the fly. If you elect to prevent the plugin from modifying the scripts directory, you will be unable to modify certain settings from the plugin settings page (as discussed in the "Executable Scripts" section, below). You have three options:

rake redmine_git_hosting:install_scripts RAILS_ENV=production WEB_USER=www-data
rake redmine_git_hosting:install_scripts RAILS_ENV=production WEB_USER=www-data READ_ONLY=true 
rake selinux:install RAILS_ENV=production WEB_USER=www-data

Note that your selected rake task must assign the web user (e.g. "www-data") as an owner of the new scripts directory. If you omit the WEB_USER argument, the script attempts to discover the web user by running "ps aux" and searching for "httpd". On systems using apache without the passenger plugin, this discovery process should work properly. Not all systems will "do the right thing", however.

Note that you should reinstall scripts whenever you update the plugin.

(9) Unless you want to access your repositories exclusively via Smart HTTP users will need to set a public key to connect via SSH. To do this, open a browser, login to ChiliProject/Redmine and follow the "My Account" Link in the upper right-hand corner of the page. The right-hand column contains controls for adding your public key(s).

Keys should be unique, that is, the keys you set in ChiliProject / Redmine should not already exist in the gitolite repo. In particular, do not re-use the key you set as the gitolite admin key.

(10) The plugin is now configured, but you may now want to set some additional settings on the Administration => Plugins => Redmine Git Hosting Plugin page.

See below in the "Caching" section of this README for more information on caching and how the caching variables should be configured.

Resynchronization of gitolite configuration

Whenever a Redmine fetch_changesets() operation is executed (i.e. http://REDMINE_ROOT/sys/fetch_changesets?key=xxx), this plugin will check the gitolite keydir and configuration file for consistency. It will correct any errors that it finds. Further, regular execution of a fetch_changesets operation will make sure that repositories placed in the recycle_bin (during delete operations) will be expired and removed. Since there still seem to be some phantom synchronization problems, it is recommended that you execute fetch_changesets() regularly (every 15 to 30 minutes).

Two rake tasks can additionally be used for resynchronization (although these are redundant with executing fetch_changesets() through other means).

(1) To fixup the gitolite configuration file, fix errors, and delete expired files in the recycle_bin, execute:

rake redmine_git_hosting:update_repositories RAILS_ENV=production

(2) To perform all the above operations while at the same time fetching changesets for all repositories, execute:

rake redmine_git_hosting:fetch_changesets RAILS_ENV=production

Note that it is very important that these commands be run as www-user (or whatever your web server user happens to be), lest you get permission problems later. (The same is true of any fetch_changesets() operation initiated without using the web server, i.e. through the command line or from the cron daemon).

Interaction with non-Redmine gitolite users

This plugin respects gitolite repositories that are managed outside of Redmine or managed by both Redmine and non-Redmine users:

A Note About PATH variables

One major source of issues with this plugin is that Rails needs to be able to run both sudo and git. Specifically, these programs need to be in one of the directories specified by the PATH variable, in your Rails environment. This requirement has been known to cause problems, particularly when installing on FreeBSD.

To address this problem in the Apache + Passenger configuration, one possible solution is to do the following:

(1) Create a new file: /usr/local/bin/ruby18env, with the following code, modifying the PATH shown below to include all relevant directories:

#!/bin/sh
export PATH="/usr/local/lib/ruby/gems/1.8/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
[path_to_your_ruby_executable, e.g. /usr/local/bin/ruby18] $*

(2) Make this file executable:

chmod 755 /usr/local/bin/ruby18env

(3) In your httpd.conf file, replace (or add) your PassengerRuby directive with:

PassengerRuby /usr/local/bin/ruby18env

Note that this may be an issue for configurations other than Apache + Passenger, but as this is one of the most common configurations, instructions for that are provided above.

Thanks to user Tronix117 for helping to track down this issue and provide a solution for Apache + Passenger.

Placement and Modification of Executable Scripts

As of the most recent version of the plugin, you may place the executable scripts anywhere in the filesystem. The default location of the scripts is set with gitScriptDir == ' ' => 'REDMINE_ROOT/vendor/plugins/redmine_git_hosting/bin'. This is a good location, especially if you have multiple simultaneous redmine installations on the same host, since scripts are customized to each installation. Thus, we recommend that you consider keeping the default placement. In this location, some maintainers may not wish to allow the plugin to re-write the scripts during execution -- hence the options to make scripts read-only, as described in step (8) of the installation instructions, above. Further, when selinux is installed, the scripts are not writeable by default (because of selinux tags), since changing them could be construed to be a security hole.

When the script directory is not write-able by the web user, you also cannot alter four of the settings on the settings page (since their alteration would require the regeneration of scripts). These values are: gitScriptDir, gitUser, gitoliteIdentifyFile, and gitoliteIdentityPublicKeyFile. The settings page will make the fact that you cannot alter these values clear by marking that as '[Cannot Change in Current Configuration]'.

The simplest way to change these values after the fact is to (1) remove the old scripts, (2) alter the parameters on the settings page (after refreshing the settings page), then (3) reinstalling scripts as discussed in step (8) of the installation instructions. Scripts can be removed with:

rake redmine_git_hosting:remove_scripts RAILS_ENV=production

An alternate option is to alter these four settings in the init.rb file, then execute:

rake redmine_git_hosting:restore_defaults RAILS_ENV=production

This command will install the new values in the database. You should follow up by reinstalling the scripts (since their values depend on the altered parameter(s)). If you are moving the location of the gitScriptDir, you might want to remove the old scripts before executing 'restore_defaults' as described above.

Deployment Credentials

As of version 0.4.6x, this plugin provides deployment credentials on a per-repository basis. These credentials are combinations of a public key and access permissions (R or RW+) which are attached directly to a repository rather than by attaching users to repositories. Deployment credentials may be added to a repository through the repository settings interface. They may be added by anyone who is a manager for the project or by the administrator.

Public keys used in this way are called "deploy keys". They are owned by the user who created them and may be edited on the user's public_key page (which is under "my account" for individual users). Since keys have unique names (per creator), they may be reused in multiple deployment credentials (in multiple repositories), simply by selecting them from the pulldown menu on the "deployment credentials create" dialog box.

One typical use-case would be for all deploy keys to be owned by the administrator and attached selectively to various repositories. Note that the "deployment credentials create" dialog is actually a convenience dialog in that it allows the simultaneous creation of both a deploy key and deployment credential in a single step, even suggesting a name for the deployment credential, with the eye to deployments that have a separate deploy key for each repository. Reusing a deploy key in another credential is a simple matter of selecting the key from a drop-down menu.

Post-Receive URLs

As of version 0.4.6x, this plugin supports the inclusion of GitHub-style Post-Receive URLs. Once added, a post-receive URL will be notified when new changes are pushed to the repository. Two versions of this functionality are available: (1) either a GitHub-style POST operation will include json-encoded information about the updated branch or (2) an empty GET request will be issued to the given URL. Post-Receive URLs may be added from the repository settings page.

Automatic Mirror Updates

As of version 0.4.0, this plugin can automatically push updates to repository mirrors when new changes are pushed to the repository. Mirrors must grant access to the public key defined in the gitolite_admin_id_rsa.pub public key file, which is displayed for convenience in the repository settings tab.

Note that 0.4.6x introduced the ability to selectively push branches to the mirror (using the git-push refspec mechanism) rather than mirroring all branches and tags.
To utilize this feature, simply select a mirror update mode of "Force Update Remote" or "Fast Forward (unforced)" instead of the default "Complete Mirroring" in the mirror create/edit dialog. More options will then become available.

Notification to CIA.vc

As of version 0.4.0, this plugin can notify CIA.vc when changes are pushed to the repository. The project identifier on CIA.vc must match the project identifier specified in ChiliProject/Redmine exactly

"Smart" HTTP Functionality

Smart HTTP is an efficient way of communicating with the git server over http/https available in git client version 1.6.6 and newer. A more detailed description of what Smart HTTP is all about can be found at: http://progit.org/2010/03/04/smart-http.html

This plugin allows you to automatically enable Smart HTTP access to your repositories. It is highly recommended that you enable Smart HTTP access only via HTTPS -- without encryption this is very insecure. However, you will require a valid SSL certificate for this to work properly, otherwise you will get permission errors. If you want to enable (insecure) access via unencrypted HTTP go to the repository settings tab and select "HTTPS and HTTP" under the "Git Smart HTTP" tab.

Where a password is required, this is your Redmine user password.

Once Smart HTTP is enabled no further configuration is necessary. You will be able to clone from/to the HTTP[S] URL specified in the URL bar in the Project/Repository tab.

Also note that you will need to ensure that Basic Auth headers are being passed properly to Rails for this to work properly. In Apache with mod_fcgid this may mean you need to add "Passheader Authorization" into the virtual host configuration file.

Further, if you are proxying requests through a third-party (such as 'ngnix'), you need to make sure that you pass the protocol information onto Redmine so that it can distinguish between HTTP and HTTPS. One way to do this is to use the X-Forwarded-Proto header (which should be set to 'https' when https is in use from the client to the proxy). On ngnix, you can do this with an "include proxy.include" statement in your configuration.

Using the Git Daemon

In order to export repositories via the git daemon (i.e. with URLs of the form 'git://'), you must first install this daemon and give it access to the gitolite repositories (outside the scope of this README). Once you do so, then control of which repositories are exported depends on two things: (1) the setting of the public flag for the project and (2) the setting of the GitDaemon parameter in the project's repository settings. A repository will be exported via the git daemon only if its corresponding project is public and its GitDaemon flag is enabled. The redmine_git_hosting plugin handles such repositories by including a special "daemon" key in the gitolite.conf file. Presence of this key, in turn, causes gitolite to insert a "git-daemon-export-ok" flag at the top-level of the corresponding repository; this flag is interpreted by the git-daemon as a sign to export the repository.

Note that the act of changing a project from public to private will set the GitDaemon flag to false automatically (to prevent accidental export of the project via the git-daemon later).

Caching Options

As of version 0.3.0 and later this plugin includes code for caching output of the git command, which is called to display the details of the git repository. Redmine/ChiliProject by default calls git directly every time this information is needed. This can result in relatively long page load times.

This plugin caches the output of git commands to dramatically improve page load times, roughly a 10x speed increase. On a 1GB VM running Ubuntu 11.04 median repository page load times dropped from 2300ms to 180ms. Testing was done with ab utility in the apache2-utils package.

There are three configurable caching parameters in the plugins settings page: Max Cache Elements, Max Cache Element Size and Max Cache Time.

This caching functionality only works in Redmine 1.2.x+ and ChiliProject v2.x+ -- while this plugin is compatible with ChiliProject 1.x the caching functionality is not.

Important note: If using mysql for your database, you must make sure that the max_allowed_packet size is set (in, e.g., /etc/my.cnf) to be at least as large as the value you specify for Max Cache Element Size above. If you do not do this, you are likely to get very strange failures of the web server. Such a setting must be placed in the [mysqld] parameter section of this file, for instance:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
max_allowed_packet=32M

The above example should allow Max Cache Element Size == 32M.

Fast Deployment with YourChili Bash Library

Instead of installing/configuring by hand, one option for quickly deploying a fully-functional system for hosting git repositories on an Ubuntu VPS is the YourChili bash library. (http://github.com/ericpaulbishop/yourchili) This library allows you to quickly deploy ChiliProject, with this plugin to an uninitialized VPS node with Ubuntu 10.10 (from e.g. Linode) using nginx and Passenger. Just run the init_nginx_stack.sh script followed by the chili_test.sh script, modifying the variables in those scripts as desired. This library is still under development, so these instructions may need to be updated in the near future.

Selinux Configuration for Redmine

This plugin can be configured to run with selinux. We have included a rakefile in tasks/selinux.rake to assist with installing with selinux. You should start by editing init.rb and migrating as described above. Then, you can execute one of the selinux rake tasks (from the Redmine root). For instance, the simplest option installs a selinux configuration for both Redmine and the redmine_git_hosting plugin:

rake selinux:install RAILS_ENV=production 

This will generate the redmine_git_hosting binaries in ./bin, install a selinux policy for these binaries (called redmine_git.pp), then install a complete context for Redmine as follows:

(1) Most of Redmine will be marked with "public_content_rw_t".

(2) The dispatch files in Rails.root/public/dispatch.* will be marked with "httpd_sys_script_exec_t"

(3) The redmine_git_hosting binaries in Rails.root/vendor/plugins/redmine_git_hosting/bin will be labeled with "httpd_redmine_git_script_exec_t", which has been crafted to allow the sudo behavior required by these binaries.

Note that this rake file has additional options. For instance, you can specify multiple Redmine roots with regular expressions (not globbed expressions!) as follows (notice the use of double quotes):

rake selinux:install RAILS_ENV=production ROOT_PATTERN="/source/.*/redmine"

These additional options are documented in the selinux.rake file. Under normal operation, you will get one selinux complaint about /bin/touch in your log each time that you visit the plugin settings page.

Once this plugin is placed under selinux control, four of the redmine_git_hosting settings can not be modified from the settings page. They are: 'gitScriptDir', 'gitUser', 'gitoliteIdentityFile', and 'gitoliteIdentityPublicKeyFile'. The plugin settings page will make this clear. One way to modify these options is to remove the old scripts, refresh the setting page, change options, then reinstall scripts. Specifically, you can remove scripts with:

rake selinux:redmine_git_hosting:remove_scripts RAILS_ENV=production

Scripts and selinux policy/tags can be reinstalled with:

rake selinux:redmine_git_hosting:install RAILS_ENV=production

Alternately, you can change the frozen parameters by altering them in the init.rb file and resetting the options to the defaults via:

rake redmine_git_hosting:restore_defaults RAILS_ENV=production

One final comment: The selinux policy exists in binary form as selinux/redmine_git.pp. Should this policy need to be rebuilt, an additional rake task exists which will build the policy from selinux/redmine_git.te:

rake selinux:redmine_git_hosting:build_policy RAILS_ENV=productinon

This task can be followed by the selinux:install task.

The rakefile and selinux configuration has been primarily tested on Redhat Enterprise Linux version 6.x with apache and fcgi. Other configurations may require slight tweaking.

Tested Configurations

This plugin has been primarily tested on Ubuntu Server 10.10 and 11.04 (32 and 64 bit) with ChiliProject v1.x, ChiliProject 2.0.0 and Redmine 1.2.1 with PostgreSQL as the database (July, 2011). It has also been tested with Redmine 1.2.1, 1.2.2, 1.3.0, and 1.3.1 with mySql on Redhat Enterprise Linux 6.x with Apache and FCGI. Limited testing has occurred with Redmine 1.1.3. It is possible that some debugging will be necessary for other configurations. Selinux configurations were tested under Redhat Enterprise Linux version 6.x with Apache and FCGI.

Required gems

In order to use this plugin you must have the following gems installed:

lockfile

inifile

net-ssh

Copyright & License

This plugin is based largely on the Gitosis plugin by Jan Schulz-Hofen for http://plan.io. Several updates/fixes were provided by github users untoldwind, tingar and ericpaulbishop. These updates were merged together and expanded upon by Eric Bishop to create this more comprehensive Git Hosting plugin.

Copyright (c) 2011 John Kubiatowicz (kubitron@cs.berkeley.edu) MIT License.

Copyright (c) 2010-2011 Eric Bishop (ericpaulbishop@gmail.com) MIT License.

Copyright (c) 2009-2010 Jan Schulz-Hofen, ROCKET RENTALS GmbH (http://www.rocket-rentals.de). MIT License.