How to create an APT tarball for sGMS.

[gatoravi]

This issue documents the creation of tarballs which are used during the sGMS install process. This will be useful whenever a debian package needs to be added/removed/modified for the sGMS.

[gatoravi]

The apt-repos for TGI are on http://repo.gsc.wustl.edu, to make changes to the packages that are stored here, TGI uses a git repo called genome-snapshot-ceps.

To get a version of genome-snapshot-deps, from within the TGI do git clone ssh://git/srv/git/genome-snapshot-deps.git

Once you have cloned this repo, to change the packages that are installed for precise go into the 'precise' folder. This has a bunch of files which show the packages that are installed with precise. These are of the form genome-snapshot-deps-*.depends. So, for example, to add a perl package, modify the genome-snapshot-deps-perl.depends file.

After modifying the .depends file, commit the changes using git commit

Next, the change log has to be manually updated for precise(this has to be done manually for changes to the precise packages, I think for lucid this is done automatically). The change log is of this format,

  genome-snapshot-deps (2014.03.10-1~Ubuntu~precise) precise; urgency=low

       * [2e1d398] add circos

  -- Avinash Ramu <aramu@genome.wustl.edu>  Thu, 10 Mar 2014 12:36:30 -0500

The first and last lines need to have the date the change is made in them. If there is more than one change on a given day use something like 2014.03.10-1, 2014.03.10-2, 2014.03.10-3 to indicate them in the first line, this has to be a unique entry for every change made otherwise there is no way to distinguish this from another entry with the same entry. '[2e1d398]' - this is the commit ID of the commit where circos was added to the '.depends' file.

After updating the changelog, commit the changes using git commit

The next step is to upload these changes to the repo.gsc.wustl.edu site. This is mostly automated and is done by running make debian DISTRO=precise this creates a bunch of files that for example look like

genome-snapshot-deps-apps-external_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-apps-tgi_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-perl_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-python_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-refcov_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-trans-abyss_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-ur_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps-workflow_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps_2014.03.10-1~Ubuntu~precise.dsc
genome-snapshot-deps_2014.03.10-1~Ubuntu~precise.tar.gz
genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_all.deb
genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes
genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.precise-genome-development.upload

The genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes (replace with your version here) file should be signed automatically by the last step in the make debian DISTRO=precise process with your GPG key unless there is some problem.

Note: if you do a make debian DISTRO=precise the GPG key of the person who made the last entry in the precise/changelog file is required to sign the 'changes' file. So you can only sign the changes file if you made the last entry in the changelog(I need to double check this with someone in APIPE, but it sounds logical.)

So make sure your GPG keys are setup correctly before attempting this process, there are a bunch of tutorials within the TGI and on the inter webs which tell you how to accomplish this.

After the make step,

try debsign genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes(replace with your changes file), if things went well to this point, this file should already be signed with your ID and this step is not necessary.

The next step is to upload these changes to repo.gsc.wustl.edu which is done by dput precise-genome-development ../genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes(replace with your changes file)`

[gatoravi]

Now that we have uploaded these changes to repo.gsc.wustl.edu, we need to make a tarball with the updated packages. Note: the sGMS installs the debian packages from the tarball(which it downloads from an FTP site https://xfer.genome.wustl.edu/gxfer1/project/gms/) instead of downloading the packages directly from 'repo.gsc.wustl.edu'. The reasons for this are unknown but might be to mitigate load on that server.

This process is accomplished with the help of another Makefile inside the 'gms' repo. To get this repo do, git clone https://github.com/genome/gms.git

Run cd gms/setup/stage; make

This will create a mirror of the repo.gsc.wustl.edu and make a tar ball with all the packages in it. The tar ball has a timestamp on it which is the current date unless specified manually. This tar ball is stored in the gms/setup/archive files. The make step also automatically copies the tar ball over to the FTP staging site which is, /gscmnt/sata102/info/ftp-staging/pub/software/gms/testdata/GMS1/setup/archive-files, currently the end of the Make process complains about setting timestamps on the files in the staging directory, this might be a permissions related issue.

After the make step, double check to make sure that the updated tar ball and a 'mirror*.config' file have been copied over to the staging directory.

Now this tar-ball can be used in the sGMS install process by updating the 'APT_DUMP_VERSION=', line in the gms Makefile to the latest version of the APT DUMP. So for example APT_DUMP_VERSION=2013.03.10 will attempt to install using apt-mirror-min-ubuntu-12.04-2014.03.10.tgz.

[malachig]

@gatoravi, when you get a chance can you reformat this documentation into a Wiki page here: https://github.com/genome/gms/wiki/Documentation

And then close this issue.

[gatoravi]

this is here, https://github.com/genome/gms/wiki/How-to-create-APT-tar-balls-for-the-GMS-from-inside-The-Genome-Institute.