Introduction

The GORpipe analysis tool is developed and released by Genuity Science. It originates from the pioneers of population based genomic analysis, deCODE genetics, headquartered in Reykjavik, Iceland.

GORpipe is a tool based on a genomic ordered relational architecture and allows analysis of large sets of genomic and phenotypic tabular data using a declarative query language, in a parallel execution engine. It is very efficient in a wide range of use-cases, including genome wide batch analysis, range-queries, genomic table joins of variants and segments, filtering, aggregation etc. The query language combines ideas from SQL and Unix shell pipe syntax, supporting seek-able nested queries, materialized views, and a rich set of commands and functions. For more information see the paper in Bioinformatics (https://dx.doi.org/10.1093%2Fbioinformatics%2Fbtw199).

Prerequisites

Before setting up GORpipe you need to have Java JDK or JRE version 11 or higher set up on your computer. For example Open JDK (https://openjdk.java.net/install/). To check your Java version, open up a terminal and enter:

java -version

Alternatively Oracle distributions can also be used (https://www.oracle.com/java/technologies/javase-downloads.html).

Getting started with GORpipe

Download the latest release of GORpipe from https://github.com/gorpipe/gor/releases.

Extract the package (gor-scripts-\<version>.zip).

Then running GOR without setting up test data can be done by generating GOR rows and then running GOR against that data. For example:

./gor-scripts-<version>/bin/gorpipe "gor <(gorrows -p chr1:1000-20000 -segment 100 -step 50 | multimap -cartesian <(norrows 100 | group -lis -sc #1))"  

Note: Substitute \<version> with the actual latest version number (e.g. gor-scripts-2.9.0).

Optional: A version of GORpipe (GORspark) including an integration with Apache Spark can be setup by downloading the latest release from https://github.com/gorpipe/gor-spark/releases. This release is much larger (~270MB) than the regular GORpipe release since it contains Apache Spark libraries.

Setting up test data (Optional)

Download the latest released GORpipe test data from https://github.com/gorpipe/gor-test-data/releases.

Extract the package (gor-test-data.zip) and assuming it is located in the same folder as the latest release of GORpipe, gor queries can been run against test data via:

 ./gor-scripts-<version>/bin/gorpipe "nor gor-test-data/gor/dbsnp_test.gor | top 10" 

The results should be as follows:

Chrom   POS     reference       allele  differentrsIDs
chr1    10179   C       CC      rs367896724
chr1    10250   A       C       rs199706086
chr10   60803   T       G       rs536478188
chr10   61023   C       G       rs370414480
chr11   61248   G       A       rs367559610
chr11   66295   C       A       rs61869613
chr12   60162   C       G       rs544101329
chr12   60545   A       T       rs570991495
chr13   19020013        C       T       rs181615907
chr13   19020145        G       T       rs28970552

If the GORspark version was setup the following query should work as well the results the same as above:

./gor-scripts-<version>/bin/gorpipe "select * from gor-test-data/gor/dbsnp_test.gor limit 10" 

Setting up an interactive shell for GORpipe

GORpipe can be invoked and used through an interactive shell session in a terminal - a sort of REPL for GOR, coined GORshell. Start a GORshell by executing:

./gor-scripts-<version>/bin/gorshell 

This will start an interactive shell session where queries can be executed:

gor gor-test-data/gor/dbsnp_test.gor | top 10

For a list of GOR input sources, pipe commands and other details, simply type help within the GOR shell.

Setting up reference data (Optional)

Go to into the gor-scripts folder (gor-scripts-\<version>/) and download the reference data found at:

https://s3.amazonaws.com/wuxinextcode-sm-public/data/standalone-project-data.tar.gz

If using a command line, this can be accomplished using wget

wget https://s3.amazonaws.com/wuxinextcode-sm-public/data/standalone-project-data.tar.gz

Since this is a large dataset (~9gb), this download could take a few minutes. After that extract the package via:

tar -xvf standalone-project-data.tar.gz

and a folder called ref should be created.

Note that the file standalone-project-data.tar.gz will remain in the folder. You may want to delete it afterwards.

To test the reference data, using aliases, try the following query while being within the gor-scripts folder:

./bin/gorpipe "gor #genes# | top 10" -aliases config/gor_aliases.txt

The results should be as follows:

Chrom   gene_start  gene_end    Gene_Symbol
chr1    11868   14412   DDX11L1
chr1    14362   29806   WASH7P
chr1    29553   31109   MIR1302-10
chr1    34553   36081   FAM138A
chr1    52472   54936   OR4G4P
chr1    62947   63887   OR4G11P
chr1    69090   70008   OR4F5
chr1    89294   133566  RP11-34P13.7
chr1    89550   91105   RP11-34P13.8
chr1    131024  134836  CICP27

Setting up Phecode gwas data (Optional)

Go to into the gor-scripts folder (gor-scripts-\<version>/) and download the Phecode gwas data found at:

https://s3.amazonaws.com/wuxinextcode-sm-public/data/standalone-project-phecode-gwas-data.tar.gz

If using a command line, this can be accomplished using wget

wget https://s3.amazonaws.com/wuxinextcode-sm-public/data/standalone-project-phecode-gwas-data.tar.gz

Since this is a large dataset (~10gb), this download could take a few minutes. After that extract the package via:

tar -xvf standalone-project-phecode-gwas-data.tar.gz

and a folder called phecode_gwas should be created.

Note that the file standalone-project-phecode-gwas-data.tar.gz will remain in the folder. You may want to delete it afterwards.

To test this data, the GOR dictionary for the data, Phecode_adjust_f2.gord can be queried via for example:

./bin/gorpipe "gor phecode_gwas/Phecode_adjust_f2.gord | top 10"

The results should be as follows:

CHROM   POS REF ALT pVal_mm OR_mm   CASE_info   GC  QQ  BONF    HOLM    Source
chr1    11008   C   G   7.0e-09 1.3071212318082575  11/520/7230 0.065424    0.24246 0.11973 0.090699    218.1
chr1    11008   C   G   6.6e-22 4.444107087017232   4/60/248    0.23978 0.35578 1.1289e-14  7.2724e-15  282.5
chr1    11012   C   G   7.0e-09 1.3071212318082575  11/520/7230 0.065424    0.24246 0.11973 0.090699    218.1
chr1    11012   C   G   6.6e-22 4.444107087017232   4/60/248    0.23978 0.35578 1.1289e-14  7.2724e-15  282.5
chr1    13116   T   G   3.7e-10 0.1992753733291237  0/9/272 0.44383 0.46853 0.0063285   0.0033634   282.5
chr1    13118   A   G   3.7e-10 0.1992753733291237  0/9/272 0.44383 0.46853 0.0063285   0.0033634   282.5
chr1    13273   G   C   4.9e-08 0.1156812360571759  0/3/261 0.50503 0.50293 0.83810 0.41659 282.5
chr1    14464   A   T   8.1e-06 4.3201606068833875  2/12/23 1.2298e-10  1.2424e-05  1.0000  1.0000  362.8
chr1    14464   A   T   7.8e-08 0.22233506547729875 0/8/278 0.51155 0.50681 1.0000  0.65797 282.5
chr1    15211   T   G   3.9e-15 0.2678814996572006  29/38/16    0.33692 0.41018 6.6706e-08  3.9344e-08  282.5

Setting environment variables

For convenience, GORpipe and GORshell can be added to path. For example on Mac by editing /etc/paths:

sudo vim /etc/paths 

and add the following line:

<PATH_TO_GOR_SCRIPTS>/bin

Then GORpipe and GORshell can be started via gorpipe and gorshell from any location.

Build GORpipe from source

For developers, to get started with GORpipe, first clone the repo via:

git clone https://github.com/gorpipe/gor

Test data for GOR is then obtained by cloning the GOR test data repository (https://github.com/gorpipe/gor-test-data) as a submodule into the tests/data folder:

git submodule update --init --recursive

The code is built via:

make build

Now gor queries can been run against test data. For example:

./gortools/build/install/gor-scripts/bin/gorpipe "gor tests/data/gor/dbsnp_test.gor | top 10"

The results should be as follows:

Chrom   POS     reference       allele  differentrsIDs
chr1    10179   C       CC      rs367896724
chr1    10250   A       C       rs199706086
chr10   60803   T       G       rs536478188
chr10   61023   C       G       rs370414480
chr11   61248   G       A       rs367559610
chr11   66295   C       A       rs61869613
chr12   60162   C       G       rs544101329
chr12   60545   A       T       rs570991495
chr13   19020013        C       T       rs181615907
chr13   19020145        G       T       rs28970552

GORshell can also be started up via:

./gortools/build/install/gor-scripts/bin/gorshell

Find older packages

Previous to the migration to gitlab package registry GOR artifacts were stored in jfrog artifactory, the contents of our libs-release-local repo can be found in s3://wxnc-build-artifacts and credentials for it are in secret server #3610.

How to get help?

Documentation for GORpipe can be found at: http://docs.gorpipe.org/. Additionally, help can be found while using GORpipe by executing gorpipe help or just help within the GOR shell.

Citations

If you make use of GORpipe in your research, we would appreciate a citation to the following paper:

GORpipe: a query tool for working with sequence data based on a Genomic Ordered Relational (GOR) architecture
Bioinformatics, Volume 32, Issue 20, 15 October 2016, Pages 3081–3088,
https://dx.doi.org/10.1093%2Fbioinformatics%2Fbtw199

License

GORpipe is free software: you can redistribute it and/or modify
it under the terms of the AFFERO GNU General Public License as published by
the Free Software Foundation.

GORpipe is distributed "AS-IS" AND WITHOUT ANY WARRANTY OF ANY KIND,
INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY,
NON-INFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. See
the AFFERO GNU General Public License for the complete license terms.

You should have received a copy of the AFFERO GNU General Public License
along with GORpipe.  If not, see <http://www.gnu.org/licenses/agpl-3.0.html>