The FASTX-Toolkit is a collection of command line tools for Short-Reads FASTA/FASTQ files preprocessing.
Next-Generation sequencing machines usually produce FASTA or FASTQ files, containing multiple short-reads sequences (possibly with quality information).
The main processing of such FASTA/FASTQ files is mapping (aka aligning) the sequences to reference genomes or other databases using specialized programs.
Example of such mapping programs are: Blat (http://www.kentinformatics.com/index.asp), SHRiMP (http://compbio.cs.toronto.edu/shrimp), LastZ (http://www.bx.psu.edu/miller_lab), MAQ (http://maq.sourceforge.net/) And many many others.
However, It is sometimes more productive to preprocess the FASTA/FASTQ files before mapping the sequences to the genome - manipulating the sequences to produce better mapping results.
The FASTX-Toolkit tools perform some of these preprocessing tasks.
FASTQ-to-FASTA - Converts a FASTQ file to a FASTA file..
FASTQ-Statistics - scans a FASTQ file, and produces some statistics about the quality and the sequences in the file.
FASTQ-Quality-BoxPlot, and FASTQ-Nucleotides-Distribution - Generates charts based on the statistics generated by FASTQ-Statistics. These charts can be used to quickly see the quality of the sequenced library.
FASTQ-Quality-Converter - Converts from ASCII to numeric quality scores.
FASTQ-Quality-Filter - removes low-quality sequences from FASTQ files.
FASTX-Artifacts-Filter - removes some sequencing artifacts from FASTA/Q files.
FASTX-Barcode-Splitter - A common practice is to sequence multiple biological samples in the same library (marking each sample using a dedicated barcode). The resulting FASTA/Q file contains intermixed sequences from those samples. This tool separates FASTA/Q files into several individual files, based on the barcodes.
FASTX-Clipper - Adapters (aka Linkers) are added to the library (before sequencing), and should be removed from the resulting FASTA/Q file. This tool removes (clips) adapters.
FASTA-Clipping-Histogram - After clipping a FASTA file, this tool generates a chart showing the length of the clipped sequences.
FASTX-Reverse-Complement - Produces a reverse-complement of FASTA/Q file. If a FASTQ file is given, the quality scores are also reversed.
FASTX-Trimmer - Extract sub-seqeunces from FASTA/Q file. Two examples are: Removing barcodes from the 5'-end of all sequences in a FASTQ file; Cutting 7 nucleotides from the 3'-end of all sequences in a FASTA file.
Galaxy (https://usegalaxy.org) is web-based framework for computational biology.
While the programs in the FASTX-Toolkit are command-line based, the package include the necessary files to integrate the tools into a Galaxy server, Allowing users to execute this tools from their web-browser.
If you run your own local mirror of a Galaxy server, you can integrate the FASTX-Toolkit into your Galaxy server.
GCC is required to compile most tools.
FASTA-Clipping-Histogram tool requires Perl, the "PerlIO::gzip", "GD::Graph::bars" modules.
Installing the perl modules can be accomplised by running:
$ sudo cpan 'PerlIO::gzip' $ sudo cpan 'GD::Graph::bars'
FASTX-Barcode-Splitter requires the GNU Sed program.
FASTQ-Quality-Boxplot and FASTQ-Nucleotides-Distribution requires the 'gnuplot' program.
When downloading the git repository from github, use the following:
$ git clone https://github.com/agordon/fastx_toolkit $ cd fastx_toolkit $ ./reconf $ ./configure $ make
When downloading a released version archive:
$ wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2 $ tar -xjvf fastx_toolkit-0.0.14.tar.bz2 $ cd fastx_toolkit-0.0.14 $ ./configure $ make
The available releases are here: https://github.com/agordon/fastx_toolkit/releases
To install the tools, run (as root):
$ sudo make install
This will install the tools into /usr/local/bin. To install the tools to a different location, change the 'configure' step to:
$ ./configure --prefix=/DESTINATION/DIRECTORY
The libgtextutils package is required to build fastx-toolkit, see https://github.com/agordon/libgtextutils/ .
Most tools support "-h" argument to show a short help screen.
Better documentation is not available at this moment.
Some more details and examples are available in the
Galaxy Installation should be done manually, and requires technical understading of the Galaxy framework.
build and install the command line tools (as described above).
Make backup of your galaxy installation (better safe than sorry).
Run the 'install_galaxy_files.sh' script, and specify the galaxy root directory. This script copies the files from the 'galaxy' sub-directory into your galaxy mirror directory.
Manually add the content of ./galaxy/fastx_toolkit_conf.xml file, into your Galaxy's tool_conf.xml
Edit [YOUR-GALAXY]/tool-data/fastx_clipper_sequences.txt file, And add your custom adapters/linkers.
Modify the "fastx_barcode_splitter_galaxy_wrapper.sh" as explained Below (see section "Special configuration for Barcode-Splitter").
Restart Galaxy.
Always make backup of your galaxy server files before trying to install the FASTX-Toolkit.
The following tools support Galaxy's functional testing: (Run from Galaxy's main directory) $ sh run_functional_tests.sh -id cshl_fastq_qual_conv $ sh run_functional_tests.sh -id cshl_fastq_to_fasta $ sh run_functional_tests.sh -id cshl_fastq_qual_stat $ sh run_functional_tests.sh -id cshl_fastx_trimmer $ sh run_functional_tests.sh -id cshl_fastx_reverse_complement $ sh run_functional_tests.sh -id cshl_fastx_artifacts_filter $ sh run_functional_tests.sh -id cshl_fasta_collapser $ sh run_functional_tests.sh -id cshl_fastx_clipper
When running the barcode-splitter tool from the command line you specify a prefix direcotry - the output files will be written to that directory (similar to GNU's split program usage).
Running the barcode-splittter inside galaxy requires a special hack beacuse (I don't know how to|Galaxy can't) create a variable number of output datasets. The number of required output files is determined by the tool only AFTER reading the barcodes description file.
The Galaxy-version of Barcode-Splitter works like this:
A FASTA/FASTQ file, and a Barcode description file are fed to the tool.
The tool produces a single output dataset (inside galaxy). This output is an HTML file, containing links to the split FASTA files.
Users can use the links to get the split FASTA files. (Since Galaxy's 'upload data' tool accepts URLs, this is not a real problem).
As the galaxy administrator, you'll have to edit 'fastx_barcode_splitter_galaxy_wrapper.sh' script and change BASEPATH and PUBLICURL to point to a publicly accesibly path on your server.
Example:
fastx_barcode_splitter_galaxy_wrapper.sh contains:
BASEPATH="/media/sdb1/galaxy/barcode_splits/" PUBLICURL="http://tango.cshl.edu/barcode_splits/"
When a user runs the barcode splitter tool, the FASTA files will be generated in
"/media/sdb1/galaxy/barcode_splits/".
The URL "http://tango.cshl.edu/barcode_splits" is set (in an apache server) to
serve files from "/media/sdb1/galaxy/barcode_splits/", with the following
configuration:
Alias /barcode_splits "/media/sdb1/galaxy/barcode_splits/"
<Directory "/media/sdb1/galaxy/barcode_splits/">
AllowOverride None
Order allow,deny
Allow from all
</Directory>FASTX-Toolkit is distributed under the Affero GPL version 3 or later (AGPLv3),
EXCEPT
All files under the 'galaxy' sub-directory are distributed under the same license as Galaxy itself (which is an MIT-style license).
While IANAL, these licenses basically mean that:
You're free to use FASTX-toolkit,
You're free to integrate FASTX-toolkit in your Galaxy mirror server (or any other server).
You're free to modify the files under 'galaxy', without making your modifications public.
If you modify the FASTX-toolkit tools, and make those modifications publicly available (either as downloadable tools, part of another product), or as a web-based server - you must make the modified source code freely available (free as in speech).
See the COPYING file for the full Affero GPL. See the GALAXY-LICENSE file for galaxy's license.
Please remember: THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
============= Please send all comments, suggestions, bug reports (or better yet - bug fixes) to assafgordon@gmail.com .