Readme for art_modern

Modernized ART that is parallelized and modularized using modern C++.

WARNING Largely under development in internal Git hosting. The Git repository on GitHub may not reflect latest status.

Motivation

ART is an excellent software for simulating reads from a reference genome. However, it comes with following limitations:

• The implementation is not parallelized.
• The implementation may consume too much memory or fail on genomes that contains limited number of chromosomes of enormous sizes.
• The implementation may consume too much memory or fail on transcriptome or template collection that contains enormous number of contigs (cDNAs or other types of template molecules).
• The implementation only supports unified coverage, which is not suitable for transcriptome simulation where each cDNA molecule have its own expression level.
• The implementation outputs alignments in SAM format, which is not space-efficient.

So, we developed art_modern with the following ideas:

• Parallelization is implemented using another layer of abstraction, simulation job, which can be taken as a unit of work in one thread \& process \& co-routine, etc.
• The SAM output format was re-implemented using HTSLib, which allows BAM output format with minimal modifications of code.
• Multiple FASTA parsers were added. For example, the htslib parser allows on-disk random access of enormous genomes without reading them into memory, while the stream parser allows streaming of FASTA files.

Quick Start

Build the project using:

mkdir -p build_release
env -C build_release cmake -DCMAKE_BUILD_TYPE=Release -DCEU_CM_SHOULD_ENABLE_TEST=FALSE ..
env -C build_release make

The project binary will be available at build_release/art_modern.

Installation

Dependencies

• CMake, the building system used in our project. That further require:
  • A CMake Generator.
  • Under GNU/Linux, using Ninja or GNU Make is preferred.
  • Under GNU/Linux, our project further requires GNU Bash.
• A working C and C++ compiler that supports C++ 14. Following compilers are supported:
  • GCC;
  • Clang;
  • Intel oneAPI DPCPP.
• Boost C++ Library. The modules used in this project are namely:
  • REQUIRED FileSystem;
  • REQUIRED Regex;
  • REQUIRED Program Options;
  • REQUIRED Thread;
  • REQUIRED Log;
  • REQUIRED StackTrace;
  • OPTIONAL Test: For unit testing only. Can be absent for non-developers;
  • OPTIONAL Timer: For displaying CPU and wall-clock time at the end of the program.
  • OPTIONAL stacktrace_backtrace: See here for details.
• A working HTSLib.
  • To use bundled HTSLib sources, you need to have:
  • REQUIRED zlib;
  • REQUIRED pthread;
  • OPTIONAL libbz2;
  • OPTIONAL liblzma;
  • OPTIONAL libdeflate;
  • See official HTSLib documentation for more details.
  • To use external HTSLib, consult your system administrator.

CMake Variables

CMake variables should be set when invoking cmake. For example,

cmake -DBUILD_SHARED_LIBS=ON

sets BUILD_SHARED_LIBS to ON.

• BUILD_SHARED_LIBS: Whether to build shared libraries
  • ON (DEFAULT): Will search for shared libraries and use dynamic linking.
  • OFF: Will search for static libraries and use static linking.
• USE_HTSLIB: Use which HTSLib implementation
  • unset (DEFAULT): Will use bundled HTSLib.
  • hts: Will use the HTSLib found in system.
• CEU_CM_SHOULD_ENABLE_TEST: Whether test should be enabled.
  • unset (DEFAULT): Depends on CMAKE_BUILD_TYPE.
  • OFF: Will disable test.
  • ON: Will enable test.
• CEU_CM_SHOULD_USE_NATIVE: Whether to build the binaries using -mtune=native, if possible. This would result in faster executable but impaired portability.
  • OFF (DEFAULT): Will not build native executables/libraries.
  • ON: Will not native executables/libraries.
• CMAKE_BUILD_TYPE: The CMake build type.
  • Debug (DEFAULT): For developers with debugging needs.
  • Optimization is turned off with debugging symbols and compiler warnings enabled.
  • If CEU_CM_SHOULD_ENABLE_TEST is unset, it will be set to TRUE.
  • Release: Optimized executables/libraries without debug symbols.
  • Optimization is turned on with compiler warnings disabled.
  • If CEU_CM_SHOULD_ENABLE_TEST is unset, it will be set to FALSE.
  • RelWithDebInfo: Optimized executables/libraries with debug symbols.
  • Optimization is turned on with compiler warnings disabled.
  • If CEU_CM_SHOULD_ENABLE_TEST is unset, it will be set to TRUE.

Usage

Mode

The parallelization strategy of different modes and input parsers are as follows:

Input Formats

Currently, we support input in FASTA and PBSim3 transcripts format.

FOR FASTA FORMAT: For read names, only characters before blank space are read.

A compatibility matrix is as follows:

Library Construction Methods

FASTA Parsers

Changes Compared to Official ART Implementation

Changes on software function:

• [ ] Supports 3 modes: wgs, trans and templ, similar to pbsim3.
• [ ] Supports 3 FASTA parsers: memory, htslib and stream.
• [X] Supports 3 library construction methods: se, pe and mp.
• [X] Except FASTQ, support output in SAM and BAM format through HTSLib.
• [X] Support over masking detection was dropped.
• [X] Support over sequencers except Illumina dropped.
• [X] Support over the aln output format was dropped.
• [X] Built-in profiles are no longer supported. User must specify path to existing profiles.

Changes on software engineering stuff:

• Build system changed to CMake.
• All C++ code were re-implemented in C++14 with radical removal of duplicated or unused code.
• Random generator was changed from GNU Science Library (GSL) to Boost or C++ standard library.
• Logging re-implemented using Boost.
• Multithreading support implemented using Boost.
• Largely eliminated POSIX-only routines by Boost.
• Argument parser implemented in Boost.

Acknowledgements

This simulator is based on the works of Weichun Huang whduke@gmail.com et al., under GNU GPL v3 license. The software is originally distributed here with following reference:

• W. Huang, L. Li, J. R. Myers, and G. T. Marth, ART: a next-generation sequencing read simulator, Bioinformatics (Oxford, England), vol. 28, no. 4, pp. 593–594, Feb. 2012, doi: 10.1093/bioinformatics/btr708.

The bundled HTSLib library used MIT License with following reference:

• J. K. Bonfield et al., HTSlib: C library for reading/writing high-throughput sequencing data, GigaScience, vol. 10, no. 2, p. giab007, Jan. 2021, doi: 10.1093/gigascience/giab007.

TODO

• Refactor output-related argument parser to OutputDispatcher.

• Design and implement a job scheduling system using boost::lockfree and boost::signal2.

• Make it faster.

• Update the HTSLib CMake routine for setting macros like HAVE_LIBBZ2 correct.

• Support running under Microsoft Windows.

• Support Illumina Complete Long Read?

• Support MAF output format?

FAQ

How to split produced pair-end/mate-pair sequencing results to 2 files?

This can be done through seqtk. For example, to split tmp/test_small_pe/NC_001416.1.fq:

# Read 1
seqtk seq tmp/test_small_pe/NC_001416.1.fq -1 > tmp/test_small_pe/NC_001416.1_1.fq
# Read 2
seqtk seq tmp/test_small_pe/NC_001416.1.fq -2 > tmp/test_small_pe/NC_001416.1_2.fq