search

marbl / ModDotPlot

MIT License
101 stars 7 forks source link

readme

Cite

Alexander P. Sweeten, Michael C. Schatz, Adam M. Phillippy, ModDotPlot - Rapid and interactive visualization of complex repeats bioRxiv 2024.04.15.589623; doi: https://doi.org/10.1101/2024.04.15.589623

If you use ModDotPlot for your research, please cite our software!

About

ModDotPlot is a dot plot visualization tool designed for large sequences and whole genomes. ModDotPlot outputs an identity heatmap similar to StainedGlass by rapidly approximating the Average Nucleotide Identity between pairwise combinations of genomic intervals. This significantly reduces the computational time required to produce these plots, enough to view multiple layers of resolution in real time!

Installation

git clone https://github.com/marbl/ModDotPlot.git
cd ModDotPlot

Although optional, it's recommended to setup a virtual environment before using ModDotPlot:

python -m venv venv
source venv/bin/activate

Once activated, you can install the required dependencies:

python -m pip install .

Finally, confirm that the installation was installed correctly by running moddotplot -h:

  __  __           _   _____        _     _____  _       _   
 |  \/  |         | | |  __ \      | |   |  __ \| |     | |  
 | \  / | ___   __| | | |  | | ___ | |_  | |__) | | ___ | |_ 
 | |\/| |/ _ \ / _` | | |  | |/ _ \| __| |  ___/| |/ _ \| __|
 | |  | | (_) | (_| | | |__| | (_) | |_  | |    | | (_) | |_ 
 |_|  |_|\___/ \__,_| |_____/ \___/ \__| |_|    |_|\___/ \__|

usage: moddotplot [-h] {interactive,static} ...

ModDotPlot: Visualization of Complex Repeat Structures

positional arguments:
  {interactive,static}  Choose mode: interactive or static
    interactive         Interactive mode commands
    static              Static mode commands

options:
  -h, --help            show this help message and exit`

Usage

ModDotPlot must be run either in interactive mode, or static mode:

Interactive Mode

moddotplot interactive <ARGS>

This will launch a Dash application on your machine's localhost. Open any web browser and go to http://127.0.0.1:<PORT_NUMBER> to view the interactive plot (this should happen automatically, but depending on your environment you might need to copy and paste this URL into your web browser). Running Ctrl+C on the command line will exit the Dash application. The default port number used by Dash is 8050, but this can be customized using the --port command (see interactive mode commands for further info).

Static Mode

moddotplot static <ARGS>

This skips running Dash and quickly creates plots under the specified output directory using plotnine. By default, running ModDotPlot in static mode this will produce the following files:

See static mode commands for further info.

Standard arguments

The following arguments are the same in both interactive and static mode:

-f / --fasta <file>

Fasta files to input. Multifasta files are accepted. Interactive mode will only support a maximum of two sequences at a time.

-k / --kmer <int>

K-mer size to use. This should be large enough to distinguish unique k-mers with enough specificity, but not too large that sensitivity is removed. Default: 21.

-o / --output-dir <string>

Name of output directory for bed file & plots. Default is current working directory.

-id / --identity <int>

Minimum sequence identity cutoff threshold. Default is 86. While it is possible to go as low as 50% sequence identity, anything below 80% is not recommended.

--delta <float>

Each partition takes into account a fraction of its neighboring partitions k-mers. This is to avoid sub-optimal identity scores when partitons don't overlap identically. Default is 0.5, and the accepted range is between 0 and 1. Anything greater than 0.5 is not recommended.

-m / --modimizer <int>

Modimizer sketch size. Must be lower than window size w. A lower sketch size means less k-mers to compare (and faster runtime), at the expense of lower accuracy. Recommended to be kept > 1000.

-r / --resolution <int>

Dotplot resolution. This corresponds to the number of windows each input sequence is partitioned into. Default is 1000. Overrides the --window parameter.

--compare <bool>

If set when 2 or more sequences are input into ModDotPlot, this will show an a vs. b style plot, in addition to a self-identity plot. Note that interactive mode currently only supports a maximum of two sequences. If more than two sequences are input, only the first two will be shown.

--compare-only <bool>

If set when 2 or more sequences are input into ModDotPlot, this will show an a vs. b style plot, without showing self-identity plots.

--ambiguous <bool>

By default, k-mers that are homopolymers of ambiguous IUPAC codes (eg. NNNNNNNNNNN’s) are excluded from identity estimation. This results in gaps along the central diagonal for these regions. If desired, these can be kept by setting the —-ambiguous flag in both interactive and static mode.

--no-plot <bool>

Save matrix to file, but skip rendering of plots. In interactive mode, this must be used alongside the --save flag

Interactive Mode Commands

--port <int>

Port to display ModDotPlot on. Default is 8050, this can be changed to any accepted port.

-w / --window <int>

Minimum window size. By default, interactive mode sets a minimum window size based on the sequence length n/2000 (eg. a 3Mbp sequence will have a 1500bp window). The maximum window size will always be set to n/1000 (3000bp under the same example). This means that 2 matrices will be created.

-q / --quick <bool>

This will automatically run interactive mode with a minimum window size equal to the maximum window size (n/1000). This will result in a quick launch, however the resolution of the plot will not improve upon zooming in.

-s / --save <bool>

Save the matrices produced in interactive mode. By default, a folder called interactive_matrices will be saved in --output_dir, containing each matrix in compressed NumPy format, as well as metadata for each matrix in a pickle. Modifying the files in interactive_matrices will cause errors when attempting to load them in the future.

-l / --load <directory>

Load previously saved matrices. Used instead of -f/--fasta

Static Mode Commands

-c / --config <.json file>

Run moddotplot static with a config file, rather than (sample syntax). Recommended when creating a really customized plot. Used instead of -f/--fasta.

-b / --bed <.bed file>

Create a plot from a previously computed pairwise bed file. Skips Average Nucleotide Identity computation. Used instead of -f/--fasta.

-w / --window <int>

Window size. Unlike interactive mode, only one matrix will be created, so this represents the only window size. Default is set to n/1000 (eg. 3000bp for a 3Mbp sequence).

--no-bed <bool>

Skip output of bed file.

--no-hist <bool>

Skip output of histogram legend.

--width <int>

Adjust width of self dot plots. Default is 9 inches.

--dpi <bool>

Image resolution in dots per inch (not to be confused with dotplot resolution). Default is 600.

--palette <str>

List of accepted palettes can be found here. The syntax is to have the name of the palette, followed by an underscore with the number of colors, eg. OrRd_8. Default is Spectral_11.

--palette-orientation <bool>

Flip sequential order of color palette. Set to - by default for divergent palettes.

--color <list of hexcodes>

List of custom colors in hexcode format can be entered sequentially, mapped from low to high identity.

--breakpoints <list of ints>

Add custom identity threshold breakpoints. Note that the number of breakpoints must be equal to the number of colors + 1.

-t / --axes-ticks <list of ints>

Custom tickmarks for x and y axis. Values outside of the axes-limits will not be shown.

-a / --axes-limits <int>

Change axis limits for x and y axis. Useful for comparing multiple plots, allowing them to stay in scale.

--bin-freq <bool>

By default, histograms are evenly spaced based on the number of colors and the identity threshold. Select this argument to bin based on the frequency of observed identity values.

Sample run - Interactive Mode

$ moddotplot interactive -f sequences/Chr1_cen.fa     

  __  __           _   _____        _     _____  _       _   
 |  \/  |         | | |  __ \      | |   |  __ \| |     | |  
 | \  / | ___   __| | | |  | | ___ | |_  | |__) | | ___ | |_ 
 | |\/| |/ _ \ / _` | | |  | |/ _ \| __| |  ___/| |/ _ \| __|
 | |  | | (_) | (_| | | |__| | (_) | |_  | |    | | (_) | |_ 
 |_|  |_|\___/ \__,_| |_____/ \___/ \__| |_|    |_|\___/ \__|

Running ModDotPlot in interactive mode

Retrieving k-mers from Chr1:14000000-18000000.... 

Progress: |████████████████████████████████████████| 100.0% Completed

Chr1:14000000-18000000 k-mers retrieved! 

Building self-identity matrices for Chr1:14000000-18000000, using a minimum window size of 2000.... 

Layer 1 using window length 2000

Progress: |████████████████████████████████████████| 100.0% Completed

Layer 2 using window length 4000

Progress: |████████████████████████████████████████| 100.0% Completed

ModDotPlot interactive mode is successfully running on http://127.0.0.1:8050/ 

Dash is running on http://127.0.0.1:8050/

The plotly plot can be navigated using the zoom (magnifying glass) and pan (hand) icons. The plot can be reset by double-clicking or selecting the home button. The identity threshold can be modified by seelcting the slider. Colors can be readjusted according to the same gradient based on the new identity levels.

Sample run - Port Forwarding

Running interactive mode on an HPC environment can be accomplished through the use of port forwarding. On your remote server, run ModDotPlot as normal:

moddotplot interactive -f INPUT_FASTA_FILE(S) --port HPC_PORT_NUMBER

Then on your local machine, set up a port forwarding tunnel:

ssh -N -f -L <LOCAL_PORT_NUMBER>:127.0.0.1:<HPC_PORT_NUMBER> HPC@LOGIN.CREDENTIALS

You should now be able to view interactive mode using http://127.0.0.1:<LOCAL_PORT_NUMBER>. Note that your own HPC environment may have specific instructions and/or restrictions for setting up port forwarding.

VSCode now has automatic port forwarding built into the terminal menu. See VSCode documentation for further details

Sample run - Static Plots

When running ModDotPlot to produce static plots, it is recommended to use a config file, especially when creating extremely customized plots. The config file is provided in JSON, and accepts the same syntax as the command line arguments shown above. 

$ cat config/config.json

{
    "identity": 90,
    "sparsity": 10,
    "palette": "OrRd_7",
    "breakpoints": [
        90,
        91,
        92,
        93,
        96,
        98,
        99,
        100
    ],
    "output_dir": "Chr1_cen_plots",
    "fasta": [
        "../sequences/Chr1_cen.fa"
    ]
}
$ moddotplot static -c config/config.json               
  __  __           _   _____        _     _____  _       _   
 |  \/  |         | | |  __ \      | |   |  __ \| |     | |  
 | \  / | ___   __| | | |  | | ___ | |_  | |__) | | ___ | |_ 
 | |\/| |/ _ \ / _` | | |  | |/ _ \| __| |  ___/| |/ _ \| __|
 | |  | | (_) | (_| | | |__| | (_) | |_  | |    | | (_) | |_ 
 |_|  |_|\___/ \__,_| |_____/ \___/ \__| |_|    |_|\___/ \__|

Running ModDotPlot in static mode

Retrieving k-mers from Chr1:14M-18M.... 

Progress: |████████████████████████████████████████| 100.0% Completed

Chr1:14M-18M k-mers retrieved! 

Computing self identity matrix for chr1:14M-18M... 

        Sequence length n: 4000000

        Window size w: 4000

        Modimizer sketch value: 1000

        Plot Resolution r: 1000

Progress: |████████████████████████████████████████| 100.0% Completed

Saved bed file to Chr1_cen_plots/Chr1:14M-18M.bed

Plots created! Saving to Chr1_cen_plots/Chr1:14M-18M...

Chr1_cen_plots/Chr1:14M-18M_TRI.png, Chr1_cen_plots/Chr1:14M-18M_TRI.pdf, Chr1_cen_plots/Chr1:14M-18M_FULL.png, Chr1_cen_plots/Chr1:14M-18M_FULL.png, Chr1_cen_plots/Chr1:14M-18M_HIST.png and Chr1_cen_plots/Chr1:14M-18M_HIST.pdf, saved sucessfully.

Sample run - comparing two sequences

ModDotPlot can produce an a vs. b style dotplot for each pairwise combination of input sequences. Use the --compare command line argument to include these plots. When running --compare in interactive mode, a dropdown menu will appear, allowing the user to switch between self-identity and pairwise plots. Note that a maximum of two sequences are allowed in interactive mode. If you want to skip the creation of self-identity plots, you can use --compare-only:

moddotplot interactive -f sequences/chr15_segment.fa sequences/chr21_segment.fa --compare-only

Questions

For bug reports or general usage questions, please raise a GitHub issue, or email alex ~dot~ sweeten ~at~ nih ~dot~ gov

Known Issues