.. tag:intro-begin
|Build Status| |GitHub license| |Docs|
Spfy: Platform for predicting subtypes from E.coli whole genome sequences, and builds graph data for population-wide comparative analyses.
Published as: Le,K.K., Whiteside,M.D., Hopkins,J.E., Gannon,V.P.J., Laing,C.R. Spfy: an integrated graph database for real-time prediction of bacterial phenotypes and downstream comparative analyses. Database (2018) Vol. 2018: article ID bay086; doi:10.1093/database/bay086
Live: https://lfz.corefacility.ca/superphy/spfy/
.. image:: screenshots/screen-results_list.png :align: center :alt: screenshot of the results page
link <https://docs.docker.com/compose/install/>
__). mac/windows
users have Compose bundled with Docker Engine.git clone --recursive https://github.com/superphy/spfy.git
cd spfy/
docker-compose up
ECTyper:
.. image:: https://travis-ci.org/phac-nml/ecoli_serotyping.svg?branch=superphy :target: https://travis-ci.org/phac-nml/ecoli_serotyping
PanPredic:
.. image:: https://travis-ci.org/superphy/PanPredic.svg?branch=master :target: https://travis-ci.org/superphy/PanPredic
Docker Image for Conda:
.. image:: https://travis-ci.org/superphy/docker-flask-conda.svg?branch=master :target: https://travis-ci.org/superphy/docker-flask-conda
Comparing different population groups:
|fo|
.. |fg| image:: screenshots/fishers_genomes.png :width: 20% :alt: As a factor of # Genomes per Target
.. |ft| image:: screenshots/fishers_targets.png :width: 20% :alt: As a factor of # Targets Retrieved per Genome
.. |fo| image:: screenshots/fishers_overall.png :width: 20% :alt: Overall Performance
Runtimes of subtyping modules:
.. image:: screenshots/spfy_indivs.png :width: 20% :alt: Runtimes of individual analyses
cd app/
python -m modules/savvy -i tests/ecoli/GCA_001894495.1_ASM189449v1_genomic.fna
where the argument after the -i
is your genome (FASTA) file... image:: screenshots/ontology.png :align: center :alt: screenshot of the results page
The ontology for Spfy is available at:
https://raw.githubusercontent.com/superphy/backend/master/app/scripts/spfy\_ontology.ttl
It was generated using
https://raw.githubusercontent.com/superphy/backend/master/app/scripts/generate\_ontology.py
with shared functions from Spfy's backend code. If you wish to run it,
do: 1. cd app/
2. python -m scripts/generate_ontology
which will
put the ontology in app/
You can generate a pretty diagram from the .ttl file using http://www.visualdataweb.de/webvowl/
.. note:: currently setup for just .fna files
You can bypass the front-end website and still enqueue subtyping jobs by:
First, mount the host directory with all your genome files to /datastore
in the containers.
For example, if you keep your files at /home/bob/ecoli-genomes/
, you'd
edit the docker-compose.yml
file and replace:
.. code-block:: yaml
volumes:
with:
.. code-block:: yaml
volumes:
Then take down your docker composition (if it's up) and restart it
.. code-block:: shell
docker-compose down docker-compose up -d
Drop and shell into your webserver container (though the worker containers would work too) and run the script.
.. code-block:: shell
docker exec -it backend_webserver_1 sh python -m scripts/sideload exit
Note that reisdues may be created in your genome folder.
.. image:: screenshots/docker.svg :align: center :alt: screenshot of the results page
+------+------+------+------+ | Dock | Port | Name | Des | | er | s | s | crip | | Imag | | | tion | | e | | | | +======+======+======+======+ | back | 80/t | back | the | | end- | cp, | end\ | main | | rq | 443/ | _wor | redi | | | tcp | ker\ | s | | | | _1 | queu | | | | | e | | | | | work | | | | | ers | +------+------+------+------+ | back | 80/t | back | this | | end- | cp, | end\ | hand | | rq-b | 443/ | wor | les | | laze | tcp | ker- | spfy | | grap | | blaz | ID | | h | | egra | gene | | | | ph-i | rati | | | | ds\ | on | | | | 1 | for | | | | | the | | | | | blaz | | | | | egra | | | | | ph | | | | | data | | | | | base | +------+------+------+------+ | back | 0.0. | back | the | | end | 0.0: | end\ | flas | | | 8000 | _web | k | | | ->80 | -ngi | back | | | /tcp | nx-u | end | | | , | wsgi | whic | | | 443/ | _1 | h | | | tcp | | hand | | | | | les | | | | | enqu | | | | | euei | | | | | ng | | | | | task | | | | | s | +------+------+------+------+ | supe | 0.0. | back | Blaz | | rphy | 0.0: | end\ | egra | | /bla | 8080 | _bla | ph | | zegr | ->80 | zegr | Data | | aph: | 80/t | aph\ | base | | 2.1. | cp | _1 | | | 4-in | | | | | fere | | | | | ncin | | | | | g | | | | +------+------+------+------+ | redi | 6379 | back | Redi | | s:3. | /tcp | end\ | s | | 2 | | red | Data | | | | is\ | base | | | | 1 | | +------+------+------+------+ | reac | 0.0. | back | fron | | tapp | 0.0: | end\ | t-en | | | 8090 | _rea | d | | | ->50 | ctap | to | | | 00/t | p_1 | spfy | | | cp | | | +------+------+------+------+
The superphy/backend-rq:2.0.0
image is scalable: you can create as
many instances as you need/have processing power for. The image is
responsible for listening to the multiples
queue (12 workers) which
handles most of the tasks, including RGI
calls. It also listens to
the singles
queue (1 worker) which runs ECTyper
. This is done as
RGI
is the slowest part of the equation. Worker management in
handled in supervisor
.
The superphy/backend-rq-blazegraph:2.0.0
image is not scalable: it
is responsible for querying the Blazegraph database for duplicate
entries and for assigning spfyIDs in sequential order. It's functions
are kept as minimal as possible to improve performance (as ID generation
is the one bottleneck in otherwise parallel pipelines); comparisons are
done by sha1 hashes of the submitted files and non-duplicates have their
IDs reserved by linking the generated spfyID to the file hash. Worker
management in handled in supervisor
.
The superphy/backend:2.0.0
which runs the Flask endpoints uses
supervisor
to manage inner processes: nginx
, uWsgi
.
#63 <https://github.com/superphy/backend/issues/63>
__
Alternatively, modify the endpoint accordingly under
database['blazegraph_url']
in /app/config.py
Steps required to add new modules are documented in the Developer Guide
_.
.. _Developer Guide
: http://superphy.readthedocs.io/en/latest/contributing.html
.. |Build Status| image:: https://travis-ci.org/superphy/spfy.svg?branch=master :target: https://travis-ci.org/superphy/spfy .. |GitHub license| image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://raw.githubusercontent.com/superphy/spfy/master/LICENSE .. |Docs| image:: https://readthedocs.org/projects/spfy/badge/?version=latest :target: http://spfy.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status
.. tag:intro-end