KatherineCaley
commented
8 months ago Getting data 🫴💾
In comparative genomics and phylogenetics studies, we are often required to use previously published data. However, obtaining the data needed, such as the sequence alignments used in a published paper, may not be straightforward.
In this section, you will use the RefSoil reference database (Choi et al., 2016) as an example to explore the various obstacles to reproducibility and data sharing.
Lastly, you will circumnavigate these obstacles by utilising command-line tools developed by the cogent3 dev team, including:
- A Python script to efficiently download hundreds of GenBank files from NCBI.
- EnsemblLite - a tool to retrieve valuable data, such as alignments, from Ensembl
📃 The GenBank file
Imagine you are... 💭
...part of a research team investigating soil microbial communities and their roles in nutrient cycling and plant growth. The RefSoil database, with its comprehensive collection of soil microbial genomes, becomes a crucial resource for your study.
🏁 Goal
Understand the sequence and annotation components of a GenBank file, and how these are important to serve your own research.
RefSoil consists of genome sequences and annotations for numerous soil bacteria and archaea. Annotations, or in GenBank files - "features", inform which parts of the sequences are protein coding genes, non-coding regions, and so on.
This information makes GenBank files an invaluable asset for your research, such as for identifying and retrieving the same genes across microbes that may contribute to plant growth.
Let's have a brief look at the GenBank file format.
The GenBank file format 📃
We will diverge from RefSoil and have a look at an example of the GenBank file for a tardigrade 18S rRNA sequence.
The key components of a GenBank file include:
- Metadata: The lines from LOCUS-JOURNAL contains identifying information about the sequence, such as what the sequence is, the accession ID, the study it was derived from and so forth.
- Features: The feature table lists annotated features such as the location of certain genes, and what they are.
- Origin: The actual sequence.
For more information about the specific components of a GenBank file, see this GenBank Sample Record.
🧬 Obtaining the RefSoil GenBank sequences
Shifting back to RefSoil, we will now explore the various ways of obtaining the reference sequence data. The authors have suggested several options to access the reference sequences from GenBank including:
- Downloading the raw sequence alignments from figshare
- Using their suggested python script
- With a list of accession IDs
The following sections include screen casts which show the process of retrieving the data using the suggested methods (plot twist - they don't work as expected).
🏁 Goal
Be familiar with the potential hurdles in retrieving published data
Downloading sequences via figshare 🍓
[figshare.mp4]
One approach is to access the raw sequencing data via the figshare data repository. This way requires downloading the supplementary information (.docx) from the paper, navigating to the GitHub page using the link in the document, then to figshare using the link on the GitHub page... a bit convoluted.
Then, the compressed sequence file was downloaded and attempted to be extracted. Although the video was cut short, you will encounter that the compression step will hang.
Downloading sequences via Python script 🐍
[script.mp4]
This approach involves cloning the repository and running the `fetch_genbank.py` script given a list of IDs in `refsoil_id.txt`. In the screen cast, you will see me (Fred) attempt to debug the (mainly python2-related) issues that arise.
Instead of continuing to debug the script, we will leverage `refsoil_id.txt` to rapidly download the files in the next section.
Downloading GenBank files with a list of accession IDs (slow) 🌐
NCBI provide some options for how to [download a large, custom set of records from NCBI](https://www.ncbi.nlm.nih.gov/guide/howto/dwn-records/). These include browser-based and command-line options. This last segment will introduce using Batch Entrez to download GenBank files from a browser.
##### 🛒 NCBI datasets
Before jumping into Batch Entrez, it is worth mentioning the NCBI [`datasets`](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/reference-docs/command-line/datasets/) command line tool. The goal of `datasets` is to rapidly download (as you guessed) NCBI data, given a list of accession numbers. An example command is:
```
datasets download genome accession --inputfile refsoil_ids.txt
```
Although promising and easy to use, it is restricted to downloading data hosted on "[NCBI Datasets](https://www.ncbi.nlm.nih.gov/datasets/)". So if you have data on GenBank alone, such as the RefSoil sequences, it will not work.
##### 🌐 Batch Entrez (browser download)
[batch.mp4]
The best point-and-click option is to use the browser-based [Batch Entrez](https://www.ncbi.nlm.nih.gov/sites/batchentrez) tool. You can simply upload a list of accession IDs (i.e. `refsoil_ids.txt`) and download the GenBank files in bulk.
The pros are that it is easy-to-use and does not require scripting. However, downloading from the browser can be time consuming as it retrieves a single, large file and any interruptions (such as network issues) will require downloading the file again from scratch.
🚀 Efficient Strategies for Data Retrieval from Sequence Data Sources.
How to download GenBank sequences using a list of IDs 🏎️
Here, we provide a script download_genbank.py to efficiently download GenBank files given a list of accession IDs. The script overcomes some of the previous hurdles and speeds up the process by downloading files in parallel.
Have a go at running this yourself.
Note
Ensure the command is executed from the same directory asdownload_genbank.py. You may need to runcd genbank_downloaderfirst.
Example usage
- For help and options ```bash $ python download_genbank.py Usage: download_genbank.py [OPTIONS] downloads limit accessions from GenBank Options: -p, --path PATH path to genbank accession file [required] -o, --outdir PATH path to write genbank formatted sequence files [required] -l, --limit INTEGER number of accessions to download [default: 10] --help Show this message and exit. ``` - To download GenBank (.gb.gz) files for the first 10 accessions (default limit) ```bash $ python download_genbank.py -p refsoil_id.txt -o refsoil ``` - To download GenBank (.gb.gz) files for the first 20 accessions ```bash $ python download_genbank.py -p refsoil_id.txt -o refsoil -l 20 ```
🇪❗Retrieving data from Ensembl with EnsemblLite
- [ ] To add
GavinHuttley
fredjaya
rmcar17
Draft content for block 2 of the workshop.
superset of issues #24, #25, #26.
Getting data from: