bmpvieira
commented
10 years ago Hi @olgabot and @mlovci,
I think for the first major issue you could do something like Dat -> JSON -> Pandas. That is, use Dat for versioning, archiving, distribution, etc, and Pandas for analysis. However, maybe Dat could replace Pandas and provide functionality similar to flotilla when https://github.com/maxogden/dat/issues/133 get's solved?
For the second major issue, you could for now build a pipeline/tool (have a look at gasket) with bionode-ncbi and dat. A bionode-ensembl module is planned, but for now you can use ncbi and do bionode-ncbi search gene ENSG00000100320 | dat import --json. The property name of the object returned by the search has the value RBFOX2 and taxid has the taxon id 9606 corresponding to Homo sapiens. If you do bionode-ncbi search genome txid9606 in the property assembly_name you get the common name of the reference assembly GRCh38. If you do bionode-ncbi search assembly GRCH38 You get an array in meta.FtpSites.FtpPath with two objects. The one that has type RefSeq has property _ with value ftp://ftp.ncbi.nlm.nih.gov/genomes/H_sapiens/. In that FTP folder there's a directory named GFF with the annotations.
Alternatively, you can just update bionode-ncbi and do bionode-ncbi download gff homo sapiens :-P
As we keep adding features to these projects, things will get simpler. What I'm trying to do in my PhD is similar to this use case, first get all metadata and store in Dat, second reformat and query the metadata to find what's interesting to me, third send the reference IDs to the cluster so that the pipeline fetches all the raw data and runs the heavy analysis there.
max-mapper
mlovci
jbenet
webmaven
olgabot
thadguidry
gpratt
brainstorm
saketkc
joehand
Hi @jbenet and @maxogden! Thank you so much for the time you took to meet with @mlovci and me this weekend. Here's an overview of our current data management situation and what our ideal case would be.
What we have now
Currently, we host a/datapackage.json", which we hand-curated. So if the data we use is from one of these two species, we can grab the data.
datapackages.jsonfile which containsresourceswith the names"experiment_design"(metadata on the samples, e.g. celltype and colors to plot them with),"expression"(gene expression data),"splicing"(scores of which version of a gene was used). Then, at the end of the file, we have an attribute called"species"(e.g."hg19"for the human genome build 19) that only works withhg19andmm10(Mus musculus aka house mouse genome build 10) because it points to the URL "http://sauron.ucsd.edu/flotilla_projects/Try this:
On a command line:
In Python:
This will load the data from our server from sauron.ucsd.edu, and since you haven't downloaded anything with that filename yet, it will download it. Additionally, this is a test dataset with only information from human chromosome 22, so it is loadable on a regular laptop. Feel free to look through the code and JSON files.
flotilla.data_model.Study.from_data_packagedoes most of the heavy lifting in loading the data. Keep in mind that the project is also in a pre-alpha stage, and has a long way to go :)What we would like
Two major issues are:
neural_diff_chr22datapackage into apandas.DataFrameobject which can then be imported intoflotilla.datapackage.jsonfile for that file, but it should first check locally for the data and be able to be loaded offline, if you already have the data downloaded.ENSG00000100320and get the "gene symbol" (i.e. the familiar name that we know it by) of RBFOX2 and that this gene is an RNA-binding protein involved in alternative splicing and neural development."species"attribute, but ideally it would be something likeENSEMBL_v75_homo_sapienswhich would link to the human data here: http://uswest.ensembl.org/info/data/ftp/index.html and then grab gene annotation (gtffiles)/sequence information (fastafiles) as necessary by the analysis.Ideally, we could do something like this:
Which would fetch our mouse and human neuron data, which has some kind of link to ENSEMBL and attach all the relevant metadata about mouse and human genes, and give common keys where possible.
@mlovci - please add on with anything I missed.