jdhayhurst
commented
2 years ago I think this is a good approach.
Just to expand on the metadata endpoint (1) I think /eqtl/api/v2/datasets should return a collection of (filterable) datasets.
Example response object:
{
'datasets': [
{
'dataset_id': 'QTD000001',
'study_id': 'QTS000001',
...
},
{
'dataset_id': 'QTD000002',
'study_id': 'QTS000002',
...
}
]
}Then there should be specific resources represented by something like this /eqtl/api/v2/datasets/QTD000001 which give more detailed responses with links:
{
'dataset_id': 'QTD000001',
'study_id': 'QTS000001',
...
'_links': {
'associtiations': {
'href': '/eqtl/api/v2/datasets/QTD000001/associations'
}
}
}If we are limiting the associations access to single datasets and never across datasets, I think the associations endpoint (2), is conceptually a sub-resource of the the dataset (the dataset is the parent of its metadata and association data): /eqtl/api/v2/datasets/QTD000001/associations. Happy to hear arguments against that.
Response from /eqtl/api/v2/datasets/QTD000001/associations would be a paginated response of all associations for this dataset, as before but without the metadata - metadata could be given in a link (/eqtl/api/v2/datasets/QTD000001).
Would anyone want to filter on chromosome without a bp limit?
It should be feasible to combine all chromosomes and make a joint index on chromosome and position and perhaps do the positional filtering in a chr:pos-pos style?
kauralasoo
Problem statement
Problem: Current API is designed around cross-dataset queries. As a result, all of the data has to be re-indexed every time a new dataset is added. This not going to scale as we keep adding new datasets to the catalogue.
Proposed solution: Refactor to the API to only support two types of queries:
Queries
1. Metadata about datasets
Proposed endpoint name:
/eqtl/api/datasetsor/eqtl/api/metadataFields
(Based on this existing file: https://github.com/eQTL-Catalogue/eQTL-Catalogue-resources/blob/master/tabix/tabix_ftp_paths.tsv)
dataset_id- unique id assigned to each combination of study_id, sample_group (previously qtl_group) and quant_method. Each unique id will correspond to a single HDF5 file. Proposed format: QTD000001.study_id- unique id assigned to each study. Proposed format QTS000001.study_name- human readable name of the study, eg GTEx or Lepik_2017 (previously study).sample_group- name for the subgroup of samples used for QTL mapping (previously qtl_group).sample_size- number of samples in the sample_group (integer).tissue_id- mapped tissue ontology id (previously tissue_ontology_id).tissue_label- short, human-readable tissue ontology labelcondition_label- short, human readable condition labelquant_method- quantification methodShould allow filtering based on dataset_id, study_id, study_name, tissue_ontology_id, tissue_label, condition label, quant_method. This table is not likely to exceed a few thousand rows.
Example queries:
Todo:
2. Summary statistics from a single dataset
Proposed endpoint name:
/eqtl/api/associaitonsQueries can only be made by specifying a dataset_id, eg
/eqtl/api/associaitons/QTD000001that maps to a unique HDF5 file.Returned fields: All fields available from the HDF5 file (https://github.com/eQTL-Catalogue/eQTL-Catalogue-resources/blob/master/tabix/Columns.md) + neg_log10_pvalue (calculated on the fly).
Probably no need to add additional dataset metdata fields (dataset_id, study_id, quant_method, etc ..)?
Filtering similar to what the current API allows.
Exact match: variant rsid molecular_trait_id gene_id chromosome
Range queries: p_lower and p_upper bp_lower and bp_upper - currently this is done on HDF5 files split by chromosome. Can we do it on a single dataset-specific HDF5 file where all chromosomes are present? This only makes sense if chromosome is already supplied. Do we need to build a joint index in HDF5 across. chromosomes and positions (as in tabix)?