Open jfuerth opened 3 years ago
This looks great! It'll allow me to indicate how to skip to other chromosomes so that clients don't need to paginate through the entire data. Three suggestions and one question:
index
(e.g. my index
will contain various chromosomes, but all chromosomes can still be paginated further)partitions
optional or at least the min
/max
so that producers don't need to pre-sort their data to figure out all pages before. To still indicate what the URL points to, you could allow for something like (using your example data):
"index": [
{
"ordered_by": [ "id" ],
"pages": [
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_1",
"id": "PMID:26833330-Jansen-2016-TMEM199-F1-II2"
},
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_2",
"id": "PMID:27435956-Naz_Villalba-2016-NLRP3-proband"
},
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_3",
"id": "PMID:27974811-Haliloglu-2017-PIEZO2-Patient"
}
]
}
]
For me, id
would be chromosome
, which would probably make more sense as an example.
index
an array of objects rather than an object, allowing for multiple sorting schemes. Clients not interested in the sorting may then just pick the first index, while those interested in sorting can choose from the options provided. Not needed for me, but this makes the format more flexible for others.And a question: your example (likely just because of the mockup) provides min
/max
values that don't seem to be sorted; they should be, right?
This issue was raised during discussion on #98.
The goals of this optional "page index" would be:
next_page_url
sequence)Background
We've experimented with something very basic along these lines with the index attribute here. This index (a nonstandard experimental extension to Search) allows consumers to jump directly to a catalog of interest without having to page through all preceding catalogs in the sequence.
Imagine something like this on a Table object, which advertises information about partitions and sortedness already built into the data:
This would indicate the data is partitioned by id and that the data consumer could skip partitions it does not need, and consume those that it does need in parallel.
This could be extended to partitions across multiple attributes, nested attributes, and even multiple available sort orders. We will have to balance complexity against performance benefits. Thanks to @ifokkema for initial feedback!
The above is just a rough illustration of the idea; don't take the exact format too seriously.