astrolabsoftware / fink-broker

Astronomy Broker based on Apache Spark
https://fink-broker.org
Apache License 2.0
68 stars 13 forks source link

HBase ingestion: update the table to access sorted data from both ID and time #401

Closed JulienPeloton closed 3 years ago

JulienPeloton commented 3 years ago

Describe the issue Currently the table has a rowkey of the form objectId_time. While searching along the objectId is fast (prefix search), search along the time axis remains quite slow (substring or regex search). To overcome this problem, one should use the fact that HBase is schemaless, and append new rows at the end of the current structure whose rowkey will be on the form t_time_objectId. In that way, one can search along both directions (at the cost to add more rows - but each is small as it does not contain columns, just the rowkey).

JulienPeloton commented 3 years ago

402