EarthScope / ringserver

Apache License 2.0
30 stars 17 forks source link

archive update mseedindex database #46

Open crotwell opened 10 months ago

crotwell commented 10 months ago

The combination of ringserver with portable-fdsnws-dataselect and mseedindex makes a nice combination for ingesting data and making it available. But having to run an separate program periodically to index miniseed files seems inefficient. Could the miniseed archiving portion of ringserver directly update the mseedindex time series database as it writes records to file? That would also keep the database up to date regardless of when the data arrived.

chad-earthscope commented 10 months ago

Could the miniseed archiving portion of ringserver directly update the mseedindex time series database as it writes records to file?

Sure, after all it's just software!

But on a serious note: it's complicated and would be difficult to update an index in near real-time. The index is most useful and efficient when it can present a coherent picture of contiguous time series. mseedindex is designed for mostly static data sets; sure it can update when files change, but the operation is pretty heavy for applying every record written. Instead new logic would likely be better to "extend" index entries efficiently, creating when needed, and that's non-trivial.

I think the concept is a great idea, so this should be left here.