AuScope / seismo-loader

MIT License
1 stars 0 forks source link

Get waveform timeseries from SDS files #44

Open bmotevalli opened 1 day ago

bmotevalli commented 1 day ago

The archive_data method calls get_waveform which is responsible to get the timeseries. However, the request pass to archive_data is pruned, ie, existing data (logged in db) are excluded. Hence, the get_waveform will not return the whole requested timespan. Instead, we always should read the waveforms from local drives, as that is how the flow should be:

get query>get_waveform>archive_data>check query exist in db>construct file patterns>read files>combine or prune for timespan

A number of additional methods are required to handle this.

filefolder commented 4 hours ago

The nice thing about the SDS archive structure is that you can create a client for it to essentially make it behave identically to a remote FDSN client. There are other handy methods in here: https://docs.obspy.org/packages/autogen/obspy.clients.filesystem.sds.Client.html including one that creates the file name / path from the NSLC and date parameters.

So, I agree that the fundamental client should always be the local archive, and the general flow should be 1) check database if the data exists locally 2) download it from remote clients if not (via prune requests) 3) archive it to local SDS 4) do all remaining analysis from local archive. We should be able to skip 1-3 quickly if the pruned requests are empty. In my original CLI there should hopefully be methods for this (or at least close to it!) already.