search

swiss-seismological-service / DUGseis

Software for acquisition and processing of micro-seismic data
Other
9 stars 4 forks source link

Advice for a different datalogger #3

Open cjhopp opened 2 years ago

cjhopp commented 2 years ago

Hi @krischer , Thanks for this fantastic bit of work! I'm planning to use DUGseis for an upcoming experiment but the seismic data are recorded by a different datalogger than those that have been used by ETH in the past. I've got functions for reading the output data to obspy Streams and also for converting to hdf5 (for use with previous dugseis versions).

My question is: what initial advice would you give for inserting a different waveform reader 'plugin' into DUGseis?

  1. Just copy the binary data files to ASDF files, duplicating data, but simplifying everything else
  2. Hijack the waveform_handler to read my data format. I suspect this will touch a lot, and might not be worth the trouble, especially if there are some features of ASDF that are being leveraged and would be lost.
  3. Bit the bullet and try to write my own drivers for the cards in our datalogger (best, hardest option?). I've only got about a month to make this happen though, and I'm not sure I've got the skills.
  4. Something else entirely.

Any quick thoughts you have would be really appreciated, Chet

krischer commented 2 years ago

Hi @cjhopp,

depending on how much data it is I think option 1 would be the quickest way to get started, assuming it is feasible at all.

Otherwise both options 2 and 3 are feasible - I cannot comment a lot on option 3 as I don't know the details. I agree that option 2 is potentially a lot of work but it could most likely also be done in a modular way so that DUGSeis works with ASDF and other formats.

All of the logic actually reading the files is contained here: https://github.com/swiss-seismological-service/DUGseis/blob/main/dug_seis/waveform_handler/waveform_handler.py

All the best, Lion

cjhopp commented 2 years ago

Hi @krischer ,

  1. Unfortunately, it could be many 100s of TB by the time we're through, and this will really strain our network storage (and force us to swap them twice as quickly...). So I'm really hesitant to go with option 1 unless I just write a temporary ASDF file and then delete it after processing.
  2. This is the most likely, at this point. Looking more closely at waveform_handler.py, it might not be as much work as I thought. If I keep to the regex for file naming then these are the functions that would need editing/replicating and just need to return a Trace for a given channel. Not sure about the caching still.
  3. I'm trying to get help from one of our electrical engineers in writing straight to ASDF. We're using a Measurement Computing Vibbox acquisition system (64-chan, 100 kHz), but the acq hardware is only supported by a library within the .NET framework. I don't speak C#, so I have to rely on others to try and help on this front.

Thanks again, Chet