search

USGS-CMG / stglib

Routines used by the USGS Coastal/Marine Hazards & Resources Program to process oceanographic time-series data
Other
17 stars 14 forks source link

add capability to remove more than one "incomplete" burst from the end of a file #180

Open rallen-usgs opened 8 months ago

rallen-usgs commented 8 months ago

In the recent vector data, I have found there is often more than one incomplete burst at the end. I'm not totally sure why, but I think perhaps it's because the instrument is connected to while a burst is being collected, and then it starts to get a new burst. The dat2cdf code works fine, but the cdf2nc code gets hung up by this, probably because there is more than one burst that is not full. I'm not sure of the best way to handle this, but at the moment I've been editing the output files by hand, which I'm sure is not the best way. Happy to submit example vector files if that would help.

ssuttles-usgs commented 8 months ago

Might be a good candidate for using multi-indexing approach that can handle incomplete burst. It will fill missing data with Nan.

Get Outlook for iOShttps://aka.ms/o0ukef

From: rmallen86 @.> Sent: Wednesday, January 17, 2024 10:19:20 AM To: USGS-CMG/stglib @.> Cc: Subscribed @.***> Subject: [EXTERNAL] [USGS-CMG/stglib] add capability to remove more than one "incomplete" burst from the end of a file (Issue #180)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

In the recent vector data, I have found there is often more than one incomplete burst at the end. I'm not totally sure why, but I think perhaps it's because the instrument is connected to while a burst is being collected, and then it starts to get a new burst. The dat2cdf code works fine, but the cdf2nc code gets hung up by this, probably because there is more than one burst that is not full. I'm not sure of the best way to handle this, but at the moment I've been editing the output files by hand, which I'm sure is not the best way. Happy to submit example vector files if that would help.

— Reply to this email directly, view it on GitHubhttps://github.com/USGS-CMG/stglib/issues/180, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJEPJ2HH5WNBJV3FGKEJINLYO7TXRAVCNFSM6AAAAABB6XDRWOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4DMNBRGUZTGMY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

dnowacki-usgs commented 8 months ago

Thanks Rachel. It would be good to see a file that has this issue. Assuming a full burst has 1024 samples, is it something like 

burstnum   1    2    3    ...  N-2  N-1  N
numsamples 1024 1024 1024 ...  1024 965  1005

i.e. the length of the two truncated bursts is random?

rallen-usgs commented 8 months ago

Yes, I think so. I'm attaching here files from DMP23XL201vec, a vector that was used simply to log an acoustic backscatter sensor (AnalogInput1). You can see that bursts 531-534 are shorter than all the others; burst 534 is the last one.

I haven't attached gatt or config files - do you want those?

DMPXLB03.zip