search

EarthScope / dataselect

Selection and sorting for data in miniSEED format
GNU General Public License v3.0
14 stars 7 forks source link

Dataselect writes two SDS dayfiles when -ts 2019,182 -te 2019,183 #6

Closed heavelock closed 5 years ago

heavelock commented 5 years ago

Hi! Another question/bug. I want to load 1 hour long output seeds from digitizer and convert them to a 24h long SDS file on a daily basis. In order to do that I load several files with that command:

$ ~/tools/dataselect/dataselect -v -sum -ts 2019,183 -te 2019,184 -Pr -SDS /mnt/D4To/temp/tempsds /datadir/SI.SI30.00.CHE_2019070?_??0000.miniseed

I would expect to get in return one file between midnight of 2019.183 and 2019.184. In result I get two files, for both days.

$ ls -lah /mnt/D4To/temp/tempsds
-rw-r--r-- 1 damian damian 35M juil.  5 11:10 tempsds/2019/SI/SI30/CHE.D/SI.SI30.00.CHE.D.2019.183
-rw-r--r-- 1 damian damian 512 juil.  5 11:10 tempsds/2019/SI/SI30/CHE.D/SI.SI30.00.CHE.D.2019.184

Output log confirms that those 512 bytes comes from the day 2019.184

...
Reading:  /datadir/SI.SI30.00.CHE_20190704_220000.miniseed
Files: 95, Records: 69687, Samples: 21599708
Pruning trace data
Writing output data
Wrote 0 bytes from file /datadir/SI.SI30.00.CHE_20190701_000000.miniseed
...
Wrote 1283584 bytes from file /datadir/SI.SI30.00.CHE_20190702_230000.miniseed
Wrote 512 bytes from file  /datadir/SI.SI30.00.CHE_20190703_000000.miniseed
Wrote 0 bytes from file  /datadir/SI.SI30.00.CHE_20190703_010000.miniseed
...
Wrote 35679744 bytes of 69687 records to output file(s)

Similar thing happens when I extend ts and te parameters to few days. 

$ ~/tools/dataselect/dataselect -v -sum -ts 2019,182 -te 2019,185 -Pr -SDS /mnt/D4To/temp/tempsds /datadir/SI.SI30.00.CHE_2019070?_??0000.miniseed
$ ls -lah /mnt/D4To/temp/tempsds
total 107M
-rw-r--r-- 1 damian damian 37M juil.  5 11:21 SI.SI30.00.CHE.D.2019.182
-rw-r--r-- 1 damian damian 35M juil.  5 11:21 SI.SI30.00.CHE.D.2019.183
-rw-r--r-- 1 damian damian 37M juil.  5 11:21 SI.SI30.00.CHE.D.2019.184
-rw-r--r-- 1 damian damian 512 juil.  5 11:21 SI.SI30.00.CHE.D.2019.185

If I specify times with hour, minute and seconds it behaves the same.

$ ~/tools/dataselect/dataselect -v -sum -ts 2019,182,0,0,0 -te 2019,185,0,0,0 -Pr -SDS /mnt/D4To/temp/tempsds /datadir/SI.SI30.00.CHE_2019070?_??0000.miniseed

Is that expected behaviour or a bug?

chad-earthscope commented 5 years ago

This is not expected. With those options I would expect a single record is in the day-file before the selected range (because the data starts in the earlier day but contains data from the selected range).

Can you send me the input data for the first command?

chad-earthscope commented 5 years ago

Thanks for the test data.

Now I see that this is the expected behavior given the data. The issue is that the selected time range includes the first moment of the end day (e.g. 2019,183 includes 2019,183,0,0,0.000); the times are always inclusive. The data you are using has a sample exactly at the end of the beginning of the day.

You can work around this by specifying the end time as the last sample in the desired day, e.g.:

-te 2019,183,23,59,59.999999
heavelock commented 5 years ago

Okay! Thanks for help!