Ph5ws dataselect cuts to the second, not the microsecond.

[nick-falco]

Describe the bug Ph5ws dataselect cuts to the second, not the microsecond. For MUSTANG, we want to avoid the situation above where the extra 00:00:00 sample is returned (fdsnws dataselect is inclusive of endtime), so we cut a microsecond off of our request (endtime=T23:59:59.999999) to make the R-code wrapper for dataselect non-inclusive.

To Reproduce FDSNWS returns 21600000 samples as expected: http://service.iris.edu/fdsnws/dataselect/1/query?net=YW&sta=5002&loc=--&cha=DPZ&starttime=2016-07-11&endtime=2016-07-11T23:59:59.999999&format=miniseed&nodata=404

msi -T YW.5002..DPZ.2016-07-11.2016-07-11T23.59.59.999999.mseed Source Start sample End sample Hz Samples YW_5002__DPZ 2016,193,00:00:00.000000 2016,193,23:59:59.996000 250 21600000

PH5WS cuts 249 samples from the return instead of 1, and returns 21599751 samples: http://service.iris.edu/ph5ws/dataselect/1/query?reqtype=FDSN&format=mseed&net=YW&sta=5002&loc=--&cha=DPZ&starttime=2016-07-11&endtime=2016-07-11T23:59:59.999999&nodata=404

msi -T YW.5002..DPZ.2016-07-11.2016-07-11T23.59.59.999999.PH5.mseed Source Start sample End sample Hz Samples YW_5002__DPZ 2016,192,23:59:59.999999 2016,193,23:59:58.999998 250 21599751

[derick-hess]

Hmm this is strange. I can't reproduce this on the command line.

I run the following: ph5toms -n master.ph5 --station=5002 -o mseed --channel=DPZ -s 2016-07-11T00:00:00.00 -t 2016-07-11T23:59:59.999999

and I get a miniseed file as I would expect: start time 2016 192 23:59:59.999 length 86399.996 seconds samples 21600000

[derick-hess]

The start time is as expected the current way ph5 cuts data.

[derick-hess]

And actually in this case the time is exact because there happens to be a sample exactly at 2016-07-11T23:59:59.999999

[nick-falco]

Are you running on the current master ph5 branch? Maybe I need to pull the latest for the web services.

[derick-hess]

I should note that the code base i'm using is the current master

[derick-hess]

Ah yeah. Thinking about it a bug I fixed that I was seeing not return data when you didn't give ph5toms a start time or end time could have also have fixed this issue because one line of code for determining start and end times based onb deploy/pickup times wasn't using the microseconds.

[derick-hess]

Next on my list after I get back from EGU (maybe before if I have time) is to write full unit tests for ph5toms.

[nick-falco]

On the current master I got 21600001 samples:

 msi -T YW.5002..DPZ.2016-07-10T235959.999999.ms 
   Source                Start sample             End sample        Hz  Samples
YW_5002__DPZ      2016,192,23:59:59.999999 2016,193,23:59:59.999998 250 21600001
Total: 1 trace(s) with 1 segment(s)

[nick-falco]

What is the hash of the commit you are using?

[derick-hess]

Thanks Nick. Looking at my code right now I"m working on I do have a slight change that explains this. There is a line in create_trace that should have been removed a long time ago. it specifically adds 1 sample to the end of the request. I had commented that out while testing and printing out the times that were being asked to cut, to make sure that the end of the cut times were being requested correctly

I'll make a PR to remove that line. I don't remember why we ever had that in there in the first place to request the end time + 1 sample

[nick-falco]

Okay sounds good. Will be good to get this fixed as soon as possible.

[derick-hess]

I made the change and I'm going to leave ph5toms running on a few big data sets I have here and check them later to make sure the outputs look as expected