search

PIC-IRIS / PH5

Library of PH5 clients, apis, and utilities
Other
15 stars 9 forks source link

Ph5ws subtracts a microsecond from the sample timestamps #286

Closed nick-falco closed 5 years ago

nick-falco commented 5 years ago

Describe the bug Ph5ws subtracts a microsecond from the sample timestamps (the sample values returned are the same). This causes MUSTANG to return a start time earlier than the requested day boundary, depending on the metric.

To Reproduce FDSNWS: http://service.iris.edu/fdsnws/dataselect/1/query?net=YW&sta=5002&loc=--&cha=DPZ&starttime=2016-07-11&endtime=2016-07-12&format=miniseed&nodata=404

msi -T YW.5002..DPZ.2016-07-11.2016-07-12.mseed Source Start sample End sample Hz Samples YW_5002__DPZ 2016,193,00:00:00.000000 2016,194,00:00:00.000000 250 21600001

PH5WS: http://service.iris.edu/ph5ws/dataselect/1/query?reqtype=FDSN&format=mseed&net=YW&sta=5002&loc=--&cha=DPZ&starttime=2016-07-11&endtime=2016-07-12&nodata=404

msi -T YW.5002..DPZ.2016-07-11.2016-07-12.PH5.mseed Source Start sample End sample Hz Samples YW_5002__DPZ 2016,192,23:59:59.999999 2016,193,23:59:59.999998 250 21600001 Environment (please complete the following information):

  • OS: [e.g. Ubuntu 16.04]
  • Program Version [ph5toms 2018.268]

Additional context We need the PH5 dataselect service to match the behavior of the FDSN dataselect for Mustang metrics to work properly.

derick-hess commented 5 years ago

I think we have discussed this before. The fdsn miniseed is actually incorrect. The fdsn miniseed is shifted. The PH5 miniseed is correct. There is no sample at 2016,193,00:00:00.000000. The data starts at microsecond 839999 and is continuous, so PH5 is correct in that the closest sample is at microsecond 999999.

Months back I went through and checked to make sure against the raw data. The e-mail chain should be there. PH5 grabs the closest sample to the time asked. That can be just before, exactly at or just after the time asked for.

derick-hess commented 5 years ago

Looking back it was last May when we discussed this via e-mail. The FDSN miniseed data is correct only on the first day of the data. At midnight of the first day in the fdsn data the data becomes shifted and is incorrect from that point forward (for the nodes)

derick-hess commented 5 years ago

I can probably make PH5 cut to the closest sample on or after. This is pretty much a problem that is unique to the nodes, since as far as I know they are the only datalogger in common use that doesn't start recording on the second.

Would Mustang still have issues if it is trying to look at a day of data and expects the data to start at say 00:00:00.00000 but since that is between 2 samples the data for that day would really start at 00:00:00.003999? In this case you would have to expect 1 less sample than expected in a day.

nick-falco commented 5 years ago

Pinging @gillian-iris for an answer to your Mustang question

gillian-earthscope commented 5 years ago

MUSTANG will have no issue if the first sample starts at 00:00:00.003999. For the example I looked at, a request for 00:00:00 - 00:00:00 returned 23:59:59.999999-59:59.999998 for a total of 21600001 samples (one extra sample). Returning 00:00.003999-59:59.999998 would give the expected 21600000 samples. Are there other examples where this would not be the case?

derick-hess commented 5 years ago

The only cases I know of are the nodes. They are the ones that have odd recording start times. There will be a significant amount of node data coming in from now on, but they will mostly be like the example above though some may actually start recording at a time that lines up normally.

derick-hess commented 5 years ago

I will work on getting PH5 cut to cut to the closest sample after the requested time, to line up with what MUSTANG and web services expect.