search

PIC-IRIS / PH5

Library of PH5 clients, apis, and utilities
Other
15 stars 9 forks source link

Fix 460: Truncates PH5 availability at the deploy and pickup times #468

Closed timronan closed 3 years ago

timronan commented 3 years ago

Thank your for contributing to PH5

Before submitting a PR, please review the pull request guidelines: https://github.com/PIC-IRIS/PH5/blob/master/CONTRIBUTING.md#submitting-a-pull-request

What does this PR do?

Truncates PH5 availability at the deploy and pickup times. This PR prevent PH5 availability from displaying data that falls outside of the metadata epochs. The purpose of this PR is to make the output from ph5toms match the output of PH5 availability when there are data that fall outside of the deploy or pickup times. The timeseries data that fall outside of the metadata epoch are caught by ph5_validate so we will not loose a record of them.

Relevant Issues?

Issue #460

Checklist

hrotman-pic commented 3 years ago

I just remembered another case that occurs (but fairly rare): Fairfield node experiment with multiple deployment epochs created manually, instead of created when the raw data are added to the PH5. The example is from 19-024 (validated and archived). The availability output has epochs repeated for a given SNCL, like other Fairfield multiple deployments.

Here is the availability output using fix_460: ph5availability -n master.ph5 -p . --station 1110 -S -a 2 -f t

n s     l  c   q sample-rate                    earliest                      latest

1B 1110  -- GP1        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP1        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP1        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP2        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP2        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP2        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP3        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP3        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z 1B 1110  -- GP3        1000.0 2019-08-07T23:24:58.996000Z 2019-08-19T18:30:25.000000Z

Here are the deploy and pickup times from the array table for each deployment epoch. First epoch: deploy_time/ascii_s=Wed Aug 7 23:13:37 2019 deploy_time/micro_seconds_i=607000 pickup_time/ascii_s=Sat Aug 10 19:22:00 2019 pickup_time/micro_seconds_i=0

Second epoch: deploy_time/ascii_s=Sat Aug 10 19:22:00 2019 deploy_time/micro_seconds_i=1000 pickup_time/ascii_s=Thu Aug 15 16:57:00 2019 pickup_time/micro_seconds_i=0

Third epoch: deploy_time/ascii_s=Thu Aug 15 16:57:00 2019 deploy_time/micro_seconds_i=1000 pickup_time/ascii_s=Mon Aug 19 18:30:25 2019 pickup_time/micro_seconds_i=236999

timronan commented 3 years ago

The last batch of pushes solves the issues brought up of @hrotman-pic. Can you please retest this Pull request and leave a comment?

Thanks for the help.

hrotman-pic commented 3 years ago

@timronan here's a query from 19-024 that didn't result in the sample rate warning I mentioned: ph5availability -n master.ph5 -p . --station 1110 -S -a 2 -f t

Here's a query that from 19-024 did result in the sample rate warning (warning printed once): ph5availability -n master.ph5 -p . --station 1111 -S -a 2 -f t

The warning is new (and I know it is 'just' a warning), so I'm at a loss about where it came from.

timronan commented 3 years ago

Warning has been updated to say "Using sample rate from DAS Table XXXXX." The sample rate is sometimes taken from the array table and other time taken from the DAS table. In the cases where the DAS table sample rate is used, this warning will be thrown. Because this PR seems to pass all of the logical test for PH5 availability, I think it should be merged.