Bug in the availability service (timespan inclusion)

[jschaeff]

(submitted on behalf of Helle)

Dear all,

I am trying to promote the new availability service, but I think I found a bug at the seiscomp based datacenters (or at least at BGR and GEOFON have the problem, RESIF and IRIS don't - I didn't test further).

See some links below: I suspect that the algorithm takes the first file starting AFTER requested start time until the last file that ends BEFORE the requested end time. This means that if a time window is requested which is embedded in, for example, a daily file, the data is not detected. I think (but didn't take the tests very far...) that a window across two days, for example a 24h window from mid-day to mid-day, doesn't work either.

This is a real problem for example if a user wants to check which networks and stations have data for an earthquake.

I will leave it to the ETC specialists to decide how to manage this issues with tickets, gitlab or whatever - but I'd be grateful go get any feedback and updates!

Best

Helle

---

GEOFON

https://geofon.gfz-potsdam.de/fdsnws/availability/1/query?format=text&net=GE&sta=FLT1&cha=HHZ&starttime=2019-01-01T01:00:00&endtime=2019-01-04T02:00:00&orderby=nslc_time_quality_samplerate&includerestricted=true&nodata=404

-> big window : OK. we observe there are data on Jan2nd

https://geofon.gfz-potsdam.de/fdsnws/availability/1/query?format=text&net=GE&sta=FLT1&cha=HHZ&starttime=2019-01-02T01:00:00&endtime=2019-01-02T4:00:00&orderby=nslc_time_quality_samplerate&includerestricted=true&nodata=404

-> Small window within the big window: doesn't work; the fact that there are data inside the big file of Jan 2 is not identified, so the user thinks there is no data..

BGR

https://eida.bgr.de/fdsnws/availability/1/query?format=text&net=GR&sta=FUR&cha=HH?,BH?&starttime=2019-01-02T04:00:00&endtime=2019-01-06T06:00:00&orderby=nslc_time_quality_samplerate&includerestricted=false&nodata=404

https://eida.bgr.de/fdsnws/availability/1/query?format=text&net=GR&sta=FUR&cha=HH?,BH?&starttime=2019-01-04T04:00:00&endtime=2019-01-04T06:00:00&orderby=nslc_time_quality_samplerate&includerestricted=false&nodata=404

RESIF

https://ws.resif.fr/fdsnws/availability/1/query?format=text&net=G&sta=SSB&cha=HH?,BH?&starttime=2019-01-02T04:00:00&endtime=2019-01-02T06:00:00&orderby=nslc_time_quality_samplerate&includerestricted=false&nodata=404

-> works

[jbienkowski]

Hi @jschaeff,

Got it, it's on our radar, more info will follow soon.

Cheers, J

[javiquinte]

(I sent this per mail, but will repeat here) Could this be related to the bug of a materialized view in wfcatalog, that was spotted and fixed some time ago?

[jbienkowski]

It seems to be an issue with timestamp comparison in the Mongo query, but I need to debug it to be sure.

[jbienkowski]

Hi @jschaeff and @javiquinte,

I think the problem is fixed now:

Full day query (one long segment in the DB): extent?net=NL&sta=HGN&cha=BHZ&start=2018-01-06T00:00:00&end=2018-01-07T00:00:00

#Network Station Location Channel Quality SampleRate Earliest                    Latest                      Updated              TimeSpans Restriction
NL       HGN     02       BHZ     D       40.0       2018-01-06T00:00:00.000000Z 2018-01-07T00:00:00.000000Z 2018-01-08T05:58:07Z 1         OPEN

Sub-segment query: extent?net=NL&sta=HGN&cha=BHZ&start=2018-01-06T06:00:00&end=2018-01-06T12:00:00

#Network Station Location Channel Quality SampleRate Earliest                    Latest                      Updated              TimeSpans Restriction
NL       HGN     02       BHZ     D       40.0       2018-01-06T06:00:00.000000Z 2018-01-06T12:00:00.000000Z 2018-01-07T06:11:20Z 1         OPEN

I am now making sure all other cases work as well. It would be fantastic if you could test the ODC instance as well and let me know if you find anything funny.

Thanks, Jarek

[javiquinte]

Thanks @jbienkowski ! Quick question, once you confirmed this is solved, does each node need to deploy this patch, or upgrade the code, at their side. Am I right?

[jbienkowski]

Yeah, I'm gonna ask my colleagues to review this and once we know the fix is good, I'll broadcast the news to ETC. The update itself takes 20s it's just git pull && docker build at each node running it.

[jschaeff]

Hello, this is great news. I made just a few requests and the bug seems fixed. Good job :)