SeisComp having issue with beta dataselect service responce

[mabznz]

Think Seiscomp error relates to how webserver is responding to dataselect service that Seiscomp does not like.

From response Headers to service.

beta-service

byron@byron-Latitude-E7250:~/src/github.com/GeoNet/fdsn/cmd/fdsn-ws$ curl -I "http://beta-service.geonet.org.nz/fdsnws/dataselect/1/query?station=TDHS&starttime=2016-09-01T16:40:00.000&endtime=2016-09-01T16:42:00.000"HTTP/1.1 405 Method Not Allowed
Content-Length: 18
Content-Type: text/plain; charset=utf-8
Date: Wed, 28 Jun 2017 01:39:18 GMT
Server: nginx/1.10.2
Surrogate-Control: max-age=86400
Connection: keep-alive

service

byron@byron-Latitude-E7250:~/src/github.com/GeoNet/fdsn/cmd/fdsn-ws$ curl -I "http://service.geonet.org.nz/fdsnws/dataselect/1/query?station=TDHS&starttime=2016-09-01T16:40:00.000&endtime=2016-09-01T16:42:00.000"
HTTP/1.1 200 OK
Date: Wed, 28 Jun 2017 01:41:59 GMT
Content-Type: application/vnd.fdsn.mseed
Content-Disposition: attachment; filename=fdsnws.mseed
Server: SeisComP3-FDSNWS/1.1.0

This is error from Seiscomp

11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:233) [01] Content-Type: application/vnd.fdsn.mseed
11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:233) [02] Date: Tue, 27 Jun 2017 23:44:02 GMT
11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:233) [03] Server: nginx/1.10.2
11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:233) [04] transfer-encoding: chunked
11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:233) [05] Connection: keep-alive
11:44:02 [debug/FDSNWSConnection] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp:274) Content length is 0, nothing to read
11:44:02 [debug/RecordInput] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordinput.cpp:211) RecordStream's end reached

[junghao]

Some updates about this ticket:

1. The example url no longer returns empty results - I think it's because of seed files being uploaded.
2. The "curl -I" command in your example request is using "HEAD" as method which is not accepted by our new beta service so the error message you've attached has nothing to do with empty result.

The actual response for both service.geonet.org.nz and beta-service.geonet.org.nz for "no data" should looks like this (note in the query below I use year 2019 to produce a no data result):

MacBookPro:www-geonet howardw$ curl -v "http://localhost:8080/fdsnws/dataselect/1/query?station=TDHS&starttime=2019-09-01T16:40:00.000&endtime=2019-09-01T16:42:00.000"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /fdsnws/dataselect/1/query?station=TDHS&starttime=2019-09-01T16:40:00.000&endtime=2019-09-01T16:42:00.000 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< Content-Type: text/plain; charset=utf-8
< Surrogate-Control: max-age=10
< Date: Sun, 02 Jul 2017 22:55:17 GMT
<
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact

3. After investigated source code of seiscomp3, it sends a POST request then raises an error if the response is not 200: https://github.com/SeisComP3/seiscomp3/blob/master/src/trunk/libs/seiscomp3/io/recordstream/fdsnws.cpp#L221 Which doesn't seem to be anything to do with the NO DATA message above.

However I didn't installed seiscomp3 to see the context. @mabznz Would you please give more context about this issue? Thanks.

[mabznz]

@junghao Thanks. That provides a bit more clarity. Appears to be just the 'selection of data' when using Seiscomp (i know nothing about how to use it). I'll get Jerome to try a very specific selection of data to give more context.

[salichon]

@mabznz here comes the different services I used with success in seiscomp3 or not:

Working fdsnws://service.iris.edu/fdsnws/dataselect/1/query fdsnws://service.geonet.org.nz/fdsnws/dataselect/1/query (working)

Not working fdsnws://fdsn-ws-nrt.ap-southeast-2.elasticbeanstalk.com//fdsnws/dataselect/1/query fdsnws://beta-service.geonet.org.nz/fdsnws/dataselect/1/query

[salichon]

@mabznz the near real time is fdsnws://fdsn-ws-nrt.ap-southea....

[salichon]

• [ ] Here is the return logs for the the IRIS web service (only a few NZ stations are in it) OK

09:54:16 [debug/..../recordstream/fdsnws.cpp:233) [01] Server: Apache-Coyote/1.1 09:54:16 [debug/F.../recordstream/fdsnws.cpp:233) [02] content-disposition: attachment; filename=fdsnws-dataselect_2017-07-03T21:54:16Z.mseed 09:54:16 [debug/F.../recordstream/fdsnws.cpp:233) [03] Content-Type: application/vnd.fdsn.mseed 09:54:16 [debug/F...recordstream/fdsnws.cpp:233) [04] Transfer-Encoding: chunked 09:54:16 [debug/F.../recordstream/fdsnws.cpp:236) -> enabled 'chunked' transfer 09:54:16 [debug/F.../recordstream/fdsnws.cpp:233) [05] Date: Mon, 03 Jul 2017 21:54:16 GMT 09:54:16 [debug/F.../recordstream/fdsnws.cpp:233) [06] Connection: close 09:54:20 [info/RecordInput] (/home/sysop/gitlocal/bmp/jakarta-2017.124-release/seiscomp3/src/trunk/libs/seiscomp3/io/recordinput.cpp:186) End of stream detected

• [ ] Here for the "old" geonet service OK

9:59:12 [debug/F.../recordstream/fdsnws.cpp:233) [01] Transfer-Encoding: chunked 09:59:12 [debug/F.../recordstream/fdsnws.cpp:236) -> enabled 'chunked' transfer 09:59:12 [debug/F.../recordstream/fdsnws.cpp:233) [02] Date: Mon, 03 Jul 2017 21:59:09 GMT 09:59:12 [debug/F.../recordstream/fdsnws.cpp:233) [03] Content-Type: application/vnd.fdsn.mseed 09:59:12 [debug/F.../recordstream/fdsnws.cpp:233) [04] Content-Disposition: attachment; filename=fdsnws.mseed 09:59:12 [debug/F.../recordstream/fdsnws.cpp:233) [05] Server: SeisComP3-FDSNWS/1.1.0

• [ ] here for the beta-service FAIL

0:02:27 [debug/FD.../recordstream/fdsnws.cpp:233) [01] Content-Type: text/plain; charset=utf-8 10:02:27 [debug/F...recordstream/fdsnws.cpp:233) [02] Date: Mon, 03 Jul 2017 22:02:27 GMT 10:02:27 [debug/F.../recordstream/fdsnws.cpp:233) [03] Server: nginx/1.10.2 10:02:27 [debug/F.../recordstream/fdsnws.cpp:233) [04] Surrogate-Control: max-age=10 10:02:27 [debug/F.../recordstream/fdsnws.cpp:233) [05] Connection: keep-alive 10:02:27 [debug/F.../recordstream/fdsnws.cpp:274) Content length is 0, nothing to read

So in beta service there is no creation of a mini seed data volume as in the "old" service or iris . My view: -> the data could be the problem How are stored, named and created the data in beta service with regards to the "old" service ?

• [ ] here in the fdsn near real time fdsn-ws-nrt.ap-southeast-2.... FAIL :

....recordstream/fdsnws.cpp:233) [01] Content-Type: application/vnd.fdsn.mseed ...recordstream/fdsnws.cpp:233) [02] Date: Mon, 03 Jul 2017 22:14:09 GMT .../recordstream/fdsnws.cpp:233) [03] Server: nginx/1.10.2 .../recordstream/fdsnws.cpp:233) [04] transfer-encoding: chunked .../recordstream/fdsnws.cpp:233) [05] Connection: keep-alive .../recordstream/fdsnws.cpp:274) Content length is 0, nothing to read .../io/recordinput.cpp:211) RecordStream's end reached

[salichon]

@mabznz @junghao could that be the server too with regards to the other ones ? in beta -> 10:02:27 [debug/F.../recordstream/fdsnws.cpp:233) [03] Server: nginx/1.10.2

[junghao]

Can you specify what parameter you were querying when you got the error from beta service?

[salichon]

hey I requesting the data simply through the service ... (data-select) that means a time window from 2016-07-03 22:31:39 for instance ie.

10:31:59 [...recordstream/fdsnws.cpp:162) Request: 2016,07,03,22,21,39 2016,07,03,22,36,39 NZ WTVZ EHE 10 10:31:59 [.../fdsnws.cpp:162) Request: 2016,07,03,22,21,39 2016,07,03,22,36,39 NZ WTVZ EHN 10 10:31:59 [...fdsnws.cpp:162) Request: 2016,07,03,22,21,39 2016,07,03,22,36,39 NZ WTVZ EHZ 10 10:31:59 [d....recordstream/fdsnws.cpp:192) Sending request:

NZ ALRZ 10 EHE 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000 NZ ALRZ 10 EHN 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000 NZ ALRZ 10 EHZ 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000 NZ ANWZ 10 EHE 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000 NZ ANWZ 10 EHN 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000

for a bunch of NZ stations and whatever the date it s a fail i think unless i didn't undestand what s in the data bucket.

[junghao]

It looks like seiscomp doesn't like our beta service but I'm not sure what's wrong with it. I've submitted the parameters above to our beta service and it did return data without error.

MacBookPro:Downloads howardw$ curl -X POST -o output.dat --data 'NZ ALRZ 10 EHE 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000
NZ ALRZ 10 EHN 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000
NZ ALRZ 10 EHZ 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000
NZ ANWZ 10 EHE 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000
NZ ANWZ 10 EHN 2016-07-03T22:21:39.0000 2016-07-03T22:36:39.0000
' "http://beta-service.geonet.org.nz/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  361k    0  361k  100   325   433k    389 --:--:-- --:--:-- --:--:--  432k

[mabznz]

@junghao @salichon yes it seems so. We quickly talked with Richard and the problem does seem with the implementation of the beta service as a Seiscomp server. Seiscomp client does not like something about it.

Unfortunately it does not give a lot of info back about the issue apart from the response header.

I think some diagnostic work needs to be done at the Geonet implementation of a Seiscomp server end before going to the seiscomp people with a report.

[junghao]

A note for my testing:

• service seiscomp ok:

  seiscomp exec scrttv -I fdsnws://service.geonet.org.nz/fdsnws/dataselect/1/query --debug --inventory-db Jakarta_Inv_201609.xml --config-db Jakarta_Conf_201609.xml --streams.codes="NZ.APZ.*.HHZ" --end-time="2016-07-11 04:27:00" --buffer-size=86400

• service curl ok:

  curl -o /dev/null --data 'NZ APZ * HHZ 2016-07-10T04:27:00.0000 2016-07-11T04:27:00.0000' "http://service.geonet.org.nz/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  100 7670k    0 7670k    0    62  1512k     12 --:--:--  0:00:05 --:--:-- 1494k

• beta seiscomp no-data:

  seiscomp exec scrttv -I fdsnws://beta-service.geonet.org.nz/fdsnws/dataselect/1/query --debug --inventory-db Jakarta_Inv_201609.xml --config-db Jakarta_Conf_201609.xml --streams.codes="NZ.APZ.*.HHZ" --end-time="2016-07-11 04:27:00" --buffer-size=86400

• beta curl ok:

  curl -o /dev/null --data 'NZ APZ * HHZ 2016-07-10T04:27:00.0000 2016-07-11T04:27:00.0000' "http://beta-service.geonet.org.nz/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  100 7669k    0 7669k    0    62  5741k     46 --:--:--  0:00:01 --:--:-- 7431k

• local(mock-up) seiscomp ok:

  seiscomp exec scrttv -I fdsnws://192.168.1.71:8080/fdsnws/dataselect/1/query --debug --inventory-db Jakarta_Inv_201609.xml --config-db Jakarta_Conf_201609.xml --streams.codes="NZ.ABAZ.10.EHE" --end-time="2016-03-19 10:27:00" --buffer-size=86400

• local(mock-up) curl ok:

  curl -o /dev/null --data 'NZ ABAZ 10 EHE 2016-03-18T10:27:00.0000 2016-03-19T10:27:00.0000' "http://192.168.1.71:8080/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  100 10.0M    0 10.0M    0    64  6511k     40 --:--:--  0:00:01 --:--:-- 6513k

• beta-nrt seiscomp no-data:

  seiscomp exec scrttv -I fdsnws://fdsn-ws-nrt.ap-southeast-2.elasticbeanstalk.com/fdsnws/dataselect/1/query --debug --inventory-db Jakarta_Inv_201609.xml --config-db Jakarta_Conf_201609.xml --streams.codes="NZ.APZ.*.HHZ" --end-time="2017-07-13 09:27:00" --buffer-size=86400

• beta-nrt curl ok:

  curl -o /dev/null --data 'NZ APZ * HHZ 2017-07-12T09:27:00.0000 2017-07-13T09:27:00.0000' "http://fdsn-ws-nrt.ap-southeast-2.elasticbeanstalk.com/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  100 3936k    0 3936k    0    62  3104k     48 --:--:--  0:00:01 --:--:-- 5283k

• local-nrt seiscomp ok:

  seiscomp exec scrttv -I fdsnws://192.168.1.71:8080/fdsnws/dataselect/1/query --debug --inventory-db Jakarta_Inv_201609.xml --config-db Jakarta_Conf_201609.xml --streams.codes="NZ.APZ.*.HHZ" --end-time="2017-07-13 09:27:00" --buffer-size=86400

• local-nrt curl ok:

  curl -o /dev/null --data 'NZ APZ * HHZ 2017-07-12T09:27:00.0000 2017-07-13T09:27:00.0000' "http://192.168.1.71:8080/fdsnws/dataselect/1/query"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  100 12800    0 12800    0    62  1775k   8806 --:--:-- --:--:-- --:--:-- 2487k

[nbalfour]

@junghao, is your local mock up the new FDSN service running on your local machine?

[junghao]

@salichon @nbalfour We've addressed the cause of this issue. It's a bug of seiscomp3 against aws load balancing.

[salichon]

@junghao great as mentioned in hip chat, we can address this in the seiscomp3 github and maybe get a hotfix out of it :)

[salichon]

@junghao thanks a lot for the debugging .. @mabznz @gclitheroe @natalie a ticket is raised on the seiscomp3 github [https://github.com/SeisComP3/seiscomp3/issues/132] It s "only" some upper case and lower case issues in the seiscomp3 fdsnws codes ... eventually ....

[mabznz]

@junghao that's awesome work Wow, case sensitive check. I remember seeing the difference and sanity checking that http headers were case insensitive... Can be the smallest thing.

[junghao]

Gempa had a quick fix for this problem overnight (I'm guessing he planned to roll out a version so he fixed this issue together with other fixes) and has commited to Seiscomp3 source repo.

You can now build from Seiscomp3's latest master and it works with our beta-service.

[salichon]

Hi, we can close this issue:). Keep in mind that this fix will be officially available only on late 2017 Seiscomp3 version (2017.2xx) certainly.

[nbalfour]

Yes, I believe we can