search

swiss-seismological-service / scrtdd

Double Difference Relocator for SeisComP
27 stars 10 forks source link

seedlink keeps trying to fetch unavailable stations #68

Closed rlmagnusson closed 1 year ago

rlmagnusson commented 1 year ago

I have some profiles where I have included events where some of the phases come from stations that are no longer running. Including these has been useful in relocating older events. I have been having trouble running these profiles as part of a real-time system. The module might relocate a couple events but will eventually get stuck trying to fetch waveform data from stations that are no longer available.

This results in a log output like this (taken from scrtdd.log):

2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: STATION VATN RK
2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: SELECT HHZ.D
2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: TIME 2022,12,02,14,29,14 2022,12,02,14,29,22
2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: STATION VIGR RK
2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: SELECT BHZ.D
2022/12/13 10:59:26 [debug/SLConnection] Seedlink command: TIME 2022,12,02,14,29,12 2022,12,02,14,29,20
2022/12/13 10:59:26 [debug/SLConnection] Seedlink: ignoring empty request for RR.RAH 00HHE.D 2022,12,02,14,29,29 2022,12,02,14,29,27
2022/12/13 10:59:26 [debug/SLConnection] Seedlink: ignoring empty request for SI.BJA 00HHZ.D 2022,12,02,14,29,28 2022,12,02,14,29,25
2022/12/13 10:59:26 [debug/SLConnection] Seedlink: ignoring empty request for SI.EDA 00HHN.D 2022,12,02,14,29,33 2022,12,02,14,29,29

In this log output there are two types of lines:

As can bee seen from the timestamps and the time period being requested by seedlink this has been going on for days. If I interrupt the program with a Ctrl + C it will abort the request and relocate the event that caused the data requests. Then it will of course exit the program.

Output from running in a shell (note the keyboard interrupt):

04:59:27 [debug] Seedlink command: STATION SKOLI VE
04:59:27 [debug] Seedlink command: SELECT 00HHZ.D
04:59:27 [debug] Seedlink command: TIME 2022,12,09,10,12,25 2022,12,09,10,12,33
04:59:27 [debug] Seedlink command: STATION VIDEY VE
04:59:27 [debug] Seedlink command: SELECT 00HHZ.D
04:59:27 [debug] Seedlink command: TIME 2022,12,09,10,12,25 2022,12,09,10,12,33
04:59:27 [debug] handshake done in 0.009174 seconds
^C04:59:30 [debug] trying to open stream combined://slink/seedlink-host:18000?timeout=10;fdsnws/fdsn-host:8081
04:59:30 [debug] Type1   : slink
04:59:30 [debug] Source1 : seedlink-host:18000?timeout=10
04:59:30 [debug] Type2   : fdsnws
04:59:30 [debug] Source2 : fdsn-host:8081
04:59:30 [debug] Params  :
04:59:30 [debug] Set timeout=300
04:59:30 [debug] Set timeout=10
04:59:30 [debug] Split   : 2022-12-13T08:59:30.233909Z
04:59:30 [debug] add stream 0 SI.HEI.00.HHZ
04:59:30 [debug] add stream 1 SI.EDA.00.HHE

From the output you can see my recordstream configuration: combined://slink/seedlink-host:18000?timeout=10;fdsnws/fdsn-host:8081

What I have tried to fix the problem:

I have not seen this behaviour in other seiscomp modules. Before, I would not bother to remove the seedlink bindings from stations that were closed as it seemed to have no effect. Maybe the answer is to have no phases in the catalog from inactive stations but this seems highly impractical due to rapidly changing seismic networks or even just regular outages.

As a final note I should mention the messaging system is seiscomp version 4.10.1. I started out with the seiscomp 4 version of scrtdd but had little success.

luca-s commented 1 year ago

You can find a solution to the problem in the documentation.

The issue is that seedlink tries to reconnect indefinitely after a failure, but you can pass the retries=0 option to avoid that behaviour.

Please let me know if that fixes your issue.

rlmagnusson commented 1 year ago

Thank you, this solved the problem.