search

SeisComP3 / seiscomp3

SeisComP is a seismological software for data acquisition, processing, distribution and interactive analysis.
Other
111 stars 88 forks source link

fdsnws: service hangs on SDS request (part 3) #168

Closed petrrr closed 6 years ago

petrrr commented 6 years ago

Unfortunately we observe an other case of very similar problems, as already reported in #163 and #167.

Apparently, the SDS request handler ends up in some infinite loop, never returns and therefore the FDSNWS service does stop replying. The problem is again due to a corrupt file, but this case is still no catched.

Here the relevant log:

2018/04/06 21:52:08 [info/log] [reactor] 93.63.207.206 - - [06/Apr/2018:21:52:08 +0000] "POST /fdsnws/dataselect/1/query HTTP/1.0" 200 5508
096 "-" "Wget/1.14 (linux-gnu)"
2018/04/06 21:52:15 [debug/log] request (93.63.207.206): /fdsnws/dataselect/1/query
2018/04/06 21:52:15 [debug/log] ro: ['MN'].['*'].['*'].['LH?'] 2008-10-30T23:55:00.0000Z 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.AIO..LHN 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.AIO..LHZ 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.AIO..LHE 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
[...]
2018/04/06 21:52:15 [debug/log] adding stream: MN.VLC..LHE 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.VLC..LHN 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.VLC..LHZ 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
[...]
2018/04/06 21:52:15 [debug/log] adding stream: MN.DIVS..LHE 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.DIVS..LHN 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/log] adding stream: MN.DIVS..LHZ 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/06 21:52:15 [debug/RecordStream] trying to open stream sdsarchive:///mnt/archive_aufs
2018/04/06 21:52:15 [debug/SDSARCHIVE] SDS request: 2008,10,30,23,55,00 2008,11,01,00,05,00 MN AIO LHN 
2018/04/06 21:52:15 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/AIO/LHN.D/MN.AIO..LHN.D.2008.304 (not found)
2018/04/06 21:52:15 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/AIO/LHN.D/MN.AIO..LHN.D.2008.305 (not found)
2018/04/06 21:52:15 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/AIO/LHN.D/MN.AIO..LHN.D.2008.306 (not found)
[...]
2018/04/06 21:52:16 [debug/RecordStream] trying to open stream sdsarchive:///mnt/archive_aufs
2018/04/06 21:52:16 [debug/SDSARCHIVE] SDS request: 2008,10,30,23,55,00 2008,11,01,00,05,00 MN VAE LHZ
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VAE/LHZ.D/MN.VAE..LHZ.D.2008.304 (not found)
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VAE/LHZ.D/MN.VAE..LHZ.D.2008.305 (not found)
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VAE/LHZ.D/MN.VAE..LHZ.D.2008.306 (not found)
2018/04/06 21:52:16 [debug/RecordStream] trying to open stream sdsarchive:///mnt/archive_aufs
2018/04/06 21:52:16 [debug/SDSARCHIVE] SDS request: 2008,10,30,23,55,00 2008,11,01,00,05,00 MN VLC LHE
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHE.D/MN.VLC..LHE.D.2008.304 (init:1)
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHE.D/MN.VLC..LHE.D.2008.305 (init:0)
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHE.D/MN.VLC..LHE.D.2008.306 (not found)
2018/04/06 21:52:16 [debug/RecordStream] trying to open stream sdsarchive:///mnt/archive_aufs
2018/04/06 21:52:16 [debug/SDSARCHIVE] SDS request: 2008,10,30,23,55,00 2008,11,01,00,05,00 MN VLC LHN
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHN.D/MN.VLC..LHN.D.2008.304 (init:1)
2018/04/06 21:52:16 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHN.D/MN.VLC..LHN.D.2008.305 (init:0)
2018/04/06 21:52:16 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/06 21:52:16 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/06 21:52:16 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/06 21:52:16 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/06 21:52:16 [error/SDSARCHIVE] file read exception: Invalid miniSEED header

And the compromised file: 

$ qmerge -n MN.VLC..LHN.D.2008.305
Error: unable to get header from block 394 from stream MN.VLC..LHN.D.2008.305
gempa-jabe commented 6 years ago

I could reproduce it.

gempa-jabe commented 6 years ago

It is hopefully fixed with 68b567e8a059321388687062c805bd87656e9aff. It is working for me.

gempa-jabe commented 6 years ago

Are there any updates on your side? Did the fix help?

petrrr commented 6 years ago

@gempa-jabe: Thanks for your fix and the headup!

We applied the fix some days ago. The problem we observed is gone and so far we have not observed any other similar problems. Here the related log for the same request as reported above.

2018/04/12 17:17:31 [debug/log] request (93.63.207.206): /fdsnws/dataselect/1/query?net=MN&sta=VLC&cha=LHN&start=2008-10-30T23:55:00&end=2008-11-01T00:05:00&nodata=404
2018/04/12 17:17:31 [debug/log] adding stream: MN.VLC..LHN 2008-10-30T23:55:00.0000Z - 2008-11-01T00:05:00.0000Z
2018/04/12 17:17:31 [debug/RecordStream] trying to open stream sdsarchive:///mnt/archive_aufs
2018/04/12 17:17:31 [debug/SDSARCHIVE] SDS request: 2008,10,30,23,55,00 2008,11,01,00,05,00 MN VLC LHN 
2018/04/12 17:17:31 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHN.D/MN.VLC..LHN.D.2008.304 (init:1)
2018/04/12 17:17:31 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHN.D/MN.VLC..LHN.D.2008.305 (init:0)
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [error/SDSARCHIVE] file read exception: Invalid miniSEED header
2018/04/12 17:17:31 [debug/SDSARCHIVE] + /mnt/archive_aufs/2008/MN/VLC/LHN.D/MN.VLC..LHN.D.2008.306 (not found)
2018/04/12 17:17:31 [debug/log] fdsnws-dataselect: returned 202240 bytes of mseed data
2018/04/12 17:17:31 [info/log] [reactor] 93.63.207.206 - - [12/Apr/2018:17:17:30 +0000] "GET /fdsnws/dataselect/1/query?net=MN&sta=VLC&cha=LHN&start=2008-10-30T23:55:00&end=2008-11-01T00:05:00&nodata=404 HTTP/1.0" 200 202240 "-" "curl/7.58.0"

I think this issue could be closed. Sorry for the delay in reporting back.

gempa-jabe commented 6 years ago

Thank you for confirmation. I will close the ticket and include the fixes into a new patch release.

petrrr commented 6 years ago

Thanks!