search

EIDA / routing

Server side application to provide the Routing Service used in EIDA
GNU General Public License v3.0
6 stars 4 forks source link

Spatial queries for waveforms and stations #56

Closed krischer closed 5 years ago

krischer commented 5 years ago

@javiquinte just made me aware that queries such as this are possible: http://geofon.gfz-potsdam.de/eidaws/routing/1/query?minlat=-15&maxlat=15&minlon=10&maxlon=40&format=post

This functionality is actually not documented anywhere I could find:

The use case I have in mind are spatially aware waveform and station queries, e.g. I want to download all waveforms or all stations in a certain geographical domain. I'd expect min-max/lat-lon as well as the radius based queries to work. Thus one could send a single request to the routing service and then proceed straight to querying the datacenters.

In ObsPy we currently do this (to download waveforms with the eida routing service) as we assume geographical queries don't do anything for waveform routing requests:

https://github.com/obspy/obspy/blob/454f4abe734c85bfc528af2cb43499bf7343cc72/obspy/clients/fdsn/routing/eidaws_routing_client.py#L75

  1. Use the routing service to download all the station information passing on the geographical queries to the individual centers' station services.
  2. Construct a station list and send the waveform requests to all data centers.

If these geographical parameters would work as I'd expect we could get rid of the extra station queries.

javiquinte commented 5 years ago

Hi! In the official documentation this has been specified. https://routing.readthedocs.io/en/latest/#cache-of-station-names-and-locations

We haven't tested this very much and actually there is a not fixed bug ( #52 ) about querying this via the POST method. (I kept it as low priority because no one was using it).

Now I realize that the documentation is not that visible as it should be. Suggestions on how to make it more visible are welcome.

krischer commented 5 years ago

I think it should show up at least at those two places:

The specification linked to in the second link does actually document these parameters - I just never looked there.

It should also really show up here as it seems to be the official front-page of the spec: https://www.orfeus-eu.org/data/eida/webservices/routing/ and there is already a list of parameters there so IMHO no reason to not have all of them there.

damb commented 5 years ago

@krischer, eida-federator internally implements explicit routing (necessary due to the requirements given). It implements a similar API as eida-routing.

For development purposes you might use it as a mock until @javiquinte finishes his work.

It is available under:

Example:

$ cat stationlite.query 
minlat=-15
maxlat=15
minlon=10
maxlon=40
format=post

* * * * 1970-01-01 2020-01-01

Note, that at least one stream epoch must be defined.

$ curl --output ~/tmp/out.txt --data-binary @stationlite.query "http://federator.orfeus-eu.org/eidaws/routing/1/query"

Attachments:

javiquinte commented 5 years ago

Fixed in commit 3016b1764c45e870dc38594b0770b9a465f22ebf

javiquinte commented 5 years ago

I'll contact the admins of the official services to update ASAP

krischer commented 5 years ago

Thanks guys! I opened an ObsPy issue and I hope we get around to fixing this soon.

https://github.com/obspy/obspy/issues/2380

javiquinte commented 5 years ago

@Jollyfant Could you please send @krischer a message when the Routing Service at ODC is upgraded and ready to be used? I recall that there was a problem with Python3. Thanks!

Jollyfant commented 5 years ago

I wasn't planning on updating the routing service because we are also using other software that is exclusively for Python 2 (WebDC3).

javiquinte commented 5 years ago

Can I do something to help with this? Maybe provide you a version you can run by other means (outside from apache). With uswgi and proxy it? For sure we don't need it for tomorrow, but in the next couple of weeks(?) could be nice, so that Obspy and others can use the geolocatiion filters.

Jollyfant commented 5 years ago

No it's fine. It's just the fact that we don't own our servers and can't install any new software >_> meaning that I will have to request a change and that will take some time. I'll let you know.

krischer commented 5 years ago

Okay great! Can you ping me here or in the ObsPy issue once this is ready?

Jollyfant commented 5 years ago

@javiquinte I updated to routing-1.1.2-rc1. Can you test if that is the version you need?

Jollyfant commented 5 years ago

Actually do you have a tutorial on how to use uwsgi to run this? I'm very unhappy using:

uwsgi --http-socket 0.0.0.0:9090 --wsgi-file routing.wsgi

and then proxying from Apache but it randomly returns Corrupted Content Error or a routing response with random bytes of memory dumped in between (including some segfaults).

So I reverted back to the old version using mod_wsgi.

Jollyfant commented 5 years ago

So turns out that the HTTP errors were caused by a bug fixed in #59? Please check....

Jollyfant commented 5 years ago

This version comes from the releases page. Is that the version we need? Please confirm and we can ping Lion.

javiquinte commented 5 years ago

The release containing all bugfixes is v1.1.2-rc2 This should be the one to deploy.

https://github.com/EIDA/routing/releases/tag/v1.1.2-rc2

Thanks a lot again!

Jollyfant commented 5 years ago

Okay updated to rc2. I had to disable daily synchronization because of some network topology orfeus-eu.org is not reachable from the server hosting itself (???). And this version is giving me the:

WARNING:root:The URL does not seem to be a valid Station-WS
WARNING:root:http://www.orfeus-eu.org/fdsnws/station/1/query - Reason: [Errno 110] Connection timed out

So I'm going back to the IT boys..

Jollyfant commented 5 years ago

Well turns out it crashed with a segfault.. what's your experience with uWSGI because this doesn't seem stable.

!!! uWSGI process 26350 got Segmentation Fault !!!
*** backtrace of 26350 ***
uwsgi(uwsgi_backtrace+0x2e) [0x46476e]
uwsgi(uwsgi_segfault+0x21) [0x464b01]
/usr/lib64/libc.so.6(+0x36280) [0x7f502d230280]
/usr/lib64/libpython3.4m.so.1.0(+0x7fb60) [0x7f502d87db60]
/usr/lib64/libpython3.4m.so.1.0(PyErr_PrintEx+0x93) [0x7f502d9386a3]
uwsgi(uwsgi_manage_exception+0x4e) [0x44578e]
uwsgi(python_call+0x2e) [0x472b6e]
uwsgi(uwsgi_request_wsgi+0x116) [0x474da6]
uwsgi(wsgi_req_recv+0x8e) [0x41b5fe]
uwsgi(simple_loop_run+0xc4) [0x460d94]
uwsgi(simple_loop+0xe) [0x460bbe]
uwsgi(uwsgi_ignition+0x192) [0x464d52]
uwsgi(uwsgi_worker_run+0x2ed) [0x4694dd]
uwsgi() [0x469a9f]
uwsgi(_start+0) [0x41ad4e]
/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f502d21c3d5]
uwsgi() [0x41ad77]
*** end of backtrace ***
javiquinte commented 5 years ago

We don't have problems AFAI could see. Details below. Dockerfile

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y uwsgi uwsgi-plugin-python3

cat files/etc/service/routing/run exec /usr/bin/uwsgi --master --processes 8 --die-on-term --socket :9000 --uid sysop --plugins syslog,python3 --logger syslog --manage-script-name --mount /eidaws/routing/1=/var/www/html/eidaws/routing/1/routing.wsgi --mount /eidaws/routing/internal=/var/www/html/eidaws/routing/internal/routing.wsgi

Jollyfant commented 5 years ago

Can you attach the entire Dockerfile?

javiquinte commented 5 years ago

Sent via email

Jollyfant commented 5 years ago

@javiquinte I reverted back to version 1.1.1 because 1.1.2rc-2 keeps segfaulting. I noticed that this bad request segfaults the WSGI worker:

http://www.orfeus-eu.org/eidaws/routing/1/network=SL

and this is fine:

http://www.orfeus-eu.org/eidaws/routing/1/query?network=SL

Right now both are OK because I rolled back the update to a working version. Please check..

javiquinte commented 5 years ago

This is weird. We also have it in a docker container run via uwsgi and it does not segfault.

$ http http://geofon.gfz-potsdam.de/eidaws/routing/1/query?net=SL
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization
Access-Control-Allow-Origin: *`
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: WWW-Authenticate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/xml; charset=UTF-8
Date: Mon, 06 May 2019 08:33:50 GMT
Server: nginx/1.12.2
Transfer-Encoding: chunked

<service><datacenter><url>http://www.orfeus-eu.org/fdsnws/dataselect/1/query</url><name>dataselect</name><params><cha>*</cha><net>SL</net><priority>1</priority><loc>*</loc><start>1980-01-01 00:00:00</start><sta>*</sta><end /></params></datacenter></service>

And the other query:

$ http "http://geofon.gfz-potsdam.de/eidaws/routing/1/network=SL"
HTTP/1.1 400 Bad Request
Access-Control-Allow-Headers: Authorization
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: WWW-Authenticate
Connection: keep-alive
Content-Type: text/plain
Date: Mon, 06 May 2019 08:34:07 GMT
Server: nginx/1.12.2
Transfer-Encoding: chunked
javiquinte commented 5 years ago

Could you try deleting the third parameter (sys.exc_info()) in line 293 of wsgicomm.py?

start_response(status, response_headers)

I'm really puzzled because it is impossible for me to reproduce the error. Would it be posible to deploy it in another container where I can debug it?

Jollyfant commented 5 years ago

This is weird. We also have it in a docker container run via uwsgi and it does not segfault.

I'm not using a container yet but I might after I get back from Potsdam if that works 100% for you.

javiquinte commented 5 years ago

Okay great! Can you ping me here or in the ObsPy issue once this is ready?

Hi @krischer . @jbienkowski contacted me to inform that ODC is running the latest version of the Routing Service. This means that EIDA is officially supporting the spatial query. It has been tested for some time without problems, so feel free to use it if you want. Let us know if there is any kind of problem.