fed: Timeout for simple query (list networks in text format)

[javiquinte]

When querying the list of networks in text format there is always a timeout (Error 504). That is weird because text format does not have any complication like merging of XML responses and specially because it should be present in the cache. mediator-devel.ethz.ch/fdsnws/station/1/query?level=network&format=text

Just to compare, a query like this from a node is instantaneous. http://geofon.gfz-potsdam.de/fdsnws/station/1/query?level=network&format=text

[damb]

Hi Javier,

Thanks. I'm going to have a look on it. Reason might be, that the current implementation still relies on fdsnws_fetch. fdsnws_fetch comes along with a retry mechanism as soon as an endpoint does not respond (including a waiting time between the single request retries). As a consequence this easily leads to a gateway error (HTTP status code 504).

Currently I'm implementing a streaming version for eida-federator. I guess such problems should be fixed, since there is only a retry handling planned to be implemented for stream epoch splitting (i.e. in case of HTTP status code 413 for FDSN webservice endpoints).

cheers, Daniel

[javiquinte]

But that should be in cache?! Or am I wrong? Do you really need to contact the endpoints?

[damb]

Hi Javier,

there should be a cache, however the cache is handled by Apache/mod_wsgi. AFAIK, as soon as Apache was able to serve one request successfully the response is cached. And yes, I have to connect to the endpoints - at least once. Also, the mediator-devel.ethz.ch apache2 configuration is not prepared yet to be productive. I am sure that there are plenty of configuration parameters to be optimized.

The current federator implementation does not come along with an internal cache implementation. I also suggest to postpone such an extension since at the moment I consider it as a feature to be optimized.

cheers, Daniel

[javiquinte]

When I mentioned cache I was thinking about the stationlite. AFAIK you have it already implemented and in use. This is supposed to cache most of the inventory, that means at least network codes. So, my question is why network codes are not being served from stationlite. Please, let me know if I'm misunderstanding something

[damb]

Hi Javier, ah ok I get it :). Of course, we're already using StationLite. But we use this cache exclusively to fully wildcard- and epoch-resolve the routing information (by means of fdsnws-station). In addition the stations geo-location is cached. However, we do not cache any further data. Since StationLite currently only implements the eidaws-routing interface the federator does not have access to the fdsnws-station&format=text data. Doing so, we still have to make an endpoint request to the actual fdsnws-station endpoint.

BUT: You are right when you say: For fdsnws-station&format=text no additional endpoint request is necessary. This is a good point. Hence, for level={station,network} StationLite may implement the fdsnws-station&format=text interface. For optimization purposes I might implement this feature.

Gracias, Daniel

[kaestli]

The "stationlite cache" (mirrored stream interval & routing information) currently only has a eidaws/routing interface. Adding an fdsnws/station interface [for text response] would be a reasonable improvement (as all information required to feature the responses is available locally) Taking into account the priorities we got (first streaming, second authentication), this has to queue a bit.

[Jollyfant]

Guys to be honest this is really not an enhancement as much as it is the most basic functionality of the Federator. I do not understand what causes the problem with this particular request?

1. Query routing
2. Query fdsnws-station
3. Done

Even if you don't cache this cannot take so long right?

[kaestli]

Jollyfant: not clear to me. What is not an enhancement?

[Jollyfant]

Sorry but I meant to say that being able to successfully make the query by @javiquinte is not an enhancement (as per the label) but a requirement..

[damb]

Hi Mathijs,

besides the federator-code currently installed at mediator-devel.ethz.ch is still based on fdsnws_fetch. I'm currently working on the streaming version.

And yes it is a requirement. But, assume we handle fdsnws-station&level={network,station} more efficiently (using exclusively the StationLite routing cache without requesting data from endpoints) it is an enhancement, right?

cheerio, Daniel

[javiquinte]

There is no timeout since the "streaming" capability was introduced. Well!