fed: HTTP GET (bulk) endpoint requests

[damb]

With #75 @kaestli proposed caching of station metadata. However, in order to make use of caching eida-federator would need to issue endpoint requests via HTTP GET.

In addition, eida-federator could make use of bulk HTTP GET requests by means of implementing an intelligent merging strategy. This approach would reduce the total number of requests issued to endpoints. Though, it still has to be discussed, which merging strategy to be chosen. Certainly, there is an optimum meeting the maximum number of matches with the cache.

[Jollyfant]

Do you consider to still expand to the stream level? When you come directly from the routing service the routes are "pre-merged". E.g. one route per non-distributed network.

[damb]

Yes.

One reason (IMO one of the more important): eida-federator is a webservice providing federated data access to EIDA. Routes from eida-routing are wildcard routes. (AFAIK eida-routing does not perform something like a full wildcard expansion (with e.g. fdsnws-station) just to return afterwards fully merged results (including the usage of wildcard chars). But most of the nodes host both EIDA data and data which is not part of EIDA (available at the corresponding webservices).

As a consequence, routes from eida-routing used at endpoint webservices may return results which actually are not part of EIDA at all. The result would be simply incorrect.

Ways to solve this problem:

• Explicit routing as implemented by stationlite
• Implementation of a webservice which fully decomposes routes defined within eida-routing localconfig configuration files and finally returns fully merged routes (remember that this is not trivial at all)
• EIDA nodes host EIDA data providing dedicated services

[Jollyfant]

As a consequence, routes from eida-routing used at endpoint webservices may return results which actually are not part of EIDA at all. The result would be simply incorrect.

That is technically a configuration problem at the node right? If you specify routing for network=NL & station=* in your routing service you are saying that everything form network NL belongs to EIDA.

[Jollyfant]

Hi damb, hope you are doing well 👋 - same as #32 and can be closed too?