EIDA / mediatorws

EIDA NG Mediator/Federator web services
GNU General Public License v3.0
6 stars 6 forks source link

Reimplement StationXMLNetworkCombiner #108

Closed damb closed 4 years ago

damb commented 4 years ago

Features and Changes:

Stats follow ...

damb commented 4 years ago

References #40.

damb commented 4 years ago

Stats (Measuring both eida-federator's request performance (based on explicit stream epoch granularity) and merging performance (for fdsnws-station-xml)):

Tests were performed using the docker setup from docker/prod (resource_config = { "fdsnws-station-xml": {"request_strategy": "combining", "num_threads": 5, "request_method": "GET", "proxy_netloc": "//federator-proxy"}}) with the following changes:

The backend cache (HTTP caching proxy) was warmed up, previously. Requests were performed with 5 threads ("num_threads": 5), concurrently.


Query: fdsnws/station/1/query?net=Z3&start=1999-01-22&level=response&format=xml

Before (d7442a9dfc29205077e2f973b5314ed59589a27f): No statistics here. Executing a single request takes > 10 min.

Now (081b4d69bf32d8ececcdc26b976643cccdcd10ad):

50%: [93.5212595 93.868317 ]
85%: [94.7680855  95.15366255]
90%: [95.3341337 95.7126117]
95%: [96.52142225 96.9615565 ]
99%: [100.01608135 100.36561231]

Query: fdsnws/station/1/query?net=Z3&start=1999-01-22&level=channel&format=xml

Before (d7442a9dfc29205077e2f973b5314ed59589a27f):

50%: [122.387077 122.425099]
85%: [126.12044605 126.2092826 ]
90%: [131.7131781 131.7657043]
95%: [133.0112522 133.0545596]
99%: [133.86564674 133.93073019]

Now (081b4d69bf32d8ececcdc26b976643cccdcd10ad):

50%: [94.2151695 94.2681005]
85%: [95.42866835 95.5393749 ]
90%: [96.5815141 96.6899735]
95%: [97.3710179  97.43610205]
99%: [99.38209861 99.44437263]

Response times are pretty much the same for level=channel|response with respect to the implementation provided by this PR. Merging larger <Network></Network> epoch elements does add only little additional computational overhead. Overall, the algorithm is still of order O(N) where N corresponds to the number of (sub-)requests issued (i.e. with "request_method": "combining" one request per routed stream epoch).

E.g. requesting StationXML metadata for level=channel|response using the federator leads to 1570 sub-request with responses to be merged:

$ curl -s -o - "http://mediator-testing.ethz.ch/eidaws/routing/1/query?net=Z3&service=station" | grep -v -e "^http" -e "^$" | wc -l
1570

Besides, for level=channel response time is reduced by approximately 25% (for a warm backend cache).

damb commented 4 years ago

Tests pass now. Going to be merged.