Closed damb closed 4 years ago
References #40.
Stats (Measuring both eida-federator's
request performance (based on explicit stream epoch granularity) and merging performance (for fdsnws-station-xml
)):
Tests were performed using the docker setup from docker/prod
(resource_config = { "fdsnws-station-xml": {"request_strategy": "combining", "num_threads": 5, "request_method": "GET", "proxy_netloc": "//federator-proxy"}}
) with the following changes:
fdsnws-station
requestsThe backend cache (HTTP caching proxy) was warmed up, previously. Requests were performed with 5 threads ("num_threads": 5
), concurrently.
Percentiles | Response start time | Response total time
(times in seconds)Query: fdsnws/station/1/query?net=Z3&start=1999-01-22&level=response&format=xml
Before (d7442a9dfc29205077e2f973b5314ed59589a27f): No statistics here. Executing a single request takes > 10 min
.
Now (081b4d69bf32d8ececcdc26b976643cccdcd10ad):
50%: [93.5212595 93.868317 ]
85%: [94.7680855 95.15366255]
90%: [95.3341337 95.7126117]
95%: [96.52142225 96.9615565 ]
99%: [100.01608135 100.36561231]
Query: fdsnws/station/1/query?net=Z3&start=1999-01-22&level=channel&format=xml
Before (d7442a9dfc29205077e2f973b5314ed59589a27f):
50%: [122.387077 122.425099]
85%: [126.12044605 126.2092826 ]
90%: [131.7131781 131.7657043]
95%: [133.0112522 133.0545596]
99%: [133.86564674 133.93073019]
Now (081b4d69bf32d8ececcdc26b976643cccdcd10ad):
50%: [94.2151695 94.2681005]
85%: [95.42866835 95.5393749 ]
90%: [96.5815141 96.6899735]
95%: [97.3710179 97.43610205]
99%: [99.38209861 99.44437263]
Response times are pretty much the same for level=channel|response
with respect to the implementation provided by this PR. Merging larger <Network></Network>
epoch elements does add only little additional computational overhead. Overall, the algorithm is still of order O(N) where N corresponds to the number of (sub-)requests issued (i.e. with "request_method": "combining"
one request per routed stream epoch).
E.g. requesting StationXML metadata for level=channel|response
using the federator leads to 1570 sub-request with responses to be merged:
$ curl -s -o - "http://mediator-testing.ethz.ch/eidaws/routing/1/query?net=Z3&service=station" | grep -v -e "^http" -e "^$" | wc -l
1570
Besides, for level=channel
response time is reduced by approximately 25% (for a warm backend cache).
Tests pass now. Going to be merged.
Features and Changes:
StationXMLNetworkCombinerTask
to increase merging performance. Instead of merging the element tree directly, use adict
based internal data structure.<Network></Network>|<Station></Station>
epoch elements are identified by their attributes, only. Before, both attributes and child elements (except of the corresponding child epoch elements) were taken into consideration when comparing.<Channel></Channel>
epoch elements are not merged and always appended.Stats follow ...