Frontend caching - Githubissues

Features and Changes:

eida-federator implements a frontend-cache for requests of type fdsnws-station (format=text|xml)
Two different flavors of Caching-Backends are provided:
- File system backend ("CACHE_TYPE": "fs") ( i.e. cache data is stored within a directory on the file system)
- Redis backend ("CACHE_TYPE": "redis") which may be used if a distributed cache is required
Both backends allow a configurable TTL for cached data.
Cached data is stored gzip encoded
Data is cached only if the response was returned successfully and completely
The docker/prod setup currently comes along with a configured file system caching backend (TTL = 12 hours)
The implementation solves most of the disadvantages mentioned in PR #92 (e.g. works both for HTTP GET and POST requests)

Note, that caching streamed data requires temporarily buffering responses in memory.

Examples:

Issue a first request for network=CH,AW,BW,GR:

$ $ time curl -v -o /dev/null "http://mediator-devel.ethz.ch/fdsnws/station/1/query?network=CH,AW,BW,GR&format=text"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 129.132.144.211...
* TCP_NODELAY set
* Connected to mediator-devel.ethz.ch (129.132.144.211) port 80 (#0)
> GET /fdsnws/station/1/query?network=CH,AW,BW,GR&format=text HTTP/1.1
> Host: mediator-devel.ethz.ch
> User-Agent: curl/7.58.0
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0< HTTP/1.1 200 OK
< Date: Mon, 23 Dec 2019 20:10:30 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Access-Control-Allow-Origin: *
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=utf-8
< 
{ [79 bytes data]
100 49545    0 49545    0     0   1964      0 --:--:--  0:00:25 --:--:--   799
* Connection #0 to host mediator-devel.ethz.ch left intact

real    0m25.241s
user    0m0.012s
sys 0m0.036s

Issue the same request, again:

$ time curl -v -o /dev/null "http://mediator-devel.ethz.ch/fdsnws/station/1/query?network=CH,AW,BW,GR&format=text"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 129.132.144.211...
* TCP_NODELAY set
* Connected to mediator-devel.ethz.ch (129.132.144.211) port 80 (#0)
> GET /fdsnws/station/1/query?network=CH,AW,BW,GR&format=text HTTP/1.1
> Host: mediator-devel.ethz.ch
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 23 Dec 2019 20:12:37 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Access-Control-Allow-Origin: *
< Content-Length: 49545
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=utf-8
< 
{ [14273 bytes data]
100 49545  100 49545    0     0  2546k      0 --:--:-- --:--:-- --:--:-- 2546k
* Connection #0 to host mediator-devel.ethz.ch left intact

real    0m0.032s
user    0m0.009s
sys 0m0.004s

Same but now with alias net:

$ time curl -v -o /dev/null "http://mediator-devel.ethz.ch/fdsnws/station/1/query?net=CH,AW,BW,GR&format=text"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 129.132.144.211...
* TCP_NODELAY set
* Connected to mediator-devel.ethz.ch (129.132.144.211) port 80 (#0)
> GET /fdsnws/station/1/query?net=CH,AW,BW,GR&format=text HTTP/1.1
> Host: mediator-devel.ethz.ch
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 23 Dec 2019 20:13:52 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Access-Control-Allow-Origin: *
< Content-Length: 49545
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=utf-8
< 
{ [14273 bytes data]
100 49545  100 49545    0     0  2303k      0 --:--:-- --:--:-- --:--:-- 2419k
* Connection #0 to host mediator-devel.ethz.ch left intact

real    0m0.036s
user    0m0.013s
sys 0m0.004s

Same but now with differently ordered network-codes i.e.: network=GR,AW,CH,BW

$ time curl -v -o /dev/null "http://mediator-devel.ethz.ch/fdsnws/station/1/query?network=GR,AW,CH,BW&format=text"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 129.132.144.211...
* TCP_NODELAY set
* Connected to mediator-devel.ethz.ch (129.132.144.211) port 80 (#0)
> GET /fdsnws/station/1/query?network=GR,AW,CH,BW&format=text HTTP/1.1
> Host: mediator-devel.ethz.ch
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 23 Dec 2019 20:15:07 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Access-Control-Allow-Origin: *
< Content-Length: 49545
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=utf-8
< 
{ [14273 bytes data]
100 49545  100 49545    0     0  2303k      0 --:--:-- --:--:-- --:--:-- 2303k
* Connection #0 to host mediator-devel.ethz.ch left intact

real    0m0.037s
user    0m0.006s
sys 0m0.009s

EIDA / mediatorws

Frontend caching #97