Closed junghao closed 1 month ago
Developer's note:
Test against Fastly and origin:
Fastly:
% curl -o test2.ms "https://service.geonet.org.nz/fdsnws/dataselect/1/query?station=WQAB&channel=HN?&endtime=2024-03-16T02:10:56&starttime=2024-03-16T01:56:56&format=miniseed&nodata=204"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.2M 0 10.2M 0 0 152k 0 --:--:-- 0:01:09 --:--:-- 0
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Origin:
% curl -o test.ms "http://tf-prod-fdsn-web-2006946694.ap-southeast-2.elb.amazonaws.com/fdsnws/dataselect/1/query?station=WQAB&channel=HN?&endtime=2024-03-16T02:10:56&starttime=2024-03-16T01:56:56&format=miniseed&nodata=204"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.2M 0 10.2M 0 0 104k 0 --:--:-- 0:01:40 --:--:-- 335
We can see that download from origin takes longer, and I have observed that the last 30 seconds of download against origin is a bit stalled (average speed kept dropping due to no packet received, current speed became 0). However, the download finished after the 20~30seconds stall.
Then check the downloaded file size:
% ls -l test*.ms
-rw-r--r-- 1 howardw GNS\g_howardw 10785280 31 Jul 10:18 test.ms
-rw-r--r-- 1 howardw GNS\g_howardw 10783950 31 Jul 10:19 test2.ms
The file downloaded via Fastly is 1330 bytes short.
I'm guessing this query returned many files, and many of the files don't contain the data during the time period user requested. Thus the last packet took quite long to scan thru many files, which caused a timeout for Fastly.
Next step:
Tested on dev with RDS restored from prod snapshot and miniseed files copied from prod bucket:
curl -o test.ms "http://tf-dev-fdsn-web-287969587.ap-southeast-2.elb.amazonaws.com/fdsnws/dataselect/1/query?station=WRAS&channel=HN?&endtime=2024-03-16T02:10:56&starttime=2024-03-16T01:56:56&format=miniseed&nodata=204"
The query is successful with a total time of 1:31
.
The requested station has 96
files and each with a size of ~20 MB
, the connection timeout on the server is 10
minutes, while that on Fastly is 1
minutes, which resulted in a timeout for the query on the prod service.
Increase the timeout for FDSN Fastly service.
Applied terraform and tested working
@ozym reported an issues
Just noticing some errors turning up with the FDSN service, they are appearing in some of the strong motion processing. They look like:
in amongst valid data. This query was enough to trigger one:
But it doesn't seem to happen all the time, this query run again may be okay. The ECS strong build application logs looks like below, the "PROTOCOL_ERROR is coming from the dataservice of FDSN".