Inefficient caching of FdsnNetExtender.extend()

[andres-h]

FdsnNetExtender.extend(self, net, date_string) has lru_cache(maxsize=1000), but since date_string is different most of the time, caching seems to be inefficient. In any case, I can observe urls like http://www.fdsn.org/ws/networks/1/query?fdsn_code=3E being downloaded hundreds of times. Sometimes this causes an exception, which seems to be the reason of incomplete statistics at GFZ.

Maybe date_string should be reduced to year (two different temporary networks with the same code never exist in same year?). Alternatively I would suggest caching the result of urlopen(request).

[jschaeff]

Nice spotting. I'll fix fdsnnetextender.

[jschaeff]

fdsnnetextender 3.3.0 implements caching of the FDSN request only.

Tests are OK.

Could you upgrade only this package ? If it's working alright, then we advertise the new version to everyone, it's an important fix.

[andres-h]

Seems to work OK now.

[jschaeff]

Let it run during the week end. On monday, if everything is fine, I'll make an announce for everyone to update.

Thank you for your help !

[jschaeff]

@andres-h can you confirm that this fix is improving the situation ?

[andres-h]

@andres-h can you confirm that this fix is improving the situation ?

Absolutely. Now after 5 days it is:

$ grep ^REQUESTING send_stats.log | sort | uniq -c | sort -n -r | head 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZW 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZS 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZE 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Z3 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YZ 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XN 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9J 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9H 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9C 5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=8H

while previously:

$ grep ^REQUESTING send_stats.log.old | sort | uniq -c | sort -n -r | head 931 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Z3 687 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=3E 644 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=5C 613 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YU 397 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YZ 264 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XO 258 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Y4 229 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZA 229 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XE 151 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Y9

[jschaeff]

Thank you for your help !