search

ioos / registry

Getting data services registered in the IOOS Service Registry
http://ioos.github.io/registry/
2 stars 7 forks source link

High number of CDIP THREDDS request per day from IOOS Catalog #75

Open robragsdale opened 9 years ago

robragsdale commented 9 years ago

Issue with CDIP THREDDS getting approx. 340,000 request per day from m.catalog.ioos.us:

We notice that the CDIP THREDDS service is getting roughly 340,000 request a day from 198.199.71.152 (m.catalog.ioos.us). It does not seem like it’s affected our uptime but it does seem excessive when for a few hours IOOS is requesting 1300 times every minute. Our logs show that it’s repeating each SOS requests 479 times at a rate of 5-6 a second. In addition, it looks like the SOS is failing to harvest http://catalog.ioos.us/services/filter/CDIP. Anything we can do?

robragsdale commented 9 years ago

Luke, this does seem excessive. Do you have enough info from the email to investigate or do you need something else from Darren? Is there something special about the CDIP harvesting or are we hitting every THREDDS instance this often?

robragsdale commented 9 years ago

@lukecampbell Ok, so this is new. There's something wrong with the SOS harvester, and it's causing an infinite recursion of attempts.http://catalog.ioos.us/services/54ba01928c0db305aeb6183d Ben, can you please look at this today?

robragsdale commented 9 years ago

@benjwadams I'm on it.

benjwadams commented 9 years ago

Recursion/request flood should be fixed now by https://github.com/ioos/catalog/pull/396.

Unfortunately, many of the CDIP SOS datasets are not harvesting. Will have to look into this in a separate issue in catalog.