search

spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
143 stars 58 forks source link

MAVEN test suite tripping rate limits on LASP data server #799

Closed jameswilburlewis closed 3 months ago

jameswilburlewis commented 4 months ago

Our MAVEN test suite has started failing, with HTTP 429 "Too many requests" errors appearing after the first few successfuly downloads. Bryan Harter has confirmed that LASP is performing rate limiting on incoming requests, due to a large influx of possibly malicious traffic that was crashing their server. I've suggested whitelisting the user agents for SPEDAS and PySPEDAS, but in the meantime, we should double check that we're not making too many or too rapid queries for data availability prior to starting the actual data download.

jameswilburlewis commented 3 months ago

Now it's happening for MMS, too -- also hosted at LASP:

FAILED (failures=1)
HTTPSConnectionPool(host='lasp.colorado.edu', port=443): Max retries exceeded with url: /mms/sdc/public/files/api/v1/file_info/science?start_date=2015-10-16&end_date=2015-10-16-23-59-59&sc_id=mms1&instrument_id=fgm&data_rate_mode=srvy&data_level=l2 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96a29cb2e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPSConnectionPool(host='lasp.colorado.edu', port=443): Max retries exceeded with url: /mms/sdc/public/files/api/v1/file_info/science?start_date=2015-10-16&end_date=2015-10-16-23-59-59&sc_id=mms1&instrument_id=mec&data_rate_mode=srvy&data_level=l2&descriptor=epht89q (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96a29ad520>: Failed to establish a new connection: [Errno 111] Connection refused'))
Error: Process completed with exit code 1.
jameswilburlewis commented 3 months ago

I don't see any signs of excessive traffic being generated from the PySPEDAS MAVEN or MMS test suites. I just heard from Kris Larsen at the MMS SDC, and he says that LASP is dealing with a high level of bot traffic scraping the LASP site (more than just MMS and MAVEN). Their IT people are aware and looking at possible mitigations, and hope to have things in a more normal state soon. Nothing really for us to do except wait, for now.

jameswilburlewis commented 3 months ago

I've added some sleep() calls after each test, which may help a bit. Other than that, there's nothing else we can really do except cross our fingers and hope the server admins can find a way to mitigate the bot traffic without blocking legitimate users...