search

NOAA-ORR-ERD / LibGOODS

Library for accessing data useful for the NOAA / GNOME model
https://libgoods.readthedocs.io/en/latest/
Other
1 stars 2 forks source link

NOAA servers possibly blocking me #9

Open kthyng opened 2 years ago

kthyng commented 2 years ago

I'm having more issues connecting to NOAA servers lately (COOPS, NCEI, NOMADS), I think because I have code that now preemptively checks the model output availability and therefore pings the server a bunch of times to do that.

Given that you all have been using these servers for a long time, do you have an established way around it? For example, maybe you are on a white list or just have a method to avoid getting blocked? I am not permanently blocked, but periodically.

AmyMacFadyen commented 2 years ago

Ugh, that's annoying. With GOODS I never wrote any automation for testing servers so it was individual requests only. I guess we'll have to try to find contacts. Do you think this is more of an issue when testing (e.g. repeatedly running code to test implementation) or do you think it'll be a more persistent issue ?

kthyng commented 2 years ago

Ah, that makes sense. Yes, it happens when I'm testing to make sure all the connections are still possible, not when I'm just using it more like a human. I was going to start with putting in a pause to the test loop to see if that alone would fix the problem, but wanted to see if there was an established better solution too. How about I'll reach out again if it seems like we need to escalate?

kthyng commented 2 years ago

@AmyMacFadyen Ok so I have tried with sleep statements of various lengths and still hitting issues. Is there a way for me to get on a white list or something? This is rather interrupting my progress unfortunately. Alternatively, I could decide to comment out a suite a tests for "future use" and narrow the scope of my tests so I can at least move past this issue for now.

AmyMacFadyen commented 2 years ago

Is there a particular server that is problematic or all the NOAA servers you mentioned? I have a contact for the NOMADS one and COOPS but I'd have to do some digging for NCEI.

kthyng commented 2 years ago

CO-OPS has been the biggest problem but maybe just because of what I was working on at the time. Could you start with your existing contacts at COOPS and NOMADS and then if NCEI is a problem we could deal with it then. I narrowed down my tests so they pass but I would like to be able to run through all model combinations to make sure they all work eventually.