search

hapi-server / servers

Catalogs of known HAPI servers
0 stars 4 forks source link

Server scanning gums up CDAWeb server #8

Closed jbfaden closed 1 year ago

jbfaden commented 2 years ago

The cron job which scans the HAPI servers for their info responses and compiles the data into searchable JSON files (https://github.com/hapi-server/servers/blob/master/index/makeGiantCatalog.jy) causes problems with the CDAWeb server. I caused problems for them yesterday when running it twice while I was making changes, and without any pauses in between calls. Bernie suggested any trivial pause is not going to fix the problem.

I've disabled this scan for now.

jbfaden commented 2 years ago

I've checked my code for closing connections and it looks okay. I'm stress testing it against my server now.

jbfaden commented 2 years ago

There may be a problem on my end. Whenever there is an exception, it appears that a socket is left open:

/proc/969267/fd
spot9> ls -lrt | grep socket | wc -l
9
jbfaden commented 2 years ago

I was briefly getting a problem with the following Autoplot script, where it looks like there's some issue with the spaces:

#downloadResourceAsTempFile(URL('https://jfaden.net/HapiServerDemo/hapi/info?id=Iowa+City+Forecast'),1000,monitor)
print downloadResourceAsTempFile(URL('https://jfaden.net/HapiServerDemo/hapi/info?id=Iowa City Forecast'),1000,monitor)
jbfaden commented 2 years ago

The problem was intermittent because I assumed the 1000 meant 1000ms, when it means 1000s. If I use instead:

setScriptDescription('''This was showing strange behavior with leaving things open''')
#downloadResourceAsTempFile(URL('https://jfaden.net/HapiServerDemo/hapi/info?id=Iowa+City+Forecast'),1000,monitor)
print downloadResourceAsTempFile(URL('https://jfaden.net/HapiServerDemo/hapi/info?id=Iowa City Forecast'),-1,monitor)

it will download once every 10 seconds and shows the problem. It also appears to be leaving the socket open:

/proc/969267/fd
spot9> lsof -i | grep java
java      480609  jbf   73u  IPv6 11138833      0t0  TCP ip6-localhost:39731 (LISTEN)
java      480609  jbf  357u  IPv6 11191057      0t0  TCP localhost:8008->localhost:44574 (ESTABLISHED)
java      480609  jbf  378u  IPv6 11210663      0t0  TCP *:8082 (LISTEN)
java      480609  jbf  469u  IPv6 11187129      0t0  TCP localhost:8008 (LISTEN)
java      484230  jbf    5u  IPv4 12435670      0t0  TCP *:11555 (LISTEN)
java      484230  jbf   50u  IPv6 11190903      0t0  TCP *:8084 (LISTEN)
java      484230  jbf   55u  IPv6 11190907      0t0  TCP *:8009 (LISTEN)
java      484230  jbf   77u  IPv6 11190915      0t0  TCP localhost:8025 (LISTEN)
java      969267  jbf   52u  IPv6 16987517      0t0  TCP spot9:56532->spot8:https (CLOSE_WAIT)
java      969267  jbf   54u  IPv6 17015320      0t0  TCP spot9:56534->spot8:https (CLOSE_WAIT)
java      969267  jbf   58u  IPv6 16985468      0t0  TCP spot9:56536->spot8:https (CLOSE_WAIT)
java      969267  jbf   59u  IPv6 16985471      0t0  TCP spot9:56538->spot8:https (CLOSE_WAIT)
java      969267  jbf   60u  IPv6 17015511      0t0  TCP spot9:56540->spot8:https (CLOSE_WAIT)
jbfaden commented 1 year ago

This has been resolved.