UM-Bridge (the UQ and Model Bridge) provides a unified interface for numerical models that is accessible from virtually any programming language or framework.
Issue #48 was about sporadic model crashes on Helix. When using load-balancer to start the servers, the server occasionally fails to start because the randomly selected port is in use.
Debugging
I found that some ports were vacant when checking and occupied when the server was trying to use them. It should be the case that some processes occupied the port just between the checking and using.
I noticed that some jobs did try to select a new port during the test, so the checking part should be correct.
Solution
I tried to occupy the port using nc -l $port & when checking and then release it just before the start of the server, but the releasing using fuser -k -n tcp $port was not stable (on Helix). Therefore, I didn't use this approach.
Instead, I added a timeout check in job.sh to see if the script waits for a server to respond exceeds $timeout seconds. If so, the script will call itself again to restart the server.
Since the issue #48 only happens (around) once every hundred times, the usage time should not increase significantly. Also, since there's a time limit for HQ jobs set in the job.sh, this retry won't repeat infinitely.
Just need to notice that if there are too many servers need to restart, it might be because that the $timeout set in job.sh is too small to start the server.
Other changes
Added a PORT check in the Makefile before executing the load-balancer. Since there was no error message when the port of load-balancer was occupied. (Also set HQ_SUBMIT_DELAY to 100ms)
Added my codes for debugging using MultiplyBy2 in the test folder.
The issue
Issue #48 was about sporadic model crashes on Helix. When using
load-balancerto start the servers, the server occasionally fails to start because the randomly selected port is in use.Debugging
I found that some ports were vacant when checking and occupied when the server was trying to use them. It should be the case that some processes occupied the port just between the checking and using.
I noticed that some jobs did try to select a new port during the test, so the checking part should be correct.
Solution
I tried to occupy the port using
nc -l $port &when checking and then release it just before the start of the server, but the releasing usingfuser -k -n tcp $portwas not stable (on Helix). Therefore, I didn't use this approach.Instead, I added a timeout check in
job.shto see if the script waits for a server to respond exceeds$timeoutseconds. If so, the script will call itself again to restart the server.Since the issue #48 only happens (around) once every hundred times, the usage time should not increase significantly. Also, since there's a time limit for HQ jobs set in the
job.sh, this retry won't repeat infinitely.Just need to notice that if there are too many servers need to restart, it might be because that the
$timeoutset injob.shis too small to start the server.Other changes
load-balancer. Since there was no error message when the port ofload-balancerwas occupied. (Also setHQ_SUBMIT_DELAYto 100ms)MultiplyBy2in thetestfolder.