search

twosigma / beakerx

Beaker Extensions for Jupyter Notebook
http://BeakerX.com
Apache License 2.0
2.8k stars 382 forks source link

race condition in plugin start #1600

Closed scottdraves closed 9 years ago

scottdraves commented 9 years ago

nginx-stderr>2015/05/12 16:39:30 [error] 27604#0: *65 open() "/Users/spot/.beaker/v1/nginx2304268377001982133/html/bb3ddbe/r.712940707513/rest/rsh/getShell" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "POST /bb3ddbe/r.712940707513/rest/rsh/getShell HTTP/1.1", host: "127.0.0.1:8801", referrer: "http://127.0.0.1:8801/beaker/"

seems to apply to all languages. pretty rare but very annoying.

scottdraves commented 9 years ago

possible that worker threads > 1 causes this? https://github.com/twosigma/beaker-notebook/blob/a66a03fe2bf73fe0d12fccef128f6bcf3b5bef98/core/nginx/nginx.conf.template#L1

scottdraves commented 9 years ago

happens less often but still possible.

scottdraves commented 9 years ago

Davide says he is sure the problem is that Jetty forks off a thread to open the port for the backend server, and it does not provide a mechanism to report when it's really ready. The function that starts it returns right away.

The best solution to this is to poll until it's ready then before reporting it back to the client. This will require another REST endpoint, but that should be easy since we have a class for all our backend servers https://github.com/twosigma/beaker-notebook/blob/master/plugin/jvm/src/main/java/com/twosigma/beaker/jvm/module/WebServerModule.java. After PluginServiceLocatorRest polls to confirm that nginx has been reconfigured, then it can poll the new endpoint.

scottdraves commented 9 years ago

i don't think that poll was correct, see better version here https://github.com/twosigma/beaker-notebook/tree/spot/race but still doesn't fix the problem??