Multiple instance of tornado are conflicting

[show0k]

Jupyter notebook is running inside a tornado server. While tornado web server of Jupyter is used, it is impossible to communicate with any other tornado instances on the same GIL.

To reproduce the issue, you can run a loop of positions queries for a simulated robot:

from poppy.creatures import PoppyErgoJr
import numpy as np
import time

robot = PoppyErgoJr(simulator='poppy-simu')

for position in np.linspace(-90, 90, 1000):
    robot.m1.goal_position = position
    time.sleep(0.001)

And if you try to view the simulated robot in poppy-simu, it won't move before the end of the movement. With @damiencaselli, we saw that even the Snap API is unreachable while the for loop is executed.

If we change the weberver of pypot HTTPServer (from tornado to the default wsgiref for example) it solves the issue.

I tried (not very long) to run tornado in another Process but it does not start at all.

We could propose to use wsgiref (default single-threaded webserver) or use another multi-threaded webserver, like cherryPy (name for the framework AND webserver).

What is better between adding another dependency (CherryPy) or beeing back with single-threaded HTTP API ? Any thought @pierre-rouanet and @damiencaselli ?

[pierre-rouanet]

While tornado web server of Jupyter is used, it is impossible to communicate with any other tornado instances on the same GIL.

I may have missed something there, but Jupyter and your Python for Snap are two different instances of Python Interpreter right? So I don't see the issue in this case?

[show0k]

Yes, you're right, the problem seems a bit more complicated. The Notebook server is not the same Python instance than the Python kernel.

But, something is blocking the kernel other tornado instances while the cell of the notebook with a robot instance is running.

With this notebook, other tornado instances are blocked when the third cell is executed. This issue is totally reproducable on many computers, architectures and OS.

But, if I run a kernel unrealated to a robot instance with a blocking loop

while True:
    pass

The HTTP API running in another kernel instance is not impacted.

So we can conclude that the issue is present only on a notebook which is running the robot instance.

We can try to understand why... I have some poor clue:

• The kernel Python instance is a process created from (fork()) Jupyter frontend Python instance, maybe the lock is inter processes.

EDIT : tornado IOLoop try to be a singleton and use the same object if it has already been created. I don't understand well how the kernel and the Jupyter Notebook server are connected but is is the issue. Some info at tornadoweb/tornado#1796

[pierre-rouanet]

Good catch! This makes more sense. Yeah I guess they manage to detect that they are on the same Python instance somehow.

[show0k]

Fixed by #216