"runtime disconnect" in Colab for BufferGeometry(...)

[RezaRob]

Here is the link:
https://colab.research.google.com/drive/1ZdOYamFGclGdPDbIkXExF8KRYQ6dAdGu

When running, cubeGeometry = BufferGeometry(...)

it results in "runtime disconnect."

Any idea why or how to fix it?

[jasongrout]

Does Colab support widgets beyond the core HTML widgets? My understanding is that it did not support any custom widgets.

[RezaRob]

Does Colab support widgets beyond the core HTML widgets? My understanding is that it did not support any custom widgets.

@jasongrout I'm not sure what "custom widgets" are, but Colab is displaying this 3D mesh really nicely and interactively, and I believe it's also threejs based: https://colab.research.google.com/drive/1dX-Jr2YKtYtUxdBy9Pvp_0_XxZyMmuIM

So, I'm wondering how we can get a custom mesh, cubes, etc. working.

[RezaRob]

Okay, I found this wonderful gist that shows how to render geometric primitives in the Trimesh library: https://gist.github.com/wkentaro/a2e92f6e52c418080c00ef8992c46e37

And it works really well in Colab too: https://colab.research.google.com/drive/1n0UiVcmG1Ivajb2lBvXf5Uzh9nNtWU4g

That should give us some ideas, but I'm still not certain about a geometry buffer.

[jasongrout]

trimesh looks like it is not a Jupyter Widgets library, i.e., it allows rich display, but not interactivity back to the kernel (i.e., if I understand things right, you can't get a python function to run based on interaction with it in the front end, and you can't see the state of the display in the kernel).

IIRC, Colab does not support Jupyter Widgets other than the core set of widgets (https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20List.html). In particular, it does not support pythreejs.

[jasongrout]

In order to use pythreejs, you need to use a frontend that supports Jupyter Widgets libraries, such as Jupyter Notebook or JupyterLab.

[RezaRob]

trimesh looks like it is not a Jupyter Widgets library, i.e., it allows rich display, but not interactivity back to the kernel (i.e., if I understand things right, you can't get a python function to run based on interaction with it in the front end, and you can't see the state of the display in the kernel).

hmm... so, Trimesh can create a javascript interactive "applet" and the server (Colab) can pipe that to the frontend (my browser) but cannot pipe back my interactions all the way back to some python event handler on the server?

Did I get that right? Is that what's happening?

If so, it'd be really interesting to have such functionality added to Colab I suppose. Presumably it's not much harder than what it's already doing.

[jasongrout]

Did I get that right? Is that what's happening?

Yes, essentially. It's relatively easy to send javascript to the browser. It's much harder to have the js then send messages back to the python kernel and update and maintain state in the kernel reacting to what is happening in the frontend. That's essentially the huge difference between rich display and Jupyter widgets.

If so, it'd be really interesting to have such functionality added to Colab I suppose. Presumably it's not much harder than what it's already doing.

They implemented the basic controls, but letting you install arbitrary other javascript that communicates with the kernel is much more complicated. Regardless, this is a discussion for the colab repo and @blois (one of the colab devs who worked on implementing their current widget support).

[jasongrout]

I should add that if you don't need the interactivity, i.e., you don't need things in the kernel to react to what you are doing in the frontend, then by all means use rich display. It is much simpler.

[RezaRob]

I should add that if you don't need the interactivity, i.e., you don't need things in the kernel to react to what you are doing in the frontend, then by all means use rich display. It is much simpler.

Okay, thanks. I know you said this discussion belongs elsewhere, but I'll just add that primarily, I need "one way" interactivity, i.e. the ability to modify and refresh the display state from python in a "loop" to play a "video."

That would permit certain visual displays of Tensorflow output.

I haven't yet found a way to do that properly with Trimesh.

2-way interactivity is a good to have that's also useful.

Anyway thank you.

[blois]

This is essentially a combination of https://github.com/googlecolab/colabtools/issues/587 and https://github.com/googlecolab/colabtools/issues/498. I've been pulled in various directions lately but have been mulling approaches for addressing these and will be focusing on them again very shortly.

In the meantime there are some examples of bi-directional communications in https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb.

I'm really hoping we can get something together that can work well in all environments.

[RezaRob]

This is essentially a combination of googlecolab/colabtools#587 and [googlecolab/colabtools#498] [...] I'm really hoping we can get something together that can work well in all environments.

Thanks a lot. The Colab, "heatmap view of modern era player stats" was informative and interesting: so in principle one could invoke threejs through javascript to visualize data (if I understood that correctly), provided one isn't interested in communication between javascript and python. That's really useful, although now I'd have to get the javascript/threejs/etc. working right.

It would still be very nice to have the communication and widgets working so the interaction with python and Tensorflow is smooth.

Thank you so much for all you've done. I love Colab and suppose it's very useful and convenient for students.

[RezaRob]

In the meantime there are some examples of bi-directional communications in https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb.

Hmm... these look really interesting! Can I just invoke threejs in a %%javascript block and then call python periodically to get updated data for display?

[blois]

Yes, you can do that. If packaging as Python it may be easier to use the display function:

from IPython.display import Javascript
display(Javascript('''...'''))

[RezaRob]

Yes, you can do that. If packaging as Python it may be easier to use the display function:

from IPython.display import Javascript
display(Javascript('''...'''))

I had trouble doing that. I asked on StackOverflow and got an answer which demos the three.js really well: https://stackoverflow.com/questions/60907999/embedding-three-js-in-colab

However, I'm still not sure how to link it to python. Your examples show some communication between python and javascript, but we need to pass a whole numpy array to three.js, and I'm not yet sure how.

Thanks a lot for replying.

[blois]

You could encode the numpy array as base 64 or just JSON and include it in the generated Javascript, for example this is how we pass pandas DataFrames for our interactive table view: https://github.com/googlecolab/colabtools/blob/master/google/colab/data_table.py#L207.

Alternatively you can use the server approach and serve a binary blob to the notebook: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=_7dYIo63EdgL. Note the x-colab-notebook-cache-control header if you want to avoid saving the data in the notebook.

[RezaRob]

You could encode the numpy array as base 64 or just JSON and include it in the generated Javascript, for example this is how we pass pandas DataFrames for our interactive table view: [...]

Alternatively you can use the server approach and serve a binary blob to the notebook

The first option, and if I'm not mistaken the second option as well, appear to generate static html. How do you update the data dynamically, for example javascript periodically calling python to get a changing numpy array? Do these options accomplish that? It appears they might not.

I'm also not sure how to serve a binary blob with that "class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler)"

I didn't find the right documentation.

[blois]

This example shows calling Python from Javascript: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=Ytn7tY-C9U0T

In the server example self.wfile.write is just writing a binary blob. In the browser you can use browser fetch to request it: fetch('https://localhost:${port}/')

[RezaRob]

In the browser you can use browser fetch to request it: fetch('https://localhost:${port}/')

Using fetch in the animate() function breaks the function. What am I doing wrong? If you comment out fetch, the graphics works: https://colab.research.google.com/drive/1qYa2iQdVMj8VVHt6-L1Pvoq0GwkGVN2A

[blois]

Here's a tweaked example: https://colab.research.google.com/gist/blois/f73b31d604b5ad84797aff2317bb26b3/threejs.ipynb

Fetching the data on every animation frame is going to be a lot of data. You should probably move the rescheduling of requestAnimationFrame to the end of the animate function.

Really annoying that the JS syntax highlighting is getting dropped in that script type=module block. Feel free to open a bug in https://github.com/googlecolab/colabtools/issues/new for that.

[RezaRob]

Here's a tweaked example: https://colab.research.google.com/gist/blois/f73b31d604b5ad84797aff2317bb26b3/threejs.ipynb

blois, thanks a lot!

It didn't work in Firefox. I get, "TypeError: channel.loadInfo is null"

But seems to work fine in Chrome. However, I can't pass a numpy array: https://colab.research.google.com/drive/1Gp5z6Queg90KYgNSTcfteIv_KV3ABBjz

Using numpy_array.tobytes() on the Http server returns something that appears to have zero length:

console.log(binaryData.byteLength) gives 0 in the browser console.

More importantly, I don't really know how to debug it because print() inside the http server doesn't work and I don't know how to log anything from python into the browser console either.

Fetching the data on every animation frame is going to be a lot of data. You should probably move the rescheduling of requestAnimationFrame to the end of the animate function.

Great tip! I should have paid attention to it sooner!

Really annoying that the JS syntax highlighting is getting dropped in that script type=module block. Feel free to open a bug in https://github.com/googlecolab/colabtools/issues/new for that.

Thanks.

[blois]

Setting the content-length header allowed the binary data: https://colab.research.google.com/gist/blois/97efe8fc938fe4055e2263514ec32065/copy-of-three-js-http-server-demo.ipynb

[RezaRob]

Setting the content-length header allowed the binary data:

That is awesome blois! Thanks! Could you just briefly comment on how you found the bug, or how I could have debugged this?

By the way, so that's for Chrome. Apparently things are different in Firefox (perhaps some as of yet unsupported feature or something).

[blois]

I looked at the network tab in Chrome's devtools, filtered for _proxy (I happen to know that all of these requests will end up with that in their path), and noticed that the content-length in the response was 0.

You can also use !curl to make the request from the VM.

[vidartf]

Closing this discussion as complete :)