THREE.js support

[thwee-alchemist]

Hi there,

Thank you for the ijavascript kernel. I use it a lot to document small experiments. I was wondering whether it'd be too much of a hassle to support THREE.js, as used here.

I know there's pythreejs, but it'd be nice to use the three,js library directly in ijavascript. Thanks for your time.

[thwee-alchemist]

Continued from #9, sorry, I haven't used the issue tracker here much.

I believe the WebGLRenderer expects a canvas element. Maybe it'd be possible to return a DOM element from $$.html if there's only a single node created. Or a canvas magic?

[n-riesco]

I believe the WebGLRenderer expects a canvas element. Maybe it'd be possible to return a DOM element from $$.html if there's only a single node created.

This is the issue. I'll try to explain without going deep into the technical details.

IJavascript runs inside a node session (it could even be a remote session), and the canvas element you want to manipulate lives in the frontend. This is the reason why IJavascript and the frontend can't exchange DOM elements.

Or a canvas magic?

I haven't had time to experiment with it, but I think, JSDOM supports now canvas elements. If this is the case, and JSDOM supports HTMLCanvasElement.toDataURL(), then something like this may work:

• create a DOM in IJavascript using JSDOM
• create a fake canvas element in this DOM
• manipulate this fake canvas using THREE.js
• convert the canvas to a data URL
• the data URL contains a PNG file encoded as base64, which is exactly what $$.png() needs.

This may not work if THREE.js doesn't accept the canvas element faked with JSOM.

If you have animations in mind, I suspect that performance may be an issue (I imagine converting to dataURL and sending the PNG file to the frontend would be slowest steps).

[thwee-alchemist]

Thanks for checking it out. I will will experiment with your suggestion and get back to you.

[thwee-alchemist]

THREE.js accepts the JSDOM HTMLCanvasElement, I used canvas-prebuilt because I had trouble installing the regular version. The resulting dataURL shows the cube when I paste it into Chrome's address bar, but $$.png() shows a broken image sign.

[n-riesco]

$$.png() takes only the encoded png; i.e. you need to skip the beginning of the dataURL. Something like this should work:

$$.png(dataURL.slice(1 + dataURL.indexOf(',')))

[thwee-alchemist]

Hey thanks for bearing with me! It looks like this method might be useful for presenting the end result of a calculation, but as you wrote, writing animations with it is currently infeasible. Without requestAnimationFrame, I put the rendering code in a setInterval, but even after slowing down the animation rate to one second, I only got the first frame to show up.

[n-riesco]

For animations, you need to use $$.display(). Most frontends only admit one call to $$.png(). Unfortunately, I havem't documented the use yet. See the discussion on #131.

You need to create a display (e.g. var $3js = $$.display("3js")) and update the display with $3js.png(...).

[thwee-alchemist]

Thanks for your time. I tried doing this on windows, but it would only render the first frame. Maybe linux will bring more luck. I've decided to put this on hold for now.

[n-riesco]

I've tested display animations in both, linux and windows, and they work (unless using an obsolete versions of jupyter and ijavascript's libraries).

Could you try this notebook and test if it works for you?

// In[1]:
// read png files and convert to base64
var png1 = fs.readFileSync("1.png").toString("base64");
var png2 = fs.readFileSync("2.png").toString("base64");

// create display
var $d = $$.display("test");

//animate
var n = 0;
setInterval(function() {
    $d.png([png1, png2][n++ % 2]);    
}, 1000);

"Display Animation"

You'll need to store the above notebook and the following png files in the same folder:

• 1.png:

• 2.png:

If this notebook doesn't work for you, then try to upgrade jupyter to a recent version and reinstall ijavascript.

If the notebook works, then the issue is in your code and you'll need to determine where:

• is the bug in the generation of the png images? can you save a few frames into files and confirm the frames are what you expect?
• is the bug in the update of the display? can you update the display with a simple animation like the above?

[thwee-alchemist]

Sorry, I've been busy with a new job. I'll try the notebook and get back to you, probably this weekend.

[thwee-alchemist]

Hmm, I downloaded the images and copied the notebook code, but no picture. Let me try upgrading jupyter and ijavascript.

[thwee-alchemist]

Ha, it works in jupyter notebook, but not in jupyter lab.

[n-riesco]

@thwee-alchemist Thank you for testing this. I've opened #161 to track the issue with JupyterLab.

[thwee-alchemist]

So I got a cube example working. Thanks for the advice! The only thing is that $.png seems to refresh the page, leading to strange scrolling behavior. Is there anything I can do on my end to prevent that?

var THREE = require('three');
var jsdom = require('jsdom');
var {CanvasRenderer} = require('three-canvas-renderer');

var dom = new jsdom.JSDOM(`
<!doctype html>
<html>
  <head>
    <title>Jupyter-FourD</title>
  </head>
  <body>
    <canvas id="canvas">
  </body>
</html>
`);

var width = 500;
var height = 300;

var canvas = dom.window.document.querySelector('#canvas');

$d = $$.display('jupyter-fourd');

var scene = new THREE.Scene();
var camera = new THREE.PerspectiveCamera(75, width/height, 0.1, 1000 );

var renderer = new THREE.CanvasRenderer({canvas: canvas});
renderer.setSize(width, height);

var geometry = new THREE.BoxGeometry( 1, 1, 1 );
var material = new THREE.MeshBasicMaterial( { color: 0x00ff00 } );
var cube = new THREE.Mesh( geometry, material );
scene.add( cube );

camera.position.z = 5;

var animate = function () {
    cube.rotation.x += 0.1;
    cube.rotation.y += 0.1;

    renderer.render( scene, camera );

    $d.png(canvas.toDataURL().slice(22));
};

var i = setInterval(animate, 60);

I know, 60 is probably too low, but is there by chance a way of implementing a png stream?

[n-riesco]

So I got a cube example working. Thanks for the advice!

That's great news. And thanks to you too for sharing your success.

The only thing is that $.png seems to refresh the page, leading to strange scrolling behavior. Is there anything I can do on my end to prevent that?

I need to test your example to find out what's causing the issue with scrolling (it's unlikely I can test it, this or the next week; please, ping me back if you don't hear from me).

I don't think the frontend refreshes the whole page with each update_display_data. I could imagine the issue being caused by the continuous removal and creation of the display element in the frontend's DOM.

Possible solutions (without testing what the cause is):

• 155 (implement clear_output): I've already started working on this issue and I expect to finish this work soon

• currently, IJavascript doesn't pass any metadata (such as the image size) when invoking $d.png(); I'd be keen to implement this feature
• if we don't find a way for the frontend to handle animations gracefully, another solution I'd be keen to explore is the use of custom messages (issue #100) (in fact, I'd love to have a project to test the implementation of this feature)

I know, 60 is probably too low, but is there by chance a way of implementing a png stream?

A png stream between the frontend and the kernel? What do you have in mind? Do you want to compress the stream on the kernel's side and expand it on the frontend? If that's the case, it sounds like something one would implement using custom messages (issue #100).

[thwee-alchemist]

Awesome. I look forward to helping out any way I can, and maybe learning something new in the process of making this work. Please note that the above example requires the canvas package to be installed.

[n-riesco]

Just a quick update: I have an implementation of clear_output that I'm planning to release soon. I haven't tested it with your example yet but I will (I got sidetracked by #164).

[thwee-alchemist]

Looks important... I look forward to seeing your clear_output. I can help test it, too.

[thwee-alchemist]

Will clear_output and png together prevent the page from refreshing and repositioning the notebook?

[n-riesco]

I don't know without testing. I'm hoping that clear_output with wait: true will prevent the repositioning because the DOM will always have an output node. But I'm not optimistic, because I don't understand what you see in your example. I don't understand why update_display_data would remove the output node for any noticeable period of time.

[thwee-alchemist]

It probably doesn't matter how long the output node is removed for, right? As long as it is removed and added, it will always cause scrolling at the interval of the animation.