LoadPixels not working in P3D context together with JAVA2D

[cansik]

When using loadPixels() on a P3D graphics context, either the pixels are not correctly loaded or they can not be used to be drawn onto a JAVA2D canvas. Only the first frame of the P3D scene is painted correctly onto the 2D context.

The same image should appear in the P3D and the JAVA2D context. I need a JAVA2D graphics object to send it's pixels to an external device through libusb. But the sketch should be able to be a OpenGL context (even if loadpixel is slow).

Steps to Reproduce

Here is a sketch which reproduces the behaviour.

PGraphics threeDCanvas;
PGraphics pojCanvas;

void setup()
{
  size(500, 250, P2D);

  threeDCanvas = createGraphics(250, 250, P3D);
  pojCanvas = createGraphics(250, 250, JAVA2D);
}

void draw()
{
  background(0);

  // render 3d
  render(threeDCanvas);

  // load pixels of 3d
  threeDCanvas.loadPixels();

  // draw 3d content onto 2d context
  pojCanvas.beginDraw();
  pojCanvas.image(threeDCanvas, 0, 0);
  pojCanvas.endDraw();

  // show output onto onscreen canvas
  image(threeDCanvas, 0, 0);
  image(pojCanvas, 250, 0);
}

void render(PGraphics canvas)
{
  canvas.beginDraw();
  canvas.background(55);

  canvas.pushMatrix();

  canvas.translate(canvas.width / 2, canvas.height / 2);
  canvas.rotateX(radians(frameCount % 360));
  canvas.rotateZ(radians(frameCount % 360));

  canvas.noStroke();
  canvas.fill(20, 20, 20);
  canvas.box(100);

  canvas.fill(150, 255, 255);
  canvas.sphere(60);

  canvas.popMatrix();
  canvas.endDraw();
}

Environment

• Processing version: 3.3.6
• Operating System and OS version: MacOS High Sierra 10.13.3
• Other information: MacBook Pro with Radeon Pro 460 4 GB & Intel HD Graphics 530 1536 MB

[benfry]

You're not calling updatePixels().

[cansik]

Could you please give me a hint where to put the updatePixels() and on which graphics object. Because I can not getting it to run, even with loadPixels() call at the beginning on both graphic objects and updatePixels() at the end of the whole operation.

The documentation mentions, that it is only necessary to update the pixels if we write to the pixels array. Here we read from the 3D graphics object, where I use loadPixels() to load the frame into the pixels array, then drawing the pixels onto the JAVA2D context. This drawing is done by the image() method, which runs in between the beginDraw() and endDraw() context, where we should not have to call the updatePixels() method. Is this presumption correct?

[benfry]

Actually, why are you using loadPixels() at all? You need that to use the pixels[] array (after which you'd call updatePixels(), but you're not doing it here.

And why are you calling endDraw() twice on canvas?

[cansik]

The end draw was just a copy mistake, I removed it. But I use loadPixels, because otherwise the image of the 3d graphics is not drawn onto the JAVA2D context.

With loadpixels, it looks like the following (first frame is loaded from the GPU, but not more). That's the behaviour I recognise as bug. (left is 3D canvas, right JAVA2D):

Without loadpixels, nothing is loaded and so the picture is just black (which seems ok, if my understanding below is correct):

So why I used loadPixels() at all is because of my understanding, how a 3D Graphics works. As far as I know, the pixels of the 3d graphics are calculated and stored somewhere easy to reach by the GPU, accessing it from the CPU would take a lot of time. With loadPixels() I give it the explicit command to copy the framebuffer into the pixels structure of the PGraphics object (PGraphicsOpenGL.loadPixels()). This seems to work quite well, but takes a lot of time.

Now the problem is that if I would like to draw the pgraphics onto the JAVA2D context, but even with the loaded pixels, it does work just for the first framebuffer.

Maybe may understanding of the whole process is totally wrong, but if so, it would be nice to know, why loadpixels() still has an impact on the drawing behaviour in JAVA2D.

[cansik]

It seems that load pixels works quite as expected. If you add following to the loop method, it is easy to see that the pixels are correctly loaded from the colorbuffer. The pixel value is changing with the animation.

println("P: " + threeDCanvas.pixels[125 * 125]);

The JAVA2D (AWT) implementation seems to use an ImageCache to draw images. So the first time the ImageCache is initialised, then it will use this cache. Maybe there is a problem with the cache, that it won't be invalid and so it only shows the first cached frame. I will investigate there a bit further.

[cansik]

Ok, found the a way to "prove" the bug. It has todo with the static ImageBuffer cache, which is not updated even if the pixels have changed. I think the bug is either in the check if the pixels have updated, or in the OpenGL implementation, where the flag wether they have updated is not set correctly.

/**
 * Update the pixels of the cache image. Already determined that the tint
 * has changed, or the pixels have changed, so should just go through
 * with the update without further checks.
 */

PGraphicsJava2D.java#L1657-L1661

So for example, if we change the tint for the JAVA2D context every frame, the ImageBuffer gets updated and draws the frame of the PGraphicsOpenGL correctly.

// draw 3d content onto 2d context
pojCanvas.beginDraw();
pojCanvas.background(0);

// randomly change the tint to force the cache invalidation
pojCanvas.tint(random(200, 255));

pojCanvas.image(threeDCanvas, 0, 0);
pojCanvas.endDraw();

[cansik]

Even a simpler way to fix the bug is to set the flag yourself on the PGraphicsOpenGL with the setModified() method.

So my question is now, why the setModified(); is not called in the OpenGL context?

[JakubValtar]

Yeah, this is a bug in how pixels are loaded and marked modified. I need to have a look at how PImage.loaded and PImage.modified are used internally and externally by other PGraphics and probably I'll need to make two sets of these for each use. I'm gonna have a look when I have some time.

Thanks for the report @cansik and sorry for closing the issue, it looked like a programming error. We are getting lots of these and trying to close them quickly so we can get to work on real bugs, but sometimes something slips through.

[benfry]

@JakubValtar is this a regression from the pixel updates? (and if so, it's probably time to re-think how all of that works so it's less confusing…)

[JakubValtar]

@benfry it's been like this since 2.2.1 at least