Video: Passing a Movie into PGraphics.image() does not work across renderers (Standard/P2D)

[rapatski]

Description

When I try and draw a movie into a PGraphics object that does not share the same renderer; main sketch uses P2D renderer, PGraphics created with standard renderer, the movie frame shows up as black. When the renderers match, either both standard for the main sketch and for the created PGraphics object, or both P2D, it does work as expected.

Expected Behavior

If I pass a Movie object into a pg.image() call, I expect it to be displayed, regardless of renderers ;)

Steps to Reproduce

import processing.video.*;

PGraphics pg;
Movie movie;

void setup()
{
  size(500, 500, P2D);

  movie = new Movie(this, "launch2.mp4");
  movie.loop();

  pg = createGraphics(width, height); // video does not display
  //pg = createGraphics(width, height, P2D); // video works fine
}

//void movieEvent(Movie m) {
//  m.read();
//}

void draw()
{
  surface.setTitle("FPS: "+nf(frameRate, 1, 1));

  // update pg

  if (movie.available() == true) {
    movie.read();
  }

  pg.beginDraw();
  pg.image(movie, 0, 0, pg.width, pg.height);
  pg.endDraw();

  // draw

  image(pg, 0, 0);
}

Your Environment

• Processing version: 3.5.4 and 4.0b1
• Operating System and OS version: macOS Mojave 11.4 (20F71)
• Other information:

[trackme518]

I have verified the behaviour on different computers using AMD and NVidia card so it is HW independant.

[trackme518]

Same issue when trying to get the pixels from the Movie frame directly and using P3D.

//uncomment to enable 3D camera mouse input
//import peasy.PeasyCam;
//PeasyCam cam;

import processing.video.*;
int numPixelsWide, numPixelsHigh;
int blockSize = 10;
Movie mov;
color movColors[];

PGraphics canvas;

void setup() {
  size(560, 406, P3D ); //DOES NOT WORK
  //size(560, 406 ); //WORKS

  noStroke();
  mov = new Movie(this, "launch2.mp4");
  mov.loop();
  numPixelsWide = width / blockSize;
  numPixelsHigh = height / blockSize;
  println(numPixelsWide);
  movColors = new color[numPixelsWide * numPixelsHigh];

  //cam = new PeasyCam(this, 400);
}

PImage currFrame;

// Display values from movie
void draw() {
  background(128);

  if (mov.available() == true) {
    mov.read();

    currFrame = mov.get(); // mov.get(); // mov.copy();

    currFrame.loadPixels();

    int count = 0;
    for (int j = 0; j < numPixelsHigh; j++) {
      for (int i = 0; i < numPixelsWide; i++) {
        int loc = j + i*currFrame.width;
        //println( currFrame.width ); //works       
        //movColors[count] = currFrame.get(i*blockSize, j*blockSize); //does not work
        movColors[count] = currFrame.pixels[loc]; //canvas.get(i*blockSize, j*blockSize); //does not work as well
        //println( movColors[count] ); //shows 0 = black 
        count++;
      }
    }

    currFrame.updatePixels(); //without this call, it is not possible to display the video image with image(currFrame,0,0); in P3D
  }//mov avaliable end

//does not work - display black only
  for (int j = 0; j < numPixelsHigh; j++) {
    for (int i = 0; i < numPixelsWide; i++) {
      fill(movColors[j*numPixelsWide + i]);
      rect(i*blockSize, j*blockSize, blockSize, blockSize);
    }
  }

//works
/*
  if ( currFrame != null ) {
    image(currFrame, 0, 0);
  }
  */
}

[trackme518]

Ok, I have found an workaround - while I am still confused about the underlying issue. You can setup P3D main rendrer, then create P2D or P3D PGraphics, render movie into them and still be able to manipulate or read pixels directly from the PGraphics and display as texture in main renderer.

import peasy.PeasyCam;
PeasyCam cam;

import processing.video.*;
int numPixelsWide, numPixelsHigh;
int blockSize = 10;
Movie mov;
color movColors[];

PGraphics canvas;

void setup() {
  size(560, 406, P3D ); //DOES NOT WORK

  noStroke();
  mov = new Movie(this, "launch2.mp4");
  mov.loop();
  numPixelsWide = width / blockSize;
  numPixelsHigh = height / blockSize;
  println(numPixelsWide);
  movColors = new color[numPixelsWide * numPixelsHigh];

  canvas = createGraphics(width, height, P3D); // MUST BE P2D
  cam = new PeasyCam(this, 400);
}

PImage currFrame;

// Display values from movie
void draw() {
  background(128);

  if (mov.available() == true) {
    mov.read();

    canvas.beginDraw();
    canvas.image(mov, 0, 0, canvas.width, canvas.height);
    canvas.endDraw();

    currFrame = canvas;

    //currFrame = mov.get(); // mov.get(); // mov.copy();

    canvas.loadPixels();

    int count = 0;

    for (int j = 0; j < numPixelsHigh; j++) {
      for (int i = 0; i < numPixelsWide; i++) {
        int loc = (i*blockSize + j*canvas.width);
        movColors[count] = canvas.pixels[loc]; //canvas.get(i*blockSize, j*blockSize); //does not work as well
        count++;
      }
    }
  }//mov avaliable end

  for (int j = 0; j < numPixelsHigh; j++) {
    for (int i = 0; i < numPixelsWide; i++) {
      fill(movColors[j*numPixelsWide + i]);
      rect(i*blockSize, j*blockSize, blockSize, blockSize);
    }
  }

}

[trackme518]

Another workaround is to first render video on screen (but actually offscreen) like this: video(cam, width,0); after this line you can even manipulate pixels of the video like so: cam.loadPixels(); int pix = cam.pixels[0];