GPUImage Playing a video to Unity3d

[harmonius]

Hi All, I'm trying to use GPUImage to play a video and have it displayed in Unity3d. Brad suggested to use GPUImageOutput. I'm quite new to GPUImage and do not fully understand the way the video pipeline works. I did examine SimpleVideoFileFilter and CubeExample and try to follow, however there are big gaps in understanding. Perhaps someone with more knowledge can shine some light on this topic.

From the SimpleVideoFileFilter example, I can understand how the movie is played, and from CubeExample, I sort of understand how to play a movie to another GL context. I've setup a unity context in GPUImageOpenGLESContext.m and sent to GPUImage a renderTexture id.

Question: How can the frames coming out of GPUImage video player be bounded to the unity renderTexture.

Any help is very much appreciated.

[BradLarson]

I don't know much about Unity and what it takes in, but what the CubeExample demonstrates is taking in an image from video, filtering that, extracting the texture to use on the cube, rendering a 3-D scene, and then taking the 3-D scene in as a texture and filtering that for display. The first part of this is what you want here, but you need a way to take the OpenGL ES texture that the GPUImageTextureOutput returns and hand that to Unity.

In order for this to happen, you need to have a share group between the two OpenGL ES contexts (Unity and GPUImage). The CubeExample sample does this for GPUImage and the 3-D scene, but I'm not sure how Unity exposes this same thing. Once you've matched up the share groups, you need to take the texture from GPUImageTextureOutput and hand that to Unity as a texture. Again, I don't know that means they have for outside input.

[harmonius]

Hi Brad, thanks for your guidance.
There is another project by James at IVidCapPro http://eccentric-orbits.com/eoe/site/ividcappro-unity-plugin/ that is able to send a RenderTexture pointer from unity to GPUImage and have that written to disk using GPUImageMovieWriter.

In IVidCap, James modified some code to GPUImageContext:

• (EAGLContext *)context; { //NSLog(@"GPUImageOpenGLESContext: context called...\n");

  if (_context == nil) { // eOe start EAGLContext* unity_context = ivcp_UnityGetContext();

  _context = [[EAGLContext alloc] initWithAPI:[unity_context API] sharegroup: [unity_context sharegroup]];
  NSAssert(_context != nil, @"Unable to create an OpenGL ES 2.0 context. The GPUImage framework requires OpenGL ES 2.0 support to work.");
  [EAGLContext setCurrentContext:_context];
  // eOe end
  
  // Set up a few global settings for the image processing pipeline
  glDisable(GL_DEPTH_TEST);

  }

  return _context; }

And the actual TextureID is sent to GPUImage via his custom class IVidCap.mm

// Initialize a recording session.
void ivcp_BeginRecordingSession(char* videoName,
                               int frameWidth, int frameHeight, int frameRate,
                               GLuint textureID, int captureAudio, int capFramerateLock,
                               int bitsPerSecond, int keyFrameInterval, float gamma,
                               char* commObjectName,
                               bool showDebug)

[harmonius]

From IVidCap, It's definitely super cool to have the power and flexibility of GPUImage working with the most popular game engine Unity3d.

For testing the videoPlaying from GPUImage to Unity, I downloaded IVidCap project from Jame's website and added IVidPlayer.h and IVidPlayer.mm to IVidCap project since it already has all the hooks to work with Unity.
The code is not yet working because I don't fully understand how GPUImage does the binding to the unity Render texture.

Here is IVidPlayer.h

// // IVidPlayer.h // iVidCapPro //

import <Foundation/Foundation.h>

import <AVFoundation/AVFoundation.h>

import "GPUImage.h"

@interface ivcp_VideoPlayer : NSObject {

// Video attributes.
CGSize frameSize;
int frameRate;
int bitsPerSecond;
int keyFrameInterval;
bool useDefaultCompression;
float gamma;
bool useDefaultGamma;

NSString* videoExtension;
NSString* finalVideoFileName;

GPUImagePixellateFilter* pixellateFilter;

GPUImageMovie *movieFile;
GPUImageOutput *imageOutput;
GPUImageTextureOutput *textureOutput;

// The name of the game object on the Unity-side to which we'll be sending messages.
char iVidCapGameObject[256];

// Image pipeline.
GPUImageTextureInput *textureInput;
GLuint textureID;

//GLuint texture;

}

/* -------------------------------------------------------------------------------- -- set/get Render texture ID -- -------------------------------------------------------------------------------- */

• (void) setTextureID: (GLuint)texID;
• (GLuint) getTextureID;

@end

And IVidPlayer.mm

// // IVidPlayer.mm // iVidCapPro

import CoreGraphics/CoreGraphics.h

import QuartzCore/QuartzCore.h

import UIKit/UIKit.h

import Availability.h

import OpenGLES/EAGL.h

import OpenGLES/EAGLDrawable.h

import OpenGLES/ES1/gl.h

import OpenGLES/ES1/glext.h

import OpenGLES/ES2/glext.h

import "IVidPlayer.h"

// Use this to get the Unity GL context. extern "C" EAGLContext* ivcp_UnityGetContext();

// Use this to send messages to the iVidCapPro plugin on the Unity-side. extern "C" void UnitySendMessage(const char* obj, const char* method, const char* msg);

@implementation ivcp_VideoPlayer

• (id)init { self = [super init];

  if (self) {

  NSLog(@"ivcp_VideoPlayer init...");
  
  NSURL *sampleURL = [[NSBundle mainBundle] URLForResource:@"sample_iPod" withExtension:@"m4v"];
  
  movieFile = [[GPUImageMovie alloc] initWithURL:sampleURL];
  movieFile.runBenchmark = YES;
  movieFile.playAtActualSpeed = NO;
  
  NSLog(@"ivcp_VideoPlayer Movie URL =%@\n", movieFile.url);

  } return self; }

• (void) setTextureID: (GLuint)texID { textureID = texID; }
• (GLuint) getTextureID { return textureID; }

-(BOOL) beginPlayingSession {

NSURL* movieFileURL;

// Save the Unity GL context to restore after initializing GPUImage.
//glFlush();
glFinish();
EAGLContext* unity_context = ivcp_UnityGetContext();

//NSError* error = nil;

 NSLog(@"beginPlayingSession unity_context=%@\n", unity_context);
NSLog(@"beginPlayingSession called...");

textureOutput = [[GPUImageTextureOutput alloc] init];
textureOutput.delegate = self;

pixellateFilter = [[GPUImagePixellateFilter alloc] init];

[movieFile addTarget:pixellateFilter];
[pixellateFilter addTarget:textureOutput];

[movieFile startProcessing];

// Restore to Unity's context.
//glFlush();
glFinish();
[EAGLContext setCurrentContext: unity_context];

return YES;

}

• (void)newFrameReadyFromTextureOutput:(GPUImageTextureOutput *)callbackTextureOutput; { NSLog(@"newFrameReadyFromTextureOutput called..."); dispatch_async(dispatch_get_main_queue(), ^{ textureID = callbackTextureOutput.texture; [self renderAtInternalSize]; }); }

• (void)renderAtInternalSize; {

  //[GPUImageOpenGLESContext useImageProcessingContext]; //[self setFilterFBO];

  // [GPUImageOpenGLESContext setActiveShaderProgram:colorSwizzlingProgram];

  glClearColor(1.0f, 0.0f, 0.0f, 1.0f); glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

  // This needs to be flipped to write out to video correctly //static const GLfloat squareVertices[] = { // -1.0f, -1.0f, // 1.0f, -1.0f, // -1.0f, 1.0f, // 1.0f, 1.0f, //};

  //eOe // Rearrange vertices to cause the video to be rendered right side up. static const GLfloat squareVertices[] = { -1.0f, 1.0f, 1.0f, 1.0f, -1.0f, -1.0f, 1.0f, -1.0f, };

  static const GLfloat textureCoordinates[] = { 0.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f, };

  glActiveTexture(GL_TEXTURE4); glBindTexture(GL_TEXTURE_2D, textureID); //glUniform1i(colorSwizzlingInputTextureUniform, 4);

  //glVertexAttribPointer(colorSwizzlingPositionAttribute, 2, GL_FLOAT, 0, 0, squareVertices); //glVertexAttribPointer(colorSwizzlingTextureCoordinateAttribute, 2, GL_FLOAT, 0, 0, textureCoordinates);

  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

  glFinish(); }

@end

/* -------------------------------------------------------------------------------- -- Plugin External Interface --

-------------------------------------------------------------------------------- */

extern "C" {

static ivcp_VideoPlayer *vp;
static bool ivcp_showDebug = true;

//  Display the specified message using NSLog.
void ivcp_Log(const char* message)
{
    NSString *nString = [NSString stringWithCString:message encoding:NSUTF8StringEncoding];

    NSLog(@"%@", nString);
}

// Initialize a recording session.
void ivcp_BeginPlayingSession(GLuint textureID)
{
    vp = [[ivcp_VideoPlayer alloc] init];

    [vp setTextureID:textureID];
    NSLog(@"ivcp_BeginPlayingSession textureID=%d\n", textureID);

    [vp beginPlayingSession];

}

}