Attempting to run the GL and QPU Examples

[peyton-howe]

I have compiled vc4asm and VC4CV on a Raspberry Pi CM3+ Lite and I am receiving errors when attempting to run the examples listed in the Commands.md file. Any GL commands I try throw the following error message: * failed to add service - already in use?

Any QPU commands I run throw this type of error message, which always ends with the line "QPU enable failed!": FrameBuffer: vc4drmfb; 1920x1080, 16 app, 4147200 bytes, Line Length 3840! Type: Packed Pixels, Visual: True Color, Mode: Non-Interlaced! Pixel Format: R: 5 << 11; G: 6 << 5; B: 5 << 0! SETUP: 10 instances processing 1/2 columns each, covering 80x60 tiles, plus 0 columns dropped QPU enable failed

I'm not sure if this is related to the vc4 display driver being enabled rather than the vcio2 driver, or if it is something else.

[peyton-howe]

Update: Installing the vcio2 driver from maazl's website fixed the error problems. I am now able to get the GL commands and QPU commands to run. However, not all is fixed. When running any of the three GL commands, all I see is a black screen with a grey line through it. I know that the camera is running since the LED on the camera module lights up. Any ideas as to what would be causing the black screen?

[peyton-howe]

Update #2: Tried running the GL commands without the vcio2 driver installed, instead relying only on the legacy driver. Results were the same, still a black screen with a grey line through it. Not sure what the issue is, but I am on kernel 5.10.63-v7+.

[Seneral]

My first step would be to check if the shader output is fine. You can try modifying the "glshaders/CamES/frag*" shaders to output a solid color or something. I'll just assume that will work. Next step would be to check if the QPU programs can display the camera output (after testing the basic QPU samples first ofc). If they don't have a problem, then I have no idea but it would drastically narrow the problem. So assuming they have a problem, of course the next option would be to check if the camera works at all using other tools, such as raspivid, assuming you haven't already done so. That would greatly narrow the problem down, so I can try to help. Don't have a compute module here to test though

[peyton-howe]

Upon further investigation, everything is actually working as intended, the camera feed is just extremely dark for the GL commands. I'm not sure why this is the case, but the camera feed is barely visible unless it is pointed directly at a light source. Sometimes, when running the GLBlobs command, a visible greyscale video feed will appear, but it seems random as to when that will work. Is there any way to increase the brightness of the camera feed so that it is always visible? Also, is it written to be in full color and I just can't see it because of how dark it is, or does the code just pass the Y frame, like with the --luma parameter in raspividyuv?

I also tested the QPU commands, and all of them are working as well, after uncommenting the #define RUN_CAMERA and #define USE_CAMERA statements at the top of main_qpu.cpp. For the QPU commands, the camera feed is not dark like the GL commands, but instead looks like a normal greyscale camera feed. When looking through the main_qpu.cpp code, I noticed that only the Y component was written, so I was wondering if there are any plans to incorporate the U and V components so that the video feed is in color?

Another thing to note with the QPU commands is that the cleanup for the video display is not very good. The frame will stay in place and visible until a menu or file is hovered over or another window frame is dragged in its place.

[Seneral]

Ahh good catch, didn't think of that. For either GL and QPU you can just adjust the shutter speed with "-s", which for some reason isn't set by default in the GL examples. Not sure what the camera would default to. For GL, you can change the format easily with "-c Y" for fast greyscale, "-c YUV" for color, and "-c RGB" for converted color if you must (with a performance penalty, as the GL driver/backend will somehow convert it first). For QPU, it is greyscale only for now. Changing this would involve changing the QPU programs, and since each is quite a bit of work, I didn't do that since I personally didn't need it yet. Do note if you CAN do your processing in greyscale, you can still do that and later in the CPU look up the color of individual pixels very quickly. But changing it in QPU is not too difficult, you need to redesign them a bit, but essentially you just need to keep track of new addresses to read from with the TMU. Since the color planes UV are binned 2x2 (or 2x1?), you'll likely find it easiest to process the pixels in groups of 4 or 2. Essentially with any QPU program you'll have to start with how you want the data to flow, whether you need multiple QPUs or are sufficient with a single QPU, what the access patters will be and how you can structure the QPU program around those. Last is important since you are very limited with the registry accesses which cannot be parametrized, so e.g. for my where I basically needed a circular buffer of length 5 I had to literally copy the code 5 times using the preprocessor macros (.rep).

Likely you'll have to restructure any of the programs if you want to have color, just for optimal TMU access patterns alone. If you can tell me what you need it for and stuff, I might have some ideas how to best do that.

Oh and cleanup for QPU, yes, I chose to not integrate it in windows since I use it headless and really just want any output (and writing to framebuffers is the fastest way). If you care about desktop integrations, you'll have to figure out the windowing system to use yourself and how to integrate it. As a quick and dirty option, perhaps forcing a screen refresh after you're done is enough (not sure how though), but that won't allow you to drag windows over it while it's running.

[peyton-howe]

I have discovered the source of the darkness issues with the GL programs after a lot of trial and error. It was not related to using a -s or -i parameter in the command line. Instead, I discovered the issue to be in the code for the gcs.c file, which was resolved by commenting out line 94 under "Mess with the ISP blocks" and commenting out lines 105-122, which set the exposure and AWB to constant values if the camera parameters had disabled EXP and AWB. I noticed that with the QPU programs, this is not an issue since main_qpu has the following statements in lines 44-46:

.disableEXP=false, .disableAWB=false, .disableISPBlocks=0

These parameters do not exist in the main_gl or main_gl_blobs programs, which I think is causing the set values to be used, resulting in the extremely dark frames. When I tried adding in the lines into the CamGL_Params section and tried recompiling, I got the following error:

error: request for member 'disableEXP' in '1', which is of non-class type 'int'

Not sure if it is necessary to add in these lines or just leave the lines commented out in the gcs.c file, since the GL programs work as intended and the QPU programs remain unaffected.

My overall aim with this is to create a VR display using two cameras on the compute module to produce stereo vision. Getting just the regular camera feed to display works fine using picamera or raspivid, but I also need to apply barrel distortion in real-time so the feed is not distorted by the lenses needed to view the screen. I tried using OpenCV, but the ARM CPU is nowhere near strong enough to apply the distortion at 1080p and 30fps, which is what I need to maintain the VR display. Using the built-in image effects works perfectly in real-time, so I am assuming the GPU is capable of doing the processing, but those are locked down by broadcom and I have no way of reproducing those.

After extensive research, I have been pointed to use the GPU and GL shaders or use the GPU to do the distortion, which takes some work but is doable. I'm assuming the easiest method would probably be to just use a barrel-distortion shader and use the GL programs, but I'm not sure if performance will be good enough, and I may have to rely on utilizing the QPU. Any ideas on what would be the best route for applying the barrel distortion?

[Seneral]

Oh that makes... some sense. Something I probably implemented later and didn't fix in the GL part. Sorry about that.

That's interesting, really like that. Yeah GL will likely add too much latency at the very least, and will be unable to do 1080p@30 at worst. On the other hand, implementing on the QPU will be truly a challenge.

So first, let's see what you need to do. You need a YUV->RGB conversion, calculating the distorted pixel coordinates, sample the pixels, and blend them. If you do it with GL shaders, you can tell it to do the conversion beforehand, and easily calculate the distortion and sample the image. Should be pretty easy to implement, so I recommend you start with that and see what results you can get. With QPU, you'll have to do all that yourself. And it won't be easy, I can tell you that much. You can't use the texture sampling+blending hardware easily, since the TMUs require the texture to be in a very specific mutated format for that to work. So you really would have to calculate the distorted floating point coordinate, find the four nearest pixels, for each of those gather the Y, U and V component separately, then blend them accordingly, and output. That's a LOT of complexity. Not impossible, but hard to design fast and error-prone.

Note that that complexity is the reason as to why the GL way is going to be slow, it also has to do that, just less efficiently. I'd say there's no loss in trying GL first. It's easy. Then see if you need to use the QPU manually.

[peyton-howe]

Ok, I have attempted to modify the vert.glsl shader to apply barrel distortion when using ./GLCV, but it is not really working as intended. I found an example shader for producing barrel distortion from these two websites and I am trying to merge it into the vert.glsl shader. https://www.geeks3d.com/20140213/glsl-shader-library-fish-eye-and-dome-and-barrel-distortion-post-processing-filters/2/ https://prideout.net/barrel-distortion

So far, this is the code that I have come up with for the shader:

#version 100

attribute vec3 vPos;
attribute vec2 vTex;

vec3 Distort(vec3 p)
{
    vec3 v = p.xyz;
    float radius = length(v);
    if (radius > 0.0)
    {
        float theta = atan(vTex.y, vTex.x);
        radius = pow(radius, 1.0);
        v.x = radius * cos(theta);
        v.y = radius * sin(theta);
        v.z = p.z;
    }
    return v;

void main()
{
    gl_Position = vec4(Distort(vPos), 1.0);
    uv = vTex;
}

I tried using vPos.x and vPos.y for calculating theta, but I just got the image as a normal box, where the number I put in the radius determined the zoom of the image. The current code produces a weird shape that is distorted, but nothing resembling a radial distortion at all. I am not sure what I am missing or how to make this distortion radial, but on the bright side, performance is running at a smooth 30fps at 1080p, which is promising. Here is a link to what the distortion looks like https://imgur.com/a/GYRBgUS

[Seneral]

Oh yeah, not vert, frag The way screenspace shaders work is by having 4 vertices in the corners and the fragment shader then handles each fragment (that is, all the pixels this quad spans - the whole screen). What you need to distort is the vec2 uv in the fragment shader.

Alternatively, you CAN do it in the vertex shader, but then you need to have a higher amount of vertices for interpolation. Look Method 1 and 2 on this page for more info: https://smus.com/vr-lens-distortion/

Finally, that is not barrel distortion, I don't know what it is but I don't think you need trigonometry functions for that distortion Here's simple radial distortion: http://vr.cs.uiuc.edu/node211.html There's also tangential distortion, but unless you REALLY calibrate your cameras and the viewing lenses, 2 or 3 degrees of radial distortion is as much as you'll need.

Note that you have a weird setup that's hard to calibrate except by trial and error, since you need to correct for the camera distortions and viewing lens distortions, but I'm sure you'll figure something out

[peyton-howe]

Okay, I have gotten distortion to kind of work using the frag shader. I used the code listed above and distorted the uv vector, which kind of worked, but I see what you mean about it not being barrel distortion. The pixels in the middle of the screen are distorted correctly, but the entire image is not shrunk down into the barrel-look, only the pixels within a certain radius are distorted. I tried using the references you listed for barrel distortion, but those did not work as intended either.

Performance wise, I am only able to get 12fps at 1080p, which is not adequate enough. In the link you mentioned about using the vertex shader, they state that the fragment shader is the slowest since it has to be applied to every pixel, which would be why I'm only getting 12fps. Performance wise, distortion in the vertex shader is the fastest and could possibly provide 30fps. Any idea on how I would do the distortion in the vertex shader to test if it performs better?

[Seneral]

Yeah the uvs will be centered at 0.5,0.5 instead of 0,0 That was not unecpected, what you can do is change the mesh that is rendered from a single quad to a grid in the code. Then move the distortion code back to the vertex shader (or even the CPU if you want, applying it directly on the grid). Best to make the grid dynamic so you can try out different resolutions quick, at some point you shouldn't notice that it's interpolated anymore.

[peyton-howe]

Do you have any examples for a shader that would do that? It doesn't have to be a barrel distortion shader, I just need an example that I can play with so that I can determine how the shader works since I am still learning how to use them.

[Seneral]

You already pretty much had it with the vert shader, except not modify the vert position but uv. The frag should be the same as it is on the repo, it just needs to map uv and sample the texture. So all you need to do is set up a grid of vertices, modifying this should suffice: https://github.com/Seneral/VC4CV/blob/master/main_gl.cpp#L104 Just fill an array of width x height x 5 floats with tuples of (x, y, 0, x, y) with x,y in [-1,1]x[-1,1]

Then in the vert shader you can modify these uvs according to your distortion. Make sure to then map it to [0,1]x[0,1] for the fragment shader.

If you want to improve performance, then you can save yourself the square root in the shader by writing it into the unused z coordinate. In the vert you'd have to make sure to set it to 0 after using it in the algorithm. Alternatively, you can write all the distortion code into the mesh uvs (so in fact you do not need to modify the shaders from the repo at all), you'd have to update them every time you update the distortion coefficients at runtime though (in constrast to just updating a shader parameter) which you'll have to figure out yourself, the Mesh class I wrote is not designed for this I think. It involved updating some vertex buffer objects.

[peyton-howe]

Bear with me, but I am still pretty new to working with openGL and shaders, so I am not entirely sure what you mean by filling an array and then mapping to that array in the fragment shader. I tried just changing the values in the existing code:

    SSQuad = new Mesh ({ POS, TEX }, {
        -1,  1, 0, -1, 1,
        -1,  1, 0, -1, 1,
        -1,  1, 0, -1, 1,
        -1,  1, 0, -1, 1,
        -1,  1, 0, -1, 1,
        -1,  1, 0, -1, 1,
    }, {});

this just resulted in a black screen, so I know that this is not the way to do it. I also tried changing TEX to POS, since it is supposed to be (x,y,0,x,y), but once again a black screen. I also tried TEX, TEX for fun but I figured it would produce a black screen, which my testing confirmed, lol. I'm sure it's simple, I'm just not sure how to create the array you are recommending.

Also, I modified the vert shader by distorting the uv position rather than the vert and it looks promising as I'm not getting random triangles. Instead, it's just blurred at the edges, which I assume will be resolved once I correctly setup the vertices and map them. Any pointers for how to correctly setup the grid of vertices and then map it?

You mention writing into the unused z coordinate, but how would I do that as uv is a 2D vector and wouldn't have a z coordinate (or is this what you're referring to)?

As for the distortion code going into the mesh uvs, would that involve modifying the mesh.cpp and mesh.hpp files in the gl folder? I just want to see if I am on the right track here, I would like to try and get the shaders to work first before trying something else.

Also, I really appreciate all of your help so far, you are a godsend!!!

[Seneral]

Ok, so what you want to create is the following mesh: Except it will look squashed horizontally to be a perfect square to exactly fit the rendering area in openGL.

I was a bit wrong with what I said though, notice how the two triangles that cover the entire screen use six vertices instead of 4? Some were duplicate since GL expects a full triangle, so 2x3 vertices. That would get super complicated super quick with a full grid, so instead you enter all vertices normally and then fill the third array parameter to mesh.cpp with a list of indices (triangles x 3) to indicate which vertices build which triangles (order within a triangle determines where it us facing!). Anyway, what you did is the same point six times, in the corner, so no triangle was drawn, resulting in a black screen. Here's an idea of what a correct rectangle eould look like: https://gamedev.stackexchange.com/questions/78014/how-to-create-a-regular-grid-of-triangles-correctly

Now you just gotta make sure that hr vertices get generated at the correct position (from -1 to 1 in each axis), on the correct axis (x,y) and not (x,z), etc, and with the correct amount (you want more vertices horizontally because of the screen size, but again, still from -1 to 1), so it will look distorted vertically

If you need more technical info, here's some reading: https://solarianprogrammer.com/2013/05/13/opengl-101-drawing-primitives/

[peyton-howe]

I have made a preliminary grid where I am now drawing 8 triangles (2 per each quadrant), instead of just two.

    // Create screen-space quad for rendering
    SSQuad = new Mesh ({ POS, TEX }, {
        //Quadrant 1, two triangles
        -1,  1, 0, 0, 1,
         0,  1, 0, 0.5, 1,
        -1,  0, 0, 0, 0.5,
         0,  1, 0, 0.5, 1,
         0,  0, 0, 0.5, 0.5,
        -1,  0, 0, 0, 0.5,

        //Quadrant 2, two triangles
         0,  1, 0, 0.5, 1,
         1,  1, 0, 1, 1,
         0,  0, 0, 0.5, 0.5,
         1,  1, 0, 1, 1,
         1,  0, 0, 1, 0.5,
         0,  0, 0, 0.5, 0.5,

        //Quadrant 3, two triangles
        -1,  0, 0, 0, 0.5,
         0,  0, 0, 0.5, 0.5,
        -1,  -1, 0, 0, 0,
         0,  0, 0, 0.5, 0.5, 
         0,  -1, 0, 0.5, 0,
        -1,  -1, 0, 0, 0,

        //Quadrant 4, two triangles
         0,  0, 0, 0.5, 0.5,
         1,  0, 0, 1, 0.5,
         0,  -1, 0, 0.5, 0,
         1,  0, 0, 1, 0.5,
         1,  -1, 0, 1, 0,
         0,  -1, 0, 0.5, 0,
    }, {});

Would I just need to add more triangles to perform the desired distortion in the vert shader, or I am still not setting up the grid correctly? Is there a way to automate this so I don't have to manually add in triangles, i.e. can I run a loop inside of the new Mesh function to generate the triangles?

[Seneral]

Yes. Please automate it, I sent you a link that shows how. You'll just make some kind of error this way, since the order actually matters. Could be some triangles already will look black to you. The theory is correct though. Just look at the one link and then you just send the std::vector with the data into the second mesh parameter, and the std::vector or whatever into the second. And yes, you need more triangles. The denser the mesh, the better - to a degree. Why? Because that determines how interpolated the distortions will be, since the pixels within a triangle will not use actual distortion values, but approximations interpolated from the distortions calculated at the vertices.

[peyton-howe]

Okay, I'm glad I'm on the right track here, phew. So once I get a dense enough mesh, will I go into the vert shader and apply the distortion to the uv vector and then map it to [0,1]x[0,1] for the fragment shader? I'm not sure how to do this yet, just making sure I'm on the right track.

[Seneral]

yep, in fact, you could do the distortion on the CPU and put it into the last two components (the UV) right when creating the mesh would probably be even better and save you a couple ms at runtime, or maybe it's not noticeable and all you loose is some flexibility (since it's harder to update the mesh than to update a shader parameter)

[peyton-howe]

I have been attempting to port the code over to work within the program, (I think was initially written in C#). This is what I have gotten so far, I am just not sure what to do with the commented out line.

static void CreateVertexBuffer()
{
    std::vector<std::vector<std::vector<double>>> vertices[mapSize*mapSize];

    for (int x = 0; x < mapSize; x++)
    {
        for (int z = 0; z < mapSize; z++)
        {
            //vertices[x*mapSize + z] = std::vector<std::vector<std::vector<float>>> vertices(x/10.0f, 1.0f, z/1.0f);
        }
    }
    short int indices [mapSize * mapSize * 6];
    int index = 0;

    for (int x = 0; x < mapSize; x++)
    {
        for (int z = 0; z < mapSize-1; z++)
        {
            int offset = x * mapSize + z;
            indices[index] = (short)(offset+0);
            indices[index + 1] = (short)(offset+1);
            indices[index + 2] = (short)(offset + mapSize);
            indices[index + 3] = (short)(offset+1);
            indices[index + 4] = (short)(offset + mapSize + 1);
            indices[index + 5] = (short)(offset + mapSize);
            index += 6;
        }
    }

}

My thought was then to call this function in the SSQuad = new Mesh function, like this: SSQuad = new Mesh ({POS, TEX}, {CreateVertexBuffer()}, {});

When trying this, I get the following error code

no matching function for call to ‘Mesh::Mesh(<brace-enclosed initializer list>, <brace-enclosed initializer list>, <brace-enclosed initializer list>)’
  SSQuad = new Mesh ({POS, TEX}, {CreateVertexBuffer()}, {});

I am thinking I might need to call CreateVertexBuffer() first and then create the SSQuad as folows: SSQuad = new Mesh ({POS, TEX}, {vertcies, indices}, {});

Am I on the right track here? Also, do you know how to get the commented out line to work correctly? I'm not super fluent in the C++ syntax, so debugging has been difficult for me (most of my experience has been with python up to this point).

[Seneral]

Sorry, I guess it's better if you start learning some basic C++ syntax around vectors: https://www.codeguru.com/cplusplus/c-tutorial-a-beginners-guide-to-stdvector-part-1/ Also take a look at mesh.hpp to see what the function is expecting. Hint: elements parameter is the indices parameter, that's just a different name for the same concept. Also no nested vectors needed, just the two I named in my last comment. Scratch that, turns out the angle brackets didn't render. Here it is again, with the correct sizes: std::vector<float> vertices(mapSize*mapSize*5); std::vector<unsigned int> indices((mapSize-1)*(mapSize-1)*6);

[peyton-howe]

Still working through auto-generating the triangles, but I a bit confused on the elements (indices) parameter. With the default code in the repository, the created mesh has a blank elements parameter. The vertices parameter is populated with the POS and TEX elements, which I think correlate to the vPos and vTex vectors in the shaders, which then creates the uv vector from the vTex vector (please correct me if I'm wrong). With this being the case, do I need to populate the indices parameter when I generate the mesh, or do I just need to populate the vertices parameter with the triangle coordinates?

I'm asking because when I manually calculate what the code generates, I get the following for the indices vector for a 7x7 grid:

indices[0] = 0
indices[1] = 1
indices[2] = 7
indices[3] = 1
indices[4] = 8
indices[5] = 7 

I think this is an issue since the values range from [-1,1] to [-1,1] for x and y for the pos vector and [0,1], [0,1] for the tex vector. However, if the generated indices are going to go into the elements parameter rather than the vertices parameter, then this may not be an issue.

[Seneral]

yep, as I said, elements = indices you are just indexing into the vertex array, which is just plain POS, TEX You do not NEED to populate the elements/indices parameter, but then it will only render the triangles you put into vertex array. Consider a single quad with 4 vertices. If you just use the vertex array, you have to submit 6 vertices, three for both triangles. The diagonal vertices will be twice in that array. Duplicate data = badm so alternatively, you can put only the 4 vertices in the array, and then create 6 indices that point to the 4 vertices, and the diagonals just point to the same vertex. If you need further explanation, please just use google ("opengl index buffer"). If you consider that, the example index array you showed should start to make sense (yes it is correct, as far as a quick glance tells).

[peyton-howe]

I read up on how to use the index buffer and am almost done implementing the automatic vertices generation correctly into the program. Unfortunately, I have run into an issue when I try to use the elements parameter for the mesh included in the source code.

Here is what I have done to create the mesh with just two triangles:

    SSQuad = new Mesh ({ POS, TEX }, {
                 //Triangle 1
        -1, -1, 0, 0, 0, //0
        -1,  1, 0, 0, 1, //1
         1,  1, 0, 1, 1, //2

                 //Triangle 2
         //1,  1, 0, 1, 1,  //2
         1, -1, 0, 1, 0,  //3
        //-1, -1, 0, 0, 0,  //0
    }, {
        0,1,2,   //draw triangle 1
        2,3,0,   //draw triangle 2
        });

When I run the ./GLCV -c YUV -w 1920 -h 1080 -f 30, this is what I get in the console:

Mesh (type 5, FpV 5): 4 vertices (20 floats) / 6 elements
Initializing Camera GL!
Starting Camera GL!
CamGL: Started GCS!
CamGL: GL error: line 335 error 0x0500
CamGL: Stopping GCS!
CamGL: CamGL encountered an error!
CamGL: GL error: line 268 error 0x0500
Camera GL was interrupted with code 1!

I know that the error 0x0500 code is due to passing an invalid GLenum, but I'm not sure where I should be looking to fix this.

[peyton-howe]

Update: I have figured out what the issue is and I am pretty sure I know how to solve it, but I am unsure of how to add the necessary dependency to the make file.

The issue arises from https://github.com/Seneral/VC4CV/blob/master/gl/mesh.cpp#L72

since OpenGL ES only accepts GL_UNSIGNED_BYTE or GL_UNSIGNED_SHORT as the data types for glDrawElements by default. Changing to one of these data types fixes the error, but I get a black screen no-matter what combination of indices I provide, which is likely because all of the code is written for the GL_UNSIGNED_INT data type. I have a feeling that rewriting everything to be an unsigned short or an unsigned byte could fix this, but there is another way.

According to https://stackoverflow.com/questions/4413909/how-do-you-use-gldrawelements-with-gl-unsigned-int-for-the-indices, including the glext.h file should fix the issue and allow for GL_UNSIGNED_INT to be passed into the glDrawElements function and should resolve the black screen issue. I know I need to add this include into the mesh.h file and include the dependency inside of the make file, but I don't know how to modify the make/cmake file since there are a lot of files, most of which are auto-generated. Is there an easy way to add in the glext.h file??

[peyton-howe]

Update #2: I changed all instances of unsigned int to unsigned short in the mesh.cpp and mesh.hpp files and used the GL_UNSIGNED_SHORT for the glDrawElements in line 72 of the mesh file, and the triangles draw correctly. For now, I am going to run with this and implement the auto-generation of triangles!

Would there be any reason to switch back to the unsigned_int data type later on?

[Seneral]

Nice, got it fixed before I could look into it No, I just didn't know about this limitation since I developed this on desktop first and seemingly never tested elements on the pi. Glad to hear it's working now.

[peyton-howe]

I have successfully set up a custom size grid within the code and have also figured out the shader for barrel distortion! At 1080p, I am able to get 30fps, so I'm very optimistic about using the shaders to perform the distortion and not having to try and code the QPU. Now, I am attempting to add in stereo vision by setting up both cameras in their own preview window of 960x1080. However, I am running into issues in just implementing the ability to change which camera is being accessed.

I tried adding in this line to the gcs.c file within the gas_create function, but it appears that doing so has no effect on the output at all. I don't get any errors when compiling, it just doesn't do anything. I found the source for this code from the raspivid source file, but I may just be missing something simple here with the syntax.

MMAL_PARAMETER_INT32_T camera_num = {{MMAL_PARAMETER_CAMERA_NUM, sizeof(camera_num)}, 1};

I also tried setting it up similar to the disableEXP function, using:

MMAL_PARAMETER_CAMERA_NUM camera_num;
camera_num.hdr.id = MMAL_PARAMETER_CAMERA_NUM;
camera_num.hdr.size = sizeof(MMAL_PARAMETER_CAMERA_NUM);
camera_num.value = 1;
meal_port_parameter_set(gcs->camera->control, &camera_num.hdr);

Doing that gave an error that camera_num was undeclared even though I tried adding uint16_t camera_num; to the gcs.h file, .camera_num = 1 to the main_gl.cpp file, uint16_t camera_num; to the camGL.h file, and gcs.Params.camera_num = params->camera_num; to the camGL.c file.

The 1 should select camera 1, while a zero should select camera 0. I am not sure what I am doing wrong here or why there is no effect in the camera selection with the method that compiled without any errors. Any pointers?

[Seneral]

Never worked with two cameras, so no idea But I assume you excluded errors in getting the parameters where they should be by printing the value out? If that is correct, then there should be plenty other people using two cameras with MMAL

[peyton-howe]

I noticed a bit off an error in my previous post: I was getting the camera_num undeclared error before I added gcs.Params.camera_num = params->camera_num; to the camGL.c file, so I think I have correctly established _cameranum as a variable within the program. I also noticed that I set it as a uint16_t, when it should be a uint32_t, which I have changed in the gcs.h and camGL.h files. I will attempt later tonight to input _cameranum as a parameter when running /GLCV to see if I am correct, but if I am missing any lines of code in establishing _cameranum as a parameter, please let me know.

I will ask around on a few forums to see if anyone is able to help out on working with two cameras, although I have a hunch as to what the issue is. When looking through the source code for the raspivid, they include the _mmal_parameterscamera.h file in their code. I noticed that its referenced in https://github.com/Seneral/VC4CV/blob/master/camera/gcs.c#L100 in the gcs.c file, but is not actually listed in the #include statements at the top of the program, which might be why the lines I added don't actually do anything within the program.

If I need to include this file, how can I change the makefile so that its included when I do sudo make gl?

[Seneral]

That's not how C/C++ works, it would spit an error if it was missing. No need to include it since it's already included through other include files. It only provides the parameter enums.

As to your first problem: If in doubt, print it. Please do some basic debugging and you'll be able to tell whether the problem lies with how you modify VC4CV or with how you use MMAL.

[peyton-howe]

It was a MMAL issue, which I have since fixed and I now have both cameras being displayed and barrel distortion being applied to both! It's running at 1080p and about 26fps, which I am going to attempt to increase to 30fps later on. Right now, I am trying to implement text annotation, but I am running into an issue with a newly defined text annotation function. I created a _gcsannotation function, which sets up the annotation parameters, and a _camGL_updateannotation function, which calls the _gcsannotation function and updates the text with a new string. The _camGL_updateannotation function is then called within the main loop of the mian_gl program where I intended to feed it text to display.

Unfortunately, I am getting a implicit declaration of function gcs_annotate error, which is very confusing to me since the functions are included in the header files and should be defined before the main function. At one point, I copied and pasted all the existing functions in gcs and camGL in my attempts to get both cameras working and I never got an implicit declaration error. Am I missing something simple here?

[Seneral]

gcs_annotate/gcs_annotation perhaps?

As for optimisation, may be able to squeeze out a bit more by putting the distortion on the CPU part E.g. don't send the raw -1 to 1 UVs but the already distorted ones. Other than that, dual 1080p seems weird anyway, in the sense that it's 16:9 and thus way too wide for stereoscopic viewing normally How about cutting off the sides to lower pixel count, or reduce the horizontal pixelcount by stretching, or using a different camera mode alltogether? Make sure to look into the docs to figure out from the beginning which camera mode suits you best, e.g. 1080@30 is probably not your preferred mode due to limited FoV and aspect ratio

[peyton-howe]

Turns out I just edited the wrong gcs.h file , lol. I created a few copies of the main folder when testing how to get both cameras, and just forgot to edit the correct file.

As for possible performance boosts, I wanted to clear up what I have going on. Each camera is running at a resolution of 960x1080 in side-by-side mode, which gives a 1920x1080p resolution overall. If I disable distortion and use the original mesh, I can get 27fps max, so distortion does not seem to be dropping performance too much. However, 1 extra fps is still a pretty significant boost, so I would like to see if doing the distortion on the CPU would speed things up.

For putting the distortion on the cpu, would I just need to put the distortion code from the shader here, in between generating the mesh and calling the SSQuad function?

I'm not sure if I optimized openGL the best for the two cameras, but this is what I did to get the two working and displaying. When I tried nesting a while loop for each camera, the one on the top loop would display just fine, but the other camera (the one in the nested loop) would be frozen until I pressed "q" and then would run at like 10fps, so I dropped that Idea altogether. I also attempted to remove the if statements for the shader colors since I only plan on using YUV, but it did not appear to improve performance at all. I am open to any suggestions for speeding this up!

On a final note, now that I have text overlay working, I would like to read data from serial and display it to the camera feed. I am assuming I will need to include an additional library for the serial port (please correct me if I'm wrong) when compiling the main_gl program. Is there an easy way to modify the make file, or should I write my own make files for each of the GLCV program files?

[Seneral]

No need for a library, serial stuff is built into linux (as a device file, I think /dev/serial0(1)) and can be read by C++ directly You should be able to find a ton of tutorials out there

Are your cameras synced up? If not, you might be waiting a ton of time just to get the second camera image But I assume they are, else you'd see inconsistent frametimes, not a solid 27fps

Other than that, there might be pipelining issues that I am not aware of. In theory eglSwapBuffers should do the flush, so they both should be in the pipeline for the QPU to execute at the same time, but you never know

One thing you might want to try is changing the bit depth of the render target Right now it's set up to be 16bit total, 5,6,5, you might actually get better results with 8,8,8 (eglUtil.c) I have no clue anymore if there was a reason to not do that though, or if the potential savings in packing make up for the higher data bandwidth, or what your processing afterwards prefers (I assume h264 encoding?)

Finally, you can try to overclock your QPU, should work fine to 400mHz even (250-300 default)

[peyton-howe]

I checked if the camera were synced up, and it turns out they are (honestly better for my purposes since I don't have to jump through any loops to sync the two). I did notice that the frame rate drops once two instances of the camGL function are run, but it's not because of waiting for frames from another camera since the two are in sync, probably just due to the amount of data being passed in.

I also tried changed the render target to 8,8,8 and moved the distortion to the CPU, and both provided minor performance boosts. They mainly improved the stability of the frame rate once the maximum frame rate of the program was reached. Basically, if the program maxed out at 27fps, changing the render target and moving the distortion to the CPU produced a frame rate much closer to 27fps and the frame rate would fluctuate less than the default settings.

However, all was not lost. Overclocking the gpu to 400mHz bumped performance up to 32-33 fps without any issues! I think I am just going to stick with that and feed data over serial with a pi pico/arduino since the code appears to be as optimized as possible and is just limited by hardware at this point.

[Seneral]

Forgot to close this. Glad to hear it works well now, if you have something to share regarding your project, please do, A stereoscopic camera / VR camera sounds interesting.

[peyton-howe]

Hey Seneral, here is a link to an article I wrote on the project!

https://medium.com/@peytonhowe1206/the-journey-to-native-vr-on-a-raspberry-pi-b303cb905941