Simple transfer of Array to OpenGL texture via Image class?

[rennis250]

I know that the issue of OpenGL interoperability has been dealt with before and Forge handles a lot of the grunt work already. Is it possible that users could have public access to the texture ID for a given Image class, so that they don't have to reproduce all of that hard work if they want to run an image through a custom shader? I have a very specific type of shader that I want to run for my work, but I'm not knowledgable enough about C++ and OpenCL/CUDA to do to much on my own (I had a few unsuccessful attempts already by following the code from https://github.com/bkloppenborg/arrayfire-interop-examples); I mainly work in Rust.

Best, Rob

[9prady9]

I think whats being done in arrayfire-interop-examples is essentially what Forge tries to handle, shown below for reference.

Forge replaces GPU->Host->OpenGL(GPU Texture) memory transfers with GPU->OpenGL(GPU Texture).

I did have an idea about letting users run some custom shaders on af::array(matching a given criteria) eventually, but haven't been able to get to it. Unfortunately, We don't have it at the top of the priority-list at the moment and it may be a while before there is any such support made available via ArrayFire.

However, I believe you may be able to run custom shaders on your image(af::array) if you do the following - I have tried this though, although that was one of intentions behind the design of Forge.

• Use ArrayFire without graphics. Graphics can be disabled using AF_DISABLE_GRAPHICS=1environment variable or by installing a no-gl installer from our achrive.
• Use Forge as another external library in your project to do graphics and utilize the pixels member function to get the OpenGL texture buffer ID.

In such use case, you would have to take care of the following on your own.

• Ensure forge::Image instance created fits the data dimensions of af::array.
• Ensure data layout inside af::array is in according to what forge::Image(OpenGL texture) expects.
• Handle memory transfers between af::array(CUDA/OpenCL) objects and forge::Image(OpenGL buffer) yourself. To help with this stage, we provide a helper header along with forge, but you are free to work on your own transfer logic.

[rennis250]

Ok, thanks. I decided to try to change things myself. So far, I've edited the Forge code to add an additional function, called texture(), that acts like the pixels() method for the Image class. It returns the assigned OpenGL texture ID and seems to work as expected. Now, I am changing the Arrayfire code to export a function for converting an F32 Array to a Forge::Image and return a pointer to it (via convert_and_copy_image). I am only dealing with F32 arrays for the moment, so that is fine for now. Last, I'm exporting this code via extern "C" and trying to access it via my Rust code. I'm almost done with the last two steps, I hope, so I will let you know how it goes.

[rennis250]

So, I think I'm almost there, but a bit stuck. I tried exporting convert_and_copy_image by putting the following in include/image.h of Arrayfire:

#include <af/graphics.h>

#if defined(WITH_GRAPHICS)
#include <common/graphics_common.hpp>

using namespace graphics;
#endif

...

namespace af {
class array

#if defined(WITH_GRAPHICS)
template <typename T>
static forge::Image *convert_and_copy_image(const af_array in);
#endif

...

The library compiles fine and the include files in /usr/local/include are naturally updated to reflect this change, but when I try to use this function in my code, I get the error: error: no member named 'convert_and_copy_image' in namespace 'af'

I'm just trying to simply do the following as a test:

#include <arrayfire.h>
#include <forge.h>

...

af:array test = af::constant(0.5, 512, 512, 3);
forge::Image* img = NULL;
img = af::convert_and_copy_image<float>(test);

I thought that this is all that would be necessary to get access to the function, but apparently not. Any tips?

Best and thanks, Rob

[9prady9]

I can't really say for sure based on above code previews, I will need total code to see what exactly going.

But it looks like you defined and didn't implement the function convert_and_copy_image. Also, remove the static qualifier for it. You would have instantiate the template function where it is implemented if you didn't do that already.

[rennis250]

That's all I changed and all I did, so there is no more code to preview... convert_and_copy_image is a function that is already implemented in src/api/c/image.cpp by the Arrayfire team.

[rennis250]

I spoke with @umar456 on slack and the problem mostly stems from my lack of experience with C++. It might be working soon.

[rennis250]

Alright, I now have the following. I changed image.h so that the static keyword is removed from the function signature and I put an AFAPI declaration at the front just to be safe (I'm on *nix). In image.cpp, I wrapped the convert_and_copy_image function implementation into the af namespace and removed its static identifier. All mentions of convert_and_copy_image in image.cpp were updated to account for the namespace change. This is the only file that had calls to convert_and_copy_image. The library compiles and installs without problems.

My code remains the same: I just include the ArrayFire headers and link the library at compile time, but I still receive the same error: error: no member named 'convert_and_copy_image' in namespace 'af'. Sorry about this, since it seems like such a noob question, but from my experiences with C, this usually is a bit simpler. I'm confused what else would need to be done in C++ just to access a function from the library and I am having a bit of trouble searching for solutions.

For reference, here is a diff:

First, image.h:

@@ -10,12 +10,23 @@
 #pragma once
 #include <af/defines.h>
 #include <af/features.h>
+#include <af/graphics.h>
+
+#if defined(WITH_GRAPHICS)
+#include <common/graphics_common.hpp>
+
+using namespace graphics;
+#endif

 #ifdef __cplusplus
-namespace af
-{
+namespace af {
 class array;

+#if defined(WITH_GRAPHICS)
+template <typename T>
+AFAPI forge::Image *convert_and_copy_image(const af_array in);
+#endif
+

and second, image.cpp:

diff --git 1/image.cpp 2/image2.cpp
index b8996f94..fe663cfe 100644
--- 1/image.cpp
+++ 2/image2.cpp
@@ -52,8 +52,9 @@ Array<float> normalizePerType<float>(const Array<float>& in)
     return in;
 }

+namespace af {
 template<typename T>
-static forge::Image* convert_and_copy_image(const af_array in)
+forge::Image* convert_and_copy_image(const af_array in)
 {
     const Array<T> _in  = getArray<T>(in);
     dim4 inDims = _in.dims();
@@ -72,6 +73,7 @@ static forge::Image* convert_and_copy_image(const af_array in)

     return ret_val;
 }
+template forge::Image* convert_and_copy_image<float >(const af_array in);
+template forge::Image* convert_and_copy_image<char >(const af_array in);
+template forge::Image* convert_and_copy_image<int    >(const af_array in);
+template forge::Image* convert_and_copy_image<uint  >(const af_array in);
+template forge::Image* convert_and_copy_image<short >(const af_array in);
+template forge::Image* convert_and_copy_image<ushort>(const af_array in);
+template forge::Image* convert_and_copy_image<uchar >(const af_array in);
+}
 #endif

 af_err af_draw_image(const af_window wind, const af_array in, const af_cell* const props)
@@ -95,13 +97,13 @@ af_err af_draw_image(const af_window wind, const af_array in, const af_cell* con
         forge::Image* image = NULL;

         switch(type) {
-            case f32: image = convert_and_copy_image<float >(in); break;
-            case b8 : image = convert_and_copy_image<char  >(in); break;
-            case s32: image = convert_and_copy_image<int   >(in); break;
-            case u32: image = convert_and_copy_image<uint  >(in); break;
-            case s16: image = convert_and_copy_image<short >(in); break;
-            case u16: image = convert_and_copy_image<ushort>(in); break;
-            case u8 : image = convert_and_copy_image<uchar >(in); break;
+            case f32: image = af::convert_and_copy_image<float >(in); break;
+            case b8 : image = af::convert_and_copy_image<char  >(in); break;
+            case s32: image = af::convert_and_copy_image<int   >(in); break;
+            case u32: image = af::convert_and_copy_image<uint  >(in); break;
+            case s16: image = af::convert_and_copy_image<short >(in); break;
+            case u16: image = af::convert_and_copy_image<ushort>(in); break;
+            case u8 : image = af::convert_and_copy_image<uchar >(in); break;
             default:  TYPE_ERROR(1, type);
         }

[9prady9]

• It may be simple, but are you building Arrayfire with AF_WITH_GRAPHICS cmake variable turned on ?
• I don't see any diff for image.cpp. Did you instantiate the implementation of the static function for all the types in arrayfire - Just the type you want is enough if your target type is fixed.

[rennis250]

Yes, I have that variable enabled. I checked that other graphics functions work fine. I can open a window for instance via af::Window.

Sorry, I uploaded the diff... I hit "Submit Comment" a little too quickly. I'll search about the instantiation, but what does that mean exactly?

[rennis250]

I updated the diffs above to reflect our discussion on slack

[rennis250]

I made some progress. Turns out that it is actually related to the WITH_GRAPHICS #ifdef guards. Basically I have to put them in the relevant header file for compilation, since the function signature of convert_and_copy_image returns a Forge::Image, but once the library is installed, they prevent my separate program from seeing the function. So, I need to edit the installed header and remove those #ifdef guards around my added function signature to get it to work. This is necessary because #defining WITH_GRAPHICS in my code causes the compiler to search for other headers that aren't part of the default install. This is DEFINITELY not what one wants in the actual ArrayFire header files, but for the moment, it is fine for me to test out functionality and see how things work.

NOTE: updated to clarify how I actually deal with the #ifdef guards.

[rennis250]

Ok, I've gotten further, but I have hit a different wall that seems related to either OpenGL or the design of Forge. I'm listing all of the relevant code first. Questions and clarification follow below.

arrayfire/include/af/graphics.h

diff --git i/include/af/graphics.h w/include/af/graphics.h
index 1c44d0d3..e3fa5292 100644
--- i/include/af/graphics.h
+++ w/include/af/graphics.h
@@ -9,21 +9,27 @@

 #pragma once

-#include <af/defines.h>
 #include <af/array.h>
+#include <af/defines.h>
+
+#if defined(WITH_GRAPHICS)
+#include <common/graphics_common.hpp>
+#include <forge.h>
+
+// using namespace graphics;
+#endif

 typedef unsigned long long af_window;

 typedef struct {
-    int row;
-    int col;
-    const char* title;
-    af_colormap cmap;
+  int row;
+  int col;
+  const char *title;
+  af_colormap cmap;
 } af_cell;

#ifdef __cplusplus
+#if defined(WITH_GRAPHICS)
+AFAPI forge::Image *af_convert_and_copy_image(const af::array &in,
+                                              af::Window *window);
+#endif

namespace af
{

arrayfire/src/c/api/image.cpp

diff --git i/src/api/c/image.cpp w/src/api/c/image.cpp
index b8996f94..3fd05086 100644
--- i/src/api/c/image.cpp
+++ w/src/api/c/image.cpp
@@ -12,6 +12,7 @@
 #include <af/image.h>
 #include <af/index.h>
 #include <af/data.h>

 #include <common/ArrayInfo.hpp>
 #include <common/graphics_common.hpp>
@@ -53,7 +54,7 @@ Array<float> normalizePerType<float>(const Array<float>& in)
 }

 template<typename T>
-static forge::Image* convert_and_copy_image(const af_array in)
+forge::Image* convert_and_copy_image(const af_array in)
 {
     const Array<T> _in  = getArray<T>(in);
     dim4 inDims = _in.dims();
@@ -72,6 +73,47 @@ static forge::Image* convert_and_copy_image(const af_array in)

     return ret_val;
 }
+template forge::Image* convert_and_copy_image<float >(const af_array in);
+template forge::Image* convert_and_copy_image<char >(const af_array in);
+template forge::Image* convert_and_copy_image<int    >(const af_array in);
+template forge::Image* convert_and_copy_image<uint  >(const af_array in);
+template forge::Image* convert_and_copy_image<short >(const af_array in);
+template forge::Image* convert_and_copy_image<ushort>(const af_array in);
+template forge::Image* convert_and_copy_image<uchar >(const af_array in);
+
+forge::Image* af_convert_and_copy_image(const af::array &in, af::Window *wind) {
+    // try {
+        af_array _in = in.get();
+        const ArrayInfo& info = getInfo(_in);
+
+        af::dim4 in_dims = info.dims();
+        af_dtype type    = info.getType();
+        DIM_ASSERT(0, in_dims[2] == 1 || in_dims[2] == 3 || in_dims[2] == 4);
+        DIM_ASSERT(0, in_dims[3] == 1);
+
+        forge::Window* window = reinterpret_cast<forge::Window*>(wind->get());
+        makeContextCurrent(window);
+        forge::Image* image = NULL;
+
+        switch(type) {
+            case f32: image = convert_and_copy_image<float >(_in); break;
+            case b8 : image = convert_and_copy_image<char  >(_in); break;
+            case s32: image = convert_and_copy_image<int   >(_in); break;
+            case u32: image = convert_and_copy_image<uint  >(_in); break;
+            case s16: image = convert_and_copy_image<short >(_in); break;
+            case u16: image = convert_and_copy_image<ushort>(_in); break;
+            case u8 : image = convert_and_copy_image<uchar >(_in); break;
+            default:  TYPE_ERROR(1, type);
+        }
+
+        auto gridDims = ForgeManager::getInstance().getWindowGrid(window);
+        af_cell props{1, 1, "testing", AF_COLORMAP_DEFAULT};
+        window->setColorMap((forge::ColorMap)props.cmap);
+        // window->draw(*image); // if I uncomment this line, then an image is drawn in the window
+        return image;
+    // }
+    // CATCHALL;
+}

arrayfire/src/api/unified/graphics.cpp

diff --git i/src/api/unified/graphics.cpp w/src/api/unified/graphics.cpp
index 08f8d81d..8fc79de2 100644
--- i/src/api/unified/graphics.cpp
+++ w/src/api/unified/graphics.cpp
@@ -32,6 +32,14 @@ af_err af_set_size(const af_window wind, const unsigned w, const $
     return CALL(wind, w, h);
 }

+#if defined(WITH_GRAPHICS)
+forge::Image* af_convert_and_copy_image(const af::array &in, af::Window *window)
+{
+    CHECK_ARRAYS(in);
+    return CALL(in, window);
+}
+#endif
+
 af_err af_draw_image(const af_window wind, const af_array in, const af_cell* const $
 {
     CHECK_ARRAYS(in);

In Forge, I just made a simple method for the Image class that returns the texture ID. The code is an exact copy-and-paste of the pixels() code that returns the PBO, except I just have it return the texture ID instead. This works fine and since it is trivial copy-and-paste in many places, I won't paste that here to save space. Next is my test program. You will need access to the GLAD for OpenGL function loading (https://github.com/Dav1dde/glad) or replace it with some other library. The shaders are listed below the code.

main.cpp

#include <glad/glad.h>

// Standard Headers
#include <cstdio>
#include <cstdlib>
#include <cmath>

#include <arrayfire.h>
#include <forge.h>
#include <vector>
#include <iostream>
#include <fstream>
#include <iterator>
#include <algorithm>

using namespace std;

const unsigned DIMX = 512;
const unsigned DIMY = 512;
const unsigned IMG_SIZE = DIMX * DIMY * 4;

GLuint compileShader(const char * source, GLuint shaderType) {
    GLint  mStatus;
    GLint  mLength;

    GLuint shader = glCreateShader(shaderType);
    glShaderSource(shader, 1, & source, nullptr);
    glCompileShader(shader);
    glGetShaderiv(shader, GL_COMPILE_STATUS, & mStatus);

    // Display the Build Log on Error
    if (mStatus == false)
    {
        glGetShaderiv(shader, GL_INFO_LOG_LENGTH, & mLength);
        std::unique_ptr<char[]> buffer(new char[mLength]);
        glGetShaderInfoLog(shader, mLength, nullptr, buffer.get());
        fprintf(stderr, "%s", buffer.get());
    }

    return shader;
}

GLuint createShader(std::string const & vertfn, std::string const & fragfn) {
    GLint  mStatus;
    GLint  mLength;

    GLuint mProgram = glCreateProgram();

    // Load GLSL Shader Source from File
    std::ifstream fd(vertfn);
    auto src = std::string(std::istreambuf_iterator<char>(fd),
                          (std::istreambuf_iterator<char>()));

    // Create a Shader Object
    const char * source = src.c_str();
    GLuint vert_shader = compileShader(source, GL_VERTEX_SHADER);

    std::ifstream fd2(fragfn);
    src = std::string(std::istreambuf_iterator<char>(fd2),
                          (std::istreambuf_iterator<char>()));

    // Create a Shader Object
    source = src.c_str();
    GLuint frag_shader = compileShader(source, GL_FRAGMENT_SHADER);

    // Attach the Shader and Free Allocated Memory
    glAttachShader(mProgram, vert_shader);
    glAttachShader(mProgram, frag_shader);

    glLinkProgram(mProgram);
    glGetProgramiv(mProgram, GL_LINK_STATUS, & mStatus);
    if(mStatus == false)
    {
        glGetProgramiv(mProgram, GL_INFO_LOG_LENGTH, & mLength);
        std::unique_ptr<char[]> buffer(new char[mLength]);
        glGetProgramInfoLog(mProgram, mLength, nullptr, buffer.get());
        fprintf(stderr, "%s", buffer.get());
    }
    assert(mStatus == true);

    glDeleteShader(vert_shader);
    glDeleteShader(frag_shader);

    return mProgram;
}

int main(int argc, char * argv[]) {
    try {
        af::setDevice(0);
        af::info();

        af::array img_rgb = af::loadImage("test.jpg", true);
        af::array c = af::constant(255, img_rgb.dims());
        af::array img_norm = img_rgb / c;

        af::Window wnd(DIMX, DIMY, "Fractal Demo");
        wnd.show();

        gladLoadGL();
        fprintf(stderr, "OpenGL %s\n", glGetString(GL_VERSION));

        forge::Image* img = af_convert_and_copy_image(img_norm, &wnd);

        // Create Vertex Array Object
        GLuint vao;
        glGenVertexArrays(1, &vao);
        glBindVertexArray(vao);

        // Create a Vertex Buffer Object and copy the vertex data to it
        GLuint vbo;
        glGenBuffers(1, &vbo);

        GLfloat vertices[] = {
            -1.0f,  1.0f, 1.0f, 0.0f, 0.0f, // Top-left
             1.0f,  1.0f, 0.0f, 1.0f, 0.0f, // Top-right
             1.0f, -1.0f, 0.0f, 0.0f, 1.0f, // Bottom-right
            -1.0f, -1.0f, 1.0f, 1.0f, 1.0f  // Bottom-left
        };

        glBindBuffer(GL_ARRAY_BUFFER, vbo);
        glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW);

        // Create an element array
        GLuint ebo;
        glGenBuffers(1, &ebo);

        GLuint elements[] = {
            0, 1, 2,
            2, 3, 0
        };

        glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, ebo);
        glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(elements), elements, GL_STATIC_DRAW);

        GLuint shaderProgram = createShader("shader.vert", "shader.frag");

        // Specify the layout of the vertex data
        GLint posAttrib = glGetAttribLocation(shaderProgram, "position");
        glEnableVertexAttribArray(posAttrib);
        glVertexAttribPointer(posAttrib, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), 0);

        GLint colAttrib = glGetAttribLocation(shaderProgram, "color");
        glEnableVertexAttribArray(colAttrib);
        glVertexAttribPointer(colAttrib, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(GLfloat), (void*)(2 * sizeof(GLfloat)));

        glUseProgram(shaderProgram);

        uint mPBO = img->pixels();
        uint mTex = img->texture();
        uint mTexIndex = glGetUniformLocation(shaderProgram, "tex");

       // following code copied exactly from forge/src/backend/opengl/image_impl.cpp.
       // it works perfectly there, so why not here. i can't see why...
        glEnable(GL_BLEND);
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

        // load texture from PBO
        glActiveTexture(GL_TEXTURE0);
        glUniform1i(mTexIndex, 0);
        glBindTexture(GL_TEXTURE_2D, mTex);
        // bind PBO to load data into texture
        glBindBuffer(GL_PIXEL_UNPACK_BUFFER, mPBO);
        glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
        glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, DIMX, DIMY, GL_RGBA, GL_FLOAT, 0);
        glPixelStorei(GL_UNPACK_ALIGNMENT, 4);

        // Clear the screen to black
        glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT);

        do {
            glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0); // doesn't work
            wnd.show();
        } while(!wnd.close());

        glDeleteProgram(shaderProgram);

        glDeleteBuffers(1, &ebo);
        glDeleteBuffers(1, &vbo);

        glDeleteVertexArrays(1, &vao);
    }catch (forge::Error err) {
        std::cout << err.what() << "(" << err.err() << ")" << std::endl;
    }

    return EXIT_SUCCESS;
}

shader.vert

#version 330

in vec2 position;
in vec3 color;

out vec4 vColor;
out vec2 Position;

uniform mat4 MVP;

void main() {
    vColor = vec4(color, 1.0);
    Position = position;
    gl_Position = vec4(position, 0.0, 1.0);
}

shader.frag

#version 330

in vec4 vColor;
in vec2 Position;

out vec4 fragColor;

uniform sampler2D tex;

void main() {
    vec2 uv = (1.0 - Position.xy)/2.0;
    fragColor = texture(tex, uv);
}

First, please see my updated post just above this for how to compile and install ArrayFire to then successfully compile the example program here.

Anyway, my confusion lies mainly in two places. First, my code, although it is mainly code copy-and-pasted from the Forge and ArrayFire code bases, results in a black screen. If I print out the texture and PBO IDs and other information for the image, I can see that everything was intialized correctly for the image. However, even though the code for synchronizing the PBO and texture are copied exactly from the Forge implementation in forge/src/backend/opengl/image_impl.cpp, it doesn't work. If instead, within the ArrayFire code that I added in arrayfire/src/c/api/image.cpp, I uncomment the last line of af_convert_and_copy_image which is a call to forge::Window::draw (see diff above), then the image appears on the screen. From my perspective, in every case, my code in my program and the code in forge::Window::draw are doing the exact same thing in the same exact graphics context, so can anyone else see the problem?

[rennis250]

I have another idea. I will try it and report back.

[rennis250]

I got it to work! :D

I instead changed the implementation of the Image class to allow it to accept arbitrary shaders and changed image_impl::render in forge/src/backend/opengl/image_impl.cpp to check for the presence of a custom shader and to run slightly different and simpler code in that case when drawing the image via the PBO. I then made a second, simpler draw_image function in arrayfire/src/api/c/image.cpp which accounts for these changes and is the user-facing function for drawing images with custom shaders. The implementation is a bit messy, so I won't open a PR just yet, but I will play around with this. Pretty happy :)