Create Actor from pixel data instead of image file

[jwortmann]

Hello and thanks for creating this package!

I would like to create Actors from raw pixel data which I dynamically generate, instead of passing an image file that is stored on disk.

What I've tried so far:

using GameZero
using PNGFiles
using SimpleDirectMediaLayer
using SimpleDirectMediaLayer.LibSDL2

function MyActor(image; kv...)
    height, width = size(image)
    raw_pixeldata = UInt8[]
    for pixel in 255 * vec(image')
        push!(raw_pixeldata, red(pixel))
        push!(raw_pixeldata, green(pixel))
        push!(raw_pixeldata, blue(pixel))
        push!(raw_pixeldata, alpha(pixel))
    end
    # sf = SimpleDirectMediaLayer.LibSDL2.SDL_CreateRGBSurface(0, width, height, 32, 0, 0, 0, 0)
    # sf.pixels = Ref{Vector{UInt8}}(raw_pixeldata)  # doesn't work:   type Ptr has no field pixels
    sf = SimpleDirectMediaLayer.LibSDL2.SDL_CreateRGBSurfaceWithFormatFrom(raw_pixeldata, width, height, 8, 4, SimpleDirectMediaLayer.LibSDL2.SDL_PIXELFORMAT_RGBA8888)
    w, h = size(sf)
    a = GameZero.Actor("", sf, Rect(0, 0, Int(w), Int(h)), [1.0, 1.0], 0, 255, Dict{Symbol,Any}())
    for (k, v) in kv
        setproperty!(a, k, v)
    end
    return a
end

image = PNGFiles.load("image.png")  # this is just a simple example, in practice I want to generate it dynamically in the code
actor = MyActor(image)  # doesn't work

I tried to use functions like SDL_CreateRGBSurface or SDL_CreateRGBSurfaceWithFormatFrom, but unfortunately I can't get it to work. The problem is that I don't know how to pass the pixel data to these functions. The SDL C API seems to expect a pointer void *pixels, but I'm not very familiar with C and I have no clue how to do that in Julia. I've also tried various combinations like Ref{Vector{UInt8}}(raw_pixeldata), Ref{Cvoid}(raw_pixeldata), etc., but nothing of it worked (and basically I don't really know what I'm doing here).

The Julia wrapper for the SDL function looks like

function SDL_CreateRGBSurfaceWithFormatFrom(pixels, width, height, depth, pitch, format)
    ccall((:SDL_CreateRGBSurfaceWithFormatFrom, libsdl2), Ptr{SDL_Surface}, (Ptr{Cvoid}, Cint, Cint, Cint, Cint, Uint32), pixels, width, height, depth, pitch, format)
end

but this doesn't help me much either. https://github.com/JuliaMultimedia/SimpleDirectMediaLayer.jl/blob/82c79b94289b65d2ad15f89363a632041c0d7c06/src/LibSDL2.jl#L1773-L1787

The code above crashes with a long error message

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x1feebdc5840 -- unsafe_load at .\pointer.jl:119 [inlined]
unsafe_load at .\pointer.jl:119 [inlined]
...

Does anybody know how to use one of the linked SDL functions and could help me; or would you consider to create a constructor for Actor which allows to pass the image data directly (e.g. a Matrix{RGBA} from the Images.jl/ImageIO.jl packages)?

[jwortmann]

I've gotten it to work!

There were a few pitfalls that I had to figure out:

• the SDL function expects the pixel data as an array of UInt32 numbers, so the 8-bit color values for each channel first need to be combined into a single UInt32 for each pixel
• (I think that) the pixel array which is passed as an argument needs to be stored in a permanent variable, so that the garbage collector won't free the corresponding memory which is referenced from the SDL function
• depth is the combined bit-depth of all channels, i.e. here it should be set to 32
• excerpt from the SDL3 docs at https://wiki.libsdl.org/SDL3/SDL_CreateSurfaceFrom#remarks

  Pitch is the offset in bytes from one row of pixels to the next, e.g. width*4 for SDL_PIXELFORMAT_RGBA8888.

So with this my code example looks as follows:

using SimpleDirectMediaLayer

images = Vector{UInt32}[]

function MyImageActor(image; kv...)
    height, width = size(image)
    depth = 32  # 8 bit per channel
    pitch = 4 * width
    format = SimpleDirectMediaLayer.LibSDL2.SDL_PIXELFORMAT_RGBA8888

    pixels = UInt32[]
    sizehint!(pixels, height * width)

    for pixel in 255 * image'
        push!(pixels, UInt32(red(pixel)) << 24 | UInt32(green(pixel)) << 16 | UInt32(blue(pixel)) << 8 | UInt32(alpha(pixel)))
    end

    push!(images, pixels)  # prevent memory of the pixel array being freed by garbage collector
    sf = SimpleDirectMediaLayer.LibSDL2.SDL_CreateRGBSurfaceWithFormatFrom(pixels, width, height, depth, pitch, format)
    sf == C_NULL && error("surface is NULL")
    a = GameZero.Actor("", sf, Rect(0, 0, width, height), [1.0, 1.0], 0, 255, Dict{Symbol,Any}())
    for (k, v) in kv
        setproperty!(a, k, v)
    end
    return a
end

Probably there is potential for improvements in the code, but it seems to work so far. I still think it would be useful to have something like this built-in and managed directly by GameZero, so perhaps this issue can stay open as a feature request.