Best method for transferring large image data from python to godot

[m5tuff]

Hi, thanks for your work on this plugin, I have used it for several projects already.

I am working on a pytorch-based landscape generator, with Godot as the front-end. However, I have run into an issue where the remaining bottleneck is the conversion from python lists and numpy arrays to Godot Array and PoolByteArray on image data.

The output data I am working with currently are 1024x1024 pixel, 4 channel (RGB, depth) image arrays being output at a rate of 12 per second.

I have done some testing, and the fastest method I have found so far is converting the image data to a string, and letting Godot parse_json() convert the received variable into Godot-friendly data. This will result in a rate of approximately 1 image every 2 seconds, however still much slower than the 12 images per second my model is capable of outputting.

Converting to Array (2.7 seconds per image) and PoolByteArray (5.2 seconds per image) is quite a lot slower than above method.

Would you be able to propose a method for faster integration with Godot, in this one specific use case?

Here is a video of the project as it is currently: 5 minutes of AI generated landscapes Thank you.

[touilleMan]

Hi @m5tuff

Numpy array are regular C array under the hood, hence you should be able to use PoolByteArray without any copy (typically using numpy.frombuffer). See the en of the README:

How can I efficiently access PoolArrays?

PoolIntArray, PoolFloatArray, PoolVector3Array and the other pool arrays can't be accessed directly because they must be locked in memory first. Use the arr.raw_access() context manager to lock it:

arr = PoolIntArray() # create the array
arr.resize(10000)

with arr.raw_access() as ptr:
    for i in range(10000):
        ptr[i] = i # this is fast

# read access:
with arr.raw_access() as ptr:
    for i in range(10000):
        assert ptr[i] == i # so is this

Keep in mind great performances comes with great responsabilities: there is no boundary check so you may end up with memory corruption if you don't take care ;-)

[m5tuff]

Hi @touilleMan thank you for your response.

I am trying to understand what you mean, though I believe I may not have expressed the problem I have well enough. I have numpy arrays being output from the machine learning model of shape [1024, 1024, 3] (w, h, color). I need this data to be converted to a PoolByteArray, so that Image.create_from_data(w,h,mip,format, data ) will accept it. Converting the numpy array to bytes with np.ndarray.tobytes() results in a bytes object, and hence this error when trying to load it with create_from_data(): TypeError: Argument 'data' has incorrect type (expected godot.pool_arrays.PoolByteArray, got bytes)

How do I create a data format that is compatible with Image.create_from_data()?

Thank you for your response so far, and thank you for your work.

[touilleMan]

I was suggesting to create the bytes array through godot.pool_arrays.PoolByteArray, then accessing it underlying buffer with godot.pool_arrays.PoolByteArray.raw_access. numpy.frombuffer can then wrap this underlying buffer without copying it. You then end up with a Numpy array that can be used for your Image.create_from_data, the only gotcha is you should be careful about this numpy array object lifetime given it shares the same buffer with the PoolByteArray.

The easy way to avoid lifetime issues would be to create a PoolByteArray singleton with a fixed size when initializing your application (hence the underlying buffer is never freed)