GPU Buffer sharing with Godot Shaders

[Ivorforce]

Describe the proposed feature and its relevance to inferencing

As i understand, there is currently no way to link the inputs or outputs of IREE gpu tasks to Godot shaders (compute, or regular vert / frag).

Context or use case

Adding ability to use IREE buffers in godot shaders and vice versa would unlock a whole suit of new capabilities. For one, IREE.gd could be used instead of compute shaders for scientific tasks (i.e. non machine learning tasks).

Proposed solution

It should be possible to use GPU IREE output tensors / buffers as godot GPU buffers / textures. It should also be possible to use godot GPU buffers / textures as IREE input tensors / buffers.

Additional information

No response

[fire]

I want this. No idea how to do it.

I assume it'll be like sharing buffers in metal or vulkan.

[RechieKho]

IREE tensor is different than Godot's GPU buffer (texture) as IREE tensor could encapsulate multiple dimension of data while texture is width x height x channel count. If one one constricted with image texture then it will definitely be too limiting and not use the full power of IREE.

It is still possible to convert them manually in Godot. Though if you want to have it feels like native support, well, it is still possible to extend texture class and put IREE GPU computation inside (just conversion code in CPP). It won't escape the fate of preprocessing because a lot of tflite model only accept certain dimensions only

[fire]

extend texture class and put IREE GPU computation inside

@Ivorforce thoughts on this?

[fire]

My first impression is we need a ring buffer that syncs the two rates of the iree.gd and the frame sync on the gpu access.

But I don't know.

[Ivorforce]

IREE tensor is different than Godot's GPU buffer (texture) as IREE tensor could encapsulate multiple dimension of data while texture is width x height x channel count. If one one constricted with image texture then it will definitely be too limiting and not use the full power of IREE.

Well, there's two sides to that. For one, the output of a model is often 0D, 1D or 2D. Doesn't mean the rest of the model has to be. But other than for training, I have rarely seen more than 2D as output or input for models.

Besides that, every tensor is 1D if you flatten it (or just address the underlying buffer of the ND tensor manually). You can totally pack any ND tensor into a regular GPU buffer and just re-ravel the dimensions in the shader manually. The important part is that the data transfer is fast.

It is still possible to convert them manually in Godot.

By "manually" you mean on the CPU by downloading and re-uploading? Or by binding them together on the GPU?

Though if you want to have it feels like native support, well, it is still possible to extend texture class and put IREE GPU computation inside (just conversion code in CPP). It won't escape the fate of preprocessing because a lot of tflite model only accept certain dimensions only

What do you mean by "put the computation inside"? IREE should be able to use its own buffers for any computation, the only thing that needs to be linked is singular tensors, usually output or input. The computation should still be managed fully by IREE.

[RechieKho]

By manual I mean convert it in CPU space and put it into Godot's shader. It seems like you want to achieve something like both Godot and IREE accessing the same GPU buffer. This is beyond my understanding as well...

[Ivorforce]

Yes, for most features I'd use it would be essential to avoid downloading the data to the CPU. You got it, it would mean both would need to have access to the same buffer.

Let's keep this issue open for when anybody has a chance to take a look and dissect our options in detail.

[Ivorforce]

I just remembered Syphon exists. It's macOS only, and uses OpenGL (and apparently also Metal now), but it may be a good resource to learn from.

Specifically, the ideal way may be for Godot and IREE to share a GPU context, but if that doesn't work, then maybe intra-GPU buffer sharing in a Syphon-like fashion may be possible.

[fire]

https://github.com/godotengine/godot/pull/98788 @Ivorforce We've plumbing the mesh support in godot engine 4.4 Thoughts?

[Ivorforce]

godotengine/godot#98788 @Ivorforce We've plumbing the mesh support in godot engine 4.4 Thoughts?

I don't know much about rendering servers, but it looks like the PR implements getting a multimesh's buffer RID, which may be relevant here if somebody wants to access a multimesh's buffer to operate on it. But I don't know if it brings us closer to GPU buffer sharing.

[fire]

@Ivorforce From my point of view the multimesh buffer is a packed byte array, we can reuse the same strategy to share a different packed byte array. Batching is encapsulated as a deeper packed byte array.

Would that be sufficient? Or do we also need to support tensors with explicit types and shapes?