The fastest possible way to send [u8] rgba pixels from Rust and render as image in Flutter?

[baleksey]

I have motion graphics engine in rust (Skia based) and trying to create GUI for it in Flutter. The problem is while I renders desired image very fast on the rust size (< 1 ms) I cant send and render it fast enough on Flutter side. The first problem it's not that easy and fast to render raw pixels in flutter without any hacks (the best I get is to convert those pixels to bmp by adding special header). The second, and most important problem it send my 3000x3000 rgba pixels almost 90-100ms to flutter. It's very far from realtime which I want to achieve.

Just in case my current Rust/Flutter codes: Rust

#[flutter_rust_bridge::frb(sync)] 
pub fn renderImage(width:i64, height:i64) -> BMPimage {
    let mut img = vec![0; (width * height * 4) as usize]; // pretend some rendering goes here
    let res = BMPimage::new(width as u32, height as u32, img); //-- 29ms!  adding BMP format header as I don't know how to render raw pixels on flutter side.
    res
}

Flutter:

    BMPimage bmpImg = RustLib.instance.api.renderImage(width: 3000, height: 3000); // 110ms !
    var imageWidget = Image.memory(bmpImg.bmp);

Could anyone help to make send/render on flutter action as fast as possible?

[welcome[bot]]

Hi! Thanks for opening your first issue here! :smile:

[fzyzcjy]

The first problem it's not that easy and fast to render raw pixels in flutter without any hacks (the best I get is to convert those pixels to bmp by adding special header).

I personally also convert it to bmp to let flutter to render.

[flutter_rust_bridge::frb(sync)]

The second, and most important problem it send my 3000x3000 rgba pixels almost 90-100ms to flutter. It's very far from realtime which I want to achieve.

I see: Could you please try non-sync firstly? IIRC sync mode does not support zero-copy-buffer yet. In your scenario, I guess non-sync is better, since that mode does not block the main flutter thread - thus even if your rust thread is somehow slow it will not cause problems.

[powntheep]

There is this package which allows you to create texture using rust and rendering it in Flutter via the Texture widget. It might be faster than using the RawImage widget. Down side is that its not applicable to the web platform.

https://github.com/irondash/irondash/tree/main/texture

[fzyzcjy]

That one looks interesting!

[baleksey]

@fzyzcjy Tried async mode. Don't sure I doing it right as I get almost the same slow result. Maybe little bit faster: from 110 ms to 70ms improvement (still unusable for realtime playback).

powntheep's example gives much more speed. It is still not perfect but with irondash lib for lower resolution images (on my PC it's 2000x2000 px) it plays almost perfectly in real time. This approach has another problems I'm still fighting with (like it seems it very dependant if my system uses other GL heavy workloads like playing youtube, which causes pretty strong lags). But overall it is VERY promising for real time playback (but not for web). It would be so great if you can look into it and add those capabilities to FRB lib..

Btw, what approach works the best for you now and what results in terms of speed do you have?

[fzyzcjy]

Tried async mode. Don't sure I doing it right as I get almost the same slow result. Maybe little bit faster: from 110 ms to 70ms improvement (still unusable for realtime playback).

Hi, could you please provide a minimal reproducible sample? Then I can debug on it and see what is going on.

Btw, (just a guess, it may or may not improve a lot) your slow "adding BMP format header" may not be that slow by optimizing memory copies etc.

It would be so great if you can look into it and add those capabilities to FRB lib.

So, do you succeed in using irondash's Texture with frb, or is it somehow incompatible?

Btw, what approach works the best for you now and what results in terms of speed do you have?

I personally have very heavy computations (computer vision algorithms, etc) on Rust, thus I use async and do not care about the realtime thing. However, as mentioned above, it should be zero-copy and not that slow.

[baleksey]

@fzyzcjy Sure, here you go: test_app.zip I'm running it as: flutter run -d linux --release

So, do you succeed in using irondash's Texture with frb, or is it somehow incompatible?

Not yet, I've just played with example by adding my rust library and measuring rendering speed on flutter side. Haven't tried to connected it with frb yet. I just thought, that adding this feature to frb could help all other who struggle with some kind of real time playback and it's a cool addition to your lib :)

[fzyzcjy]

Oh, it would be great to be a github repo instead of a zip :)

I just thought, that adding this feature to frb could help all other who struggle with some kind of real time playback and it's a cool addition to your lib :)

I guess making irondash compatible with frb (if possible and if it is not yet compatible) would be a great way!

[baleksey]

Made a repo: https://github.com/baleksey/test_flutter_image Hope it helps :)

[fzyzcjy]

Looks good! I will check it in the next batch (hopefully within a week or so)

[fzyzcjy]

Btw, have you tried transferring pure Vec<u8> and test the speed, in order to isolate the problem - is it because of flutter_rust_bridge which wrongly transfer the bytes, or because of other things such as bmp-header-creation code.

EDIT: In addition, temporarily remove Image.memory to further make it minimal reproducible

EDIT: Btw, ensure flutter run --release (or --profile) is used to test speed, since the debug mode is usually super slow (to allow a lot of debug utilities) I see you are already using release, thus no problem for this!

EDIT: Maybe we should isolate flutter startup time with the real transfer time

EDIT: I am checking it

[fzyzcjy]

I see: It seems that you are using the (default) non-full-dep mode. Just set full_dep: true in your flutter_rust_bridge.yaml, and the fastest codec will be used - but you will need to follow the hints (if any) to ensure LLVM is installed on your computer.

Then it is something like:

IN RUST: 4.354µs
 => IN FLUTTER: 167 us

on my computer with code https://github.com/fzyzcjy/flutter_rust_bridge/tree/feat/1776 (look at frb_example/dart_minimal).

Feel free to ping me if changing this flag does not work for you!

[baleksey]

@fzyzcjy Here is what I get after updating my test project with your modifications: IN RUST: 10.014µs IN FLUTTER: 90737 us As you see even with full_dep: true I still have ~90ms transfer speed (I run --release of course). I've updated https://github.com/baleksey/test_flutter_image with those edits. Can it be problem with linux (and for linux) build if you do it on windows?

[fzyzcjy]

After full_dep=true, you need to re-run codegen. It seems that https://github.com/baleksey/test_flutter_image/blob/main/rust/src/frb_generated.rs shows the old code (i.e. codegen may not be run yet)

[baleksey]

You are right, I've managed to get your speed results and now Rust > Flutter data transfer is almost instant. Thank you a lot for your help and patience!

Unfortunately with all that transfer speed I still can't render images in realtime on the flutter side because of terrible flutter limitation for drawing images: On the rust side I need to convert raw pixels data to BMP witch takes huge amount of time (for 3k image ~ 17ms) or I just don't know how to speed it up. But even more problems on Flutter side: in order to load/decode received bmp image Flutter needs from 15 to 150 ms (avg ~20 ms)! It's minimum ~ 30-40 ms for all that unnecessary conversions and loading. I don't know how to solve that.. I've updated my repo in order to demonstrate that if you still have time and desire to look into it. But I totally understand it's beyond frb lib.

[fzyzcjy]

You are welcome and happy to see it runs!

I see the problem, which looks quite possible, because those decoding and loading needs (at least) several copies and other work. So I think the texture solution that @powntheep suggests may be a good alternative. Feel free to ping me if anything does not work!

[h3x4d3c1m4l]

Hi all! In MomentoBooth I also use texture streaming from Rust to Flutter. Works great. I did have to patch the Rust texture access library to use RGB instead of the default order of subpixels.

Performance is very high as texture rendering does not cause any work for Flutter.

See here for Flutter side code (creating the texture) and here for Rust side code for writing to the texture.

[fzyzcjy]

That looks great!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.