Benchmarks

[dcharkes]

We should add benchmarks to this repo.

• jni/benchmark and jnigen/benchmark
• Use https://pub.dev/packages/benchmark_harness
  • You can see some examples https://github.com/dart-lang/sdk/tree/main/benchmarks/FfiMemory
  • You can see some examples https://github.com/dart-lang/sdk/tree/main/benchmarks/FfiCall
  • (But please don't copy the non-Dart-style Dart!)
• They should be run on CI (just like the jni(gen)/example) to ensure they work (at least the Dart standalone ones)
• We should explore if there is a way to do benchmarks on CI.
  • https://github.com/benchmark-action has no Dart support yet, but if the interface is json or something we might still be able to use it.

We should include benchmarks with method channels to see how the performance compares:

• jnigen with generating C code
• jnigen with generating only Dart code
• method channels

Unfortunately, running a Flutter app is hard without a window manager, so that might make it impossible to run the method channels benchmarks in an automated fashion.

So, besides having trying to run the benchmarks on CI, we should probably also upload benchmark results manually to the repo. (Extra kudos for making the benchmarks just generate json/yaml as output and a script that takes all the benchmarks, collects all the outputs and generates a benchmark_results.md so that updating the numbers is trivial later on.)

Fur running flutter apps from CLI flutter drive. (Or use flutter run capture stdout and just exit(0); when done with benchmarking.)

[mahesh-hegde]

1. Can benchmark_harness be run on flutter Android device?

2. Can this be useful to automatically benchmark on real device? https://docs.flutter.dev/testing/integration-tests#testing-on-firebase-test-lab

[fzyzcjy]

Btw I come across this and here are my two cents:

We should explore if there is a way to do benchmarks on CI. @benchmark-action has no Dart support yet, but if the interface is json or something we might still be able to use it.

Yes, I already did it in flutter_rust_bridge v2 and it works well. You simply output to a file with the JSON format it requires, and everything works.

[HosseinYousefi]

I made this repo for various benchmarks. Different branches contain different experiments. https://github.com/HosseinYousefi/jnigen_benchmark

Adding this to 1.0.0 milestone to make sure the CI runs benchmarks and provides reports.