macos_ui package tests are very slow

[Hixie]

The macos_ui tests run as part of Flutter's infrastructure. There's about 141 unit tests, but they take 5-7 minutes to run. This is in contrast to some other tests, e.g. the flutter_reactive_ble tests take less than a minute with also about a 100 unit tests.

For example, see this log: https://api.cirrus-ci.com/v1/task/5549991612121088/logs/main.log

I believe these instructions will run the tests:

git clone https://github.com/GroovinChip/macos_ui.git tests
git -C tests checkout f1053fc31958dbaa362a0a4c8288e8cf0919ab8c
cd tests; time flutter test

[mraleph]

@Hixie do you know what kind of hardware this runs on?

[mraleph]

I looked at darwin-arm64 artifacts and frontend_server.dart.snapshot is a kernel binary rather than app-jit snapshot. I think this is due to logic here: https://github.com/flutter/engine/blob/main/build/dart/rules.gni#L256-L259

It means all M1 users have a really slow front-end performance (it warms up, but still). Somebody needs to fix those rules. Better yet we should just switch this to use AOT, though cross-compilation becomes an interesting open question (cc @a-siva).

Also I am not sure why we still have Flutter specific front-end server binary. I thought we were trying to move it to Dart SDK (@aam @jensjoha @johnniwinther).

[aam]

Also I am not sure why we still have Flutter specific front-end server binary. I thought we were trying to move it to Dart SDK (@aam @jensjoha @johnniwinther).

Only reason I could see is that the snapshot for flutter is trained with flutter-specific parameters https://github.com/flutter/engine/blob/main/flutter_frontend_server/lib/server.dart#L56.

[Hixie]

@Hixie do you know what kind of hardware this runs on?

I don't, sorry.

[Hixie]

FWIW, I believe the specific tests in the log above were running on Linux x64.

[mraleph]

FWIW, I believe the specific tests in the log above were running on Linux x64.

I guess, I should not have jumpted to the conclusion based on the name of the test suite.

@jensjoha could you take a look at these tests? (e.g. what is the breakdown between compiling time and running time, etc).

[jensjoha]

When running this in my (linux) workstation it takes 12-25 seconds. When running on a linux container inside my chromebox it takes 35-50 seconds.

Is there something else I can run to better mimic what the CI does? @Hixie can you reproduce this with the commands you gave?

[mraleph]

@jensjoha can you compare to flutter_reactive_ble? How long does it run on your workstation? (e.g. does it take 1 second? :))

Also I notice in the log that we seem to use some helper script:

++ dart flutter/dev/customer_testing/run_tests.dart --shards 3 --shard-index 1 --repeat=15 --skip-template --verbose registry/DanTup_tiler.test registry/assorted_layout_widgets.test registry/dart_rfb.test registry/flutter_cocoon.test registry/flutter_gallery.test registry/flutter_packages.test registry/flutter_portal.test registry/flutter_reactive_ble.test registry/flutter_rfb.test registry/flutter_svg.test registry/macos_ui.test registry/provider.test registry/super_editor.test registry/template.test
Starting run_tests.dart...

Maybe that does something weird?

[jensjoha]

@jensjoha can you compare to flutter_reactive_ble? How long does it run on your workstation? (e.g. does it take 1 second? :))

Pretty much :/

First run:

$ ./bin/flutter_customer_test.sh
Analyzing flutter_reactive_ble...
No issues found! (ran in 10.4s)
00:12 +47: All tests passed!
Analyzing reactive_ble_mobile...
No issues found! (ran in 1.9s)
00:10 +100: All tests passed!
Analyzing reactive_ble_platform_interface...
No issues found! (ran in 1.7s)
00:10 +4: All tests passed!

another time:

$ ./bin/flutter_customer_test.sh
Analyzing flutter_reactive_ble...
No issues found! (ran in 1.0s)
00:03 +47: All tests passed!
Analyzing reactive_ble_mobile...
No issues found! (ran in 1.1s)
00:02 +100: All tests passed!
Analyzing reactive_ble_platform_interface...
No issues found! (ran in 1.0s)
00:01 +4: All tests passed!

[jensjoha]

As for the helper script it is passed parameters to files in some "registry" folder which I don't know where comes from.

[mraleph]

But then maybe relative factor between these two test suites is similar? It's just that CI is much-much-much slower? So it is still worth looking where it spends time.

@godofredoc can somebody from Flutter infra help us to figure out what kind of vms Cirrus runs on? They seem to be really slow.

[Hixie]

The helper script is https://github.com/flutter/flutter/tree/master/dev/customer_testing but all it does is just run the commands I described above.

I agree that the relative times is what matters here. We're not expecting the CI machines to be fast in absolute terms. And it's quite possible that the macos_ui tests are doing more work. It just seems very surprising to me that they'd be that much slower than other similar-sized test suites run in the same container. We should probably profile them and see if there's something particularly expensive running.