search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.26k stars 1.58k forks source link

Add a trybot for flutter customer tests #51042

Open srawlins opened 1 year ago

srawlins commented 1 year ago

I have landed several changes into dart-lang/sdk, which have each ultimately caused some bot somewhere to fail, as each change causes one or most customer tests to fail in the flutter/tests repo. There is no trybot for these "flutter customer tests."

This current feedback loop is very expensive for me and for the flutter engine rollers:

  1. it takes several days for me to find out the change causes new failures in flutter/tests
  2. it breaks some flutter auto-roller or other
  3. breaking the auto-roller requires humans to identify the culprit, and revert my change
  4. I must fix each customer's codebase to be compatible with my change
  5. I must bump the commit SHA of each customer's codebase in flutter/tests
  6. I land again, without any customer test trybots, cycling back to the top of this feedback loop.

CC @Hixie @jason-simmons @iskakaushik to perhaps fill in details I got wrong above

srawlins commented 1 year ago

I can find more references, but as one data point, here are links for the latest cycle:

Hixie commented 1 year ago

The flutter/tests tests are designed to be runnable from any repo, so it should be possible to run them upstream. I forget if we have an environment variable to force a local engine to be used (I presume you'd have to build a local engine and use that to run the tests to test Dart changes), but if we don't we should be able to add one. cc @christopherfujino

whesse commented 1 year ago

Regarding local engine: The engine_v2 recipes run tests with a custom engine by passing a custom FLUTTER_STORAGE_BASE_URL and using flutter tools, not using the local-engine flag.

These tests are run through the adhoc_validation recipe module, and not through the standard test script 'dev/bots/test.dart'.

So to add this (to monorepo testing), we will create a new validation_tester recipe that can be triggered from the engine_v2 api, that spawns testers in new builds after the engine build is complete. We already have an engine_v2/tester that runs dev/bots/test.dart, but this should be a different recipe to run adhoc_validation steps instead.

We will have to make sure that the adhoc_validation steps run correctly on a flutter/flutter checkout with a custom artifact download URL - we have a bug filed because custom artifact download URLs break some tests that run flutter tools with a --fatal-warnings flag, but are working on fixing this.

This change would need to be made in the flutter recipes anyway, as part of the new engine_v2 Build/Archive/Test separation they are moving their builds to, so we will just do it earlier.

CC @godofredoc

The timeline for implementing this will be weeks, not days.

srawlins commented 1 year ago

Thanks much for all of the context, @whesse !!!

The timeline for implementing this will be weeks, not days.

Very understandable, haha. Having this implemented in weeks would be a huge win for the analyzer team at least, and probably more. Thanks!

srawlins commented 1 year ago

Is it possible this might be prioritized for Q3? Thanks!

a-siva commented 1 year ago

Is it possible this might be prioritized for Q3? Thanks!

@godofredoc any possibility work on this could be bumped, when we run into issues with this codebase it is problematic for us to see if we have a fix or not without having the ability to run the changes locally

godofredoc commented 1 year ago

I can take a look next week to see if we can move customer testing to dev/bots/test.dart as it is only 4 commands in bash https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/adhoc_validation/resources/customer_testing.sh

kallentu commented 1 year ago

I ran into this a couple days ago where my change in the Dart SDK broke a Flutter customer. And there were two layers of errors, so it took a day or two to figure out it was my change and then my first forward fix fixed the first layer of errors, but there were more that I was unaware about until another day or two after. Then when I was creating a fix, the turnaround time for testing it would be up to half a day.

It's not a great experience to be mostly certain about the effects of your changes and need to test by waiting for multiple versions and rolls to go through.

Would love to see additional testing on the Dart pipeline 👍

whesse commented 1 year ago

This is now on my list of priority tasks to do, it is clearly an important request.

srawlins commented 1 year ago

Thanks much Bill, really super appreciated!

srawlins commented 1 year ago

If it's helpful to have some examples to play with, I'd like to run these three CRs against the flutter customer tests in try-bots:

godofredoc commented 11 months ago

Customer_testing is now a shard test and it is ready to be added to the monorepo configs. @whesse FYI

whesse commented 11 months ago

Customer_testing has been added to monorepo, and is now running on CI. It is available as a try job for Dart SDK CLs by adding the flutter-linux-try builder with the "Choose Tryjobs" link.

srawlins commented 11 months ago

Thanks so much, Bill!!! I'll be using this a lot this week. Much appreciated!

godofredoc commented 11 months ago

Thank you @whesse