search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.06k stars 1.56k forks source link

Failures on [infra] Add more arm64 configurations to the test matrix...[CFE] First pass at internal test utility #46469

Open mkustermann opened 3 years ago

mkustermann commented 3 years ago

There are new test failures on [infra] Add more arm64 configurations to the test matrix...[CFE] First pass at internal test utility.

The tests

co19/Language/Classes/Abstract_Instance_Members/inherited_t07 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t08 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t09 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t10 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t11 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t12 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/inherited_t15 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/invocation_t03 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/invocation_t04 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/invocation_t06 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t01 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t02 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t03 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t04 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t05 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t06 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t07 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t08 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t09 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_default_value_t10 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_no_named_parameters_t02 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_no_named_parameters_t03 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/override_no_named_parameters_t04 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t01 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t02 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t04 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t05 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t06 Crash (expected Pass)
co19/Language/Classes/Abstract_Instance_Members/same_name_static_method_in_superclass_t07 Crash (expected Pass)
co19/Language/Classes/Class_Member_Conflicts/static_member_and_constructor_t01/none Crash (expected Pass)
    and 17604 more tests

are failing on configurations

dartkp-strong-mac-release-arm64
mkustermann commented 3 years ago

Some of those test failures are fine since they also fail in other existing configurations. (@athomas Do we have a way to diff current results across two configurations?)

Though the crashes we should investigate.

/cc @a-siva

athomas commented 3 years ago

These results are bogus. They were the first attempt to create the builder but it failed because of a missing gen_snapshot. The new builders are baselined on existing ones to avoid this kind of issue, but because the builder didn't work at all everything changed despite the baseline.

This is the builder I'm trying to setup in https://dart-review.googlesource.com/c/sdk/+/204505.

athomas commented 3 years ago

@whesse Do we have some way of removing those results safely?

whesse commented 3 years ago

We can remove the approved failure records, but they will not be replaced by new records representing the diff between the initial baseline and the new configuration. So removing the records, just so these (approved) failures, that are immediately changed to passes, disappear from the feed doesn't seem to gain us much.

We could create result records showing the diff between the initial baseline (from another builder) and the new builder by copying the baseline over the results.json in latest, so the next build will compare itself to the baseline.

Currently, many tests seem to be changing state (randomly?) from crash to pass or compile-time error, and back again, so it may be best to investigate these crashes before trying to get a clean comparison with a baseline.