search

dart-lang / co19

A Dart language and library conformance test suite
BSD 3-Clause "New" or "Revised" License
37 stars 28 forks source link

Please triage the failures from the latest NNBD test roll #509

Closed athomas closed 4 years ago

athomas commented 4 years ago

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8893749698941313056/+/steps/test_results/0/logs/unapproved_new_test_failures__logs_/0

Which of these failures are test issues and which are tool issues?

analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/list_length_t02 is new and failed (CompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/list_length_t04 is new and failed (CompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_aware_operator_A11_t13 failed again (CompileTimeError -> Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_check_operator_A01_t01 failed again (CompileTimeError -> StaticWarning, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_check_operator_A02_t01 is new and failed (CompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_check_operator_A02_t02 is new and failed (CompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_check_operator_A06_t01 is new and failed (StaticWarning, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/null_check_operator_A06_t02 is new and failed (StaticWarning, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t04 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t06 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t04 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t06 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t08 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t09 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t11 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t13 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t15 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t14 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t16 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t18 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t20 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t03 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t06 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t09 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A11_t02 failed again (CompileTimeError -> MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A11_t08 is new and failed (StaticWarning, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A11_t11 is new and failed (CompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A11_t13 is new and failed (CompileTimeError, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A13_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A14_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A15_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A15_t03 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t05/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t05/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t06/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t06/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t07/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t07/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t08/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t08/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t09/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t09/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t10/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t10/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t11/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t11/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t12/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t12/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t13/01 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t13/02 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t13/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t13/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t14/03 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A20_t14/04 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A24_t02 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A25_t01 is new and failed (MissingCompileTimeError, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A27_t06/01 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/static_errors_A27_t07/01 is new and failed (Pass, expected StaticWarning)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/syntax_A06_t01 failed again (CompileTimeError -> StaticWarning, expected Pass)
analyzer-asserts-strong-linux:co19/LanguageFeatures/nnbd/syntax_A06_t03 failed again (CompileTimeError -> StaticWarning, expected Pass)
sgrekhov commented 4 years ago

The result of the triage

co19/LanguageFeatures/nnbd/list_length_t02 is new and failed (CompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/list_length_t04 is new and failed (CompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/null_aware_operator_A11_t13 failed again (CompileTimeError -> Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/null_check_operator_A01_t01 failed again (CompileTimeError -> StaticWarning, expected Pass) | Issue 39723
co19/LanguageFeatures/nnbd/null_check_operator_A02_t01 is new and failed (CompileTimeError, expected Pass) | Issue 39758
co19/LanguageFeatures/nnbd/null_check_operator_A02_t02 is new and failed (CompileTimeError, expected Pass) | Issue 39758
co19/LanguageFeatures/nnbd/null_check_operator_A06_t01 is new and failed (StaticWarning, expected Pass) | Issue 39723
co19/LanguageFeatures/nnbd/null_check_operator_A06_t02 is new and failed (StaticWarning, expected Pass) | Issue 39758
co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t02 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t04 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_FIELD_t06 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t02 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t04 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t06 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t08 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t09 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t11 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t13 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_REQ_ARGS_t15 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t14 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t16 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t18 is new and failed (MissingCompileTimeError, expected Pass) | Deleted
co19/LanguageFeatures/nnbd/override_checking_A02_RETURN_t20 is new and failed (MissingCompileTimeError, expected Pass) | Deleted
co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t03 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t06 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/override_checking_A02_SETTER_t09 is new and failed (MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/static_errors_A11_t02 failed again (CompileTimeError -> MissingCompileTimeError, expected Pass) | Fixed
co19/LanguageFeatures/nnbd/static_errors_A11_t08 is new and failed (StaticWarning, expected Pass) | Issue 39865
co19/LanguageFeatures/nnbd/static_errors_A11_t11 is new and failed (CompileTimeError, expected Pass) | Nonfunction type aliases are not implemented yet
co19/LanguageFeatures/nnbd/static_errors_A11_t13 is new and failed (CompileTimeError, expected StaticWarning) | Issue 38715
co19/LanguageFeatures/nnbd/static_errors_A13_t02 is new and failed (MissingCompileTimeError, expected Pass) | Nonfunction type aliases are not implemented yet, 39866
co19/LanguageFeatures/nnbd/static_errors_A14_t02 is new and failed (MissingCompileTimeError, expected Pass) | Nonfunction type aliases are not implemented yet, 39866
co19/LanguageFeatures/nnbd/static_errors_A15_t02 is new and failed (MissingCompileTimeError, expected Pass) | Nonfunction type aliases are not implemented yet
co19/LanguageFeatures/nnbd/static_errors_A15_t03 is new and failed (MissingCompileTimeError, expected Pass) | Nonfunction type aliases are not implemented yet
co19/LanguageFeatures/nnbd/static_errors_A20_t05/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t05/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t06/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t06/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t07/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t07/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t08/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t08/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t09/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t09/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t10/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t10/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t11/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t11/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t12/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t12/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t13/01 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t13/02 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t13/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t13/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t14/03 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A20_t14/04 is new and failed (Pass, expected StaticWarning) | Issue 39714
co19/LanguageFeatures/nnbd/static_errors_A24_t02 is new and failed (MissingCompileTimeError, expected Pass) | Issue 39661
co19/LanguageFeatures/nnbd/static_errors_A25_t01 is new and failed (MissingCompileTimeError, expected Pass) | Issue 39681
co19/LanguageFeatures/nnbd/static_errors_A27_t06/01 is new and failed (Pass, expected StaticWarning) | Issue 39867
co19/LanguageFeatures/nnbd/static_errors_A27_t07/01 is new and failed (Pass, expected StaticWarning) | Issue 39867
co19/LanguageFeatures/nnbd/syntax_A06_t01 failed again (CompileTimeError -> StaticWarning, expected Pass) | NREP, probably issue 39865
co19/LanguageFeatures/nnbd/syntax_A06_t03 failed again (CompileTimeError -> StaticWarning, expected Pass) | NREP, probably Issue 39865
sgrekhov commented 4 years ago

@athomas I triaged the failures. May I close this issue?

athomas commented 4 years ago

Yes, thanks, but please also answer the questions below.

What does "Dart versioning is not implemented yet" mean?

What does NREP mean?

sgrekhov commented 4 years ago

What does "Dart versioning is not implemented yet" mean?

I thought that the reason of the failure is that support of // @dart=2.6 tags in tests is not implemented yet and therefore legacy code is treated as opted in and this causes a failure. That's why I triaged these tests as "Dart versioning is not implemented yet". But after clarification from Erik we found that tests expectations were wrong. Now all of them are fixed, triage results updated. Sorry for wrong info

What does NREP mean?

These failures are not reproducible on dartanalyzer version 2.8.0-dev.0.0. According to the tryjob logs these failures looks like SDK issue https://github.com/dart-lang/sdk/issues/39865. But this issue is reproducible on my version of SDK, issues to be triaged not. So I don't close this issue to check it on the next version of SDK

athomas commented 4 years ago

Thanks!