pub get is flaky on M1 Macs, exits 255, error message `.`

[jmagman]

Environment

Seen in Flutter CI

    • Flutter version 3.1.0-0.0.pre.1161 at /opt/s/w/ir/x/w/recipe_cleanup/tmpcvezbt2t/flutter sdk
    • Upstream repository unknown
    • Framework revision 1e53ad4efb (6 minutes ago), 2022-06-08 00:33:07 -0400
    • Engine revision b60a46cf25
    • Dart version 2.18.0 (build 2.18.0-170.0.dev)
    • DevTools version 2.14.0

https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20build_ios_framework_module_test/802/overview

Problem

pub get is flaky on Flutter M1 Mac builders and is failing without a helpful error message. The same test is not flaking on x64 Macs.

Tracked in https://github.com/flutter/flutter/issues/105609.

The test runs three times in https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8811953967788831233/+/u/run_build_ios_framework_module_test/test_stdout and the first two times fails.

Expected behavior

Actual behavior

Exit 255, error message is just a .

 MSG : Got dependencies!
 IO  : Writing 6538 characters to text file .dart_tool/package_config.json.
 FINE: Contents:
 {
   "configVersion": 2,
   "packages": [
     {
       "name": "async",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/async-2.9.0",
       "packageUri": "lib/",
       "languageVersion": "2.14"
     },
     {
       "name": "boolean_selector",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/boolean_selector-2.1.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "characters",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/characters-1.2.1",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "clock",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/clock-1.1.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "collection",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/collection-1.16.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "connectivity",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity-3.0.6",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "connectivity_for_web",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity_for_web-0.4.0+1",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "connectivity_macos",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity_macos-0.2.1+2",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "connectivity_platform_interface",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/connectivity_platform_interface-2.0.1",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "cupertino_icons",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/cupertino_icons-1.0.5",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "fake_async",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/fake_async-1.3.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "flutter",
       "rootUri": "file:///opt/s/w/ir/x/w/recipe_cleanup/tmpcvezbt2t/flutter%20sdk/packages/flutter",
       "packageUri": "lib/",
       "languageVersion": "2.17"
     },
     {
       "name": "flutter_lints",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/flutter_lints-2.0.1",
       "packageUri": "lib/",
       "languageVersion": "2.17"
     },
     {
       "name": "flutter_test",
       "rootUri": "file:///opt/s/w/ir/x/w/recipe_cleanup/tmpcvezbt2t/flutter%20sdk/packages/flutter_test",
       "packageUri": "lib/",
       "languageVersion": "2.17"
     },
     {
       "name": "flutter_web_plugins",
       "rootUri": "file:///opt/s/w/ir/x/w/recipe_cleanup/tmpcvezbt2t/flutter%20sdk/packages/flutter_web_plugins",
       "packageUri": "lib/",
       "languageVersion": "2.17"
     },
     {
       "name": "js",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/js-0.6.4",
       "packageUri": "lib/",
       "languageVersion": "2.16"
     },
     {
       "name": "lints",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/lints-2.0.0",
       "packageUri": "lib/",
       "languageVersion": "2.17"
     },
     {
       "name": "matcher",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/matcher-0.12.11",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "material_color_utilities",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/material_color_utilities-0.1.5",
       "packageUri": "lib/",
       "languageVersion": "2.13"
     },
     {
       "name": "meta",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/meta-1.8.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "package_info",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/package_info-2.0.2",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "path",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/path-1.8.2",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "plugin_platform_interface",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/plugin_platform_interface-2.1.2",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "sky_engine",
       "rootUri": "file:///opt/s/w/ir/x/w/recipe_cleanup/tmpcvezbt2t/flutter%20sdk/bin/cache/pkg/sky_engine",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "source_span",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/source_span-1.9.0",
       "packageUri": "lib/",
       "languageVersion": "2.14"
     },
     {
       "name": "stack_trace",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/stack_trace-1.10.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "stream_channel",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/stream_channel-2.1.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "string_scanner",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/string_scanner-1.1.1",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "term_glyph",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/term_glyph-1.2.0",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "test_api",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.9",
       "packageUri": "lib/",
       "languageVersion": "2.12"
     },
     {
       "name": "vector_math",
       "rootUri": "file:///opt/s/w/ir/x/w/.pub-cache/hosted/pub.dartlang.org/vector_math-2.1.2",
       "packageUri": "lib/",
       "languageVersion": "2.14"
     },
     {
       "name": "hello_project",
       "rootUri": "../",
       "packageUri": "lib/",
       "languageVersion": "2.18"
     }
   ],
   "generated": "2022-06-08T04:42:32.414521Z",
   "generator": "pub",
   "generatorVersion": "2.18.0-170.0.dev"
 }
 .
 Running "flutter pub get" in hello_project... (completed in 253ms)
 "flutter ios-framework" took 307ms.
 pub get failed (255; .)

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8811953967788831233/+/u/run_build_ios_framework_module_test/test_stdout

[jmagman]

@jonasfj do you know who can take a look at this? We're hitting 23% flaky rates. https://github.com/flutter/flutter/issues/105609#issuecomment-1156665257

[jonasfj]

hmm, I'm guessing something is eating the error message. Maybe, in flutter_tools it's pretty hard to follow what messages are filtered and not.

Do we have a pubspec.yaml, then maybe we can reproduce on a m1 locally? Though I suspect this is something else, or maybe a dirty exit or something.

[sigurdm]

Could it be a VM thing? I cannot off hand think of anything in pub that should be architecture specific...

[sigurdm]

Do we know if this started at some point - or was it always like that on M1?

[jonasfj]

@jmagman is there a way to run dart pub get with FLUTTER_ROOT=/path/to/flutter?

Looking at other error messages, I suspect that this might be flutter_tools eating error messages. I'm not sure https://github.com/dart-lang/pub/issues/3463 is related, but the error message detected by flutter is just ..

[sigurdm]

@jmagman is there a way to run dart pub get with FLUTTER_ROOT=/path/to/flutter?

Alternatively to run flutter pub get --verbose

I ran flutter pub get on flutter_tools (each time removing the pubspec.lock) 100 times on a M1-pro machine - and observed no flakyness - so it doesn't seem to be a universal problem for pub on M1.

I think there must be something more going on here in this specific instance.

[zanderso]

There's no filtering in the tool AFAICT. (There's no filter argument passed to batch here: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/dart/pub.dart#L256).

[sigurdm]

I think pub is throwing an uncaught exception - that somehow makes 'dartdev' print a single '.' - why that is I'm still not sure - but I can reproduce by inserting an uncaught throw here: https://github.com/dart-lang/sdk/blob/main/pkg/dartdev/lib/dartdev.dart#L44.

I have no clue why that would happen flakily and why on a specific architecture.

It would probably clarify things immensely to run pub in --verbose mode during the flake.

[zanderso]

From the logs linked above, pub is invoked as:

dart __deprecated_pub --verbose get --no-precompile

[sigurdm]

From the logs linked above, pub is invoked as:

dart __deprecated_pub --verbose get --no-precompile

Hmm.

Then I think we want to modify the script try running dart pub get --verbose instead of flutter pub get to get more information.

[zanderso]

It looks like dart __deprecated_pub came from https://github.com/flutter/flutter/pull/89032.

[christopherfujino]

So I'm exploring migrating from dart __deprecated_pub to dart pub in https://github.com/flutter/flutter/pull/106969. More details on blockers in that PR.

[christopherfujino]

@rmacnak-google can you elaborate more on why you think this may have the same root cause as https://github.com/dart-lang/sdk/issues/49437? Do you think your patch may have fixed it?