[google_maps_flutter] Some iOS integration tests failing in CI

[stuartmorgan]

We're seeing OOB persistent failures in a number of tests on iOS:

00:18 +8: testInitialCenterLocationAtCenter                                                                                                                                                            
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: <400>
  Actual: <9223372036854775807>

When the exception was thrown, this was the stack:
#4      main.<anonymous closure> (file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart:340:5)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:168:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1017:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart line 340
The test description was:
  testInitialCenterLocationAtCenter

[...]

00:18 +8 -1: testGetVisibleRegion                                                                                                                                                                      
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: true
  Actual: <false>

When the exception was thrown, this was the stack:
#4      main.<anonymous closure> (file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart:376:5)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:168:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1017:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart line 376
The test description was:
  testGetVisibleRegion
════════════════════════════════════════════════════════════════════════════════════════════════════

00:18 +8 -2: testGetVisibleRegion [E]                                                                                                                                                                  
  Test failed. See exception logs above.
  The test description was: testGetVisibleRegion

[...]

00:21 +17 -2: testGetZoomLevel                                                                                                                                                                         
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: <7>
  Actual: <5.0>

When the exception was thrown, this was the stack:
#4      main.<anonymous closure> (file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart:701:5)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:168:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1017:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart line 701
The test description was:
  testGetZoomLevel
════════════════════════════════════════════════════════════════════════════════════════════════════

00:21 +17 -3: testGetZoomLevel [E]                                                                                                                                                                     
  Test failed. See exception logs above.
  The test description was: testGetZoomLevel

[...]

00:22 +17 -3: testScreenCoordinate                                                                                                                                                                     
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: ScreenCoordinate:<ScreenCoordinate(0, 0)>
  Actual: ScreenCoordinate:<ScreenCoordinate(9223372036854775807, 9223372036854775807)>

When the exception was thrown, this was the stack:
#4      main.<anonymous closure> (file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart:735:5)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:168:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1017:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Volumes/Work/s/w/ir/x/w/packages/packages/google_maps_flutter/google_maps_flutter_ios/example/ios11/integration_test/google_maps_test.dart line 735
The test description was:
  testScreenCoordinate
════════════════════════════════════════════════════════════════════════════════════════════════════

00:22 +17 -4: testScreenCoordinate [E]                                                                                                                                                                 
  Test failed. See exception logs above.
  The test description was: testScreenCoordinate

[...]

00:27 +20 -4: testTakeSnapshot - did not complete [E]  

I can't reproduce them locally, so I don't have a good way to investigate right now. To re-open the tree, I'm going to disable these tests in google_maps_flutter_ios, as well as for iOS in google_maps_flutter (all with comments referencing this issue to make them findable).

Since it's OOB and I can't repro locally, my current speculation is either:

• a server change that broke us that hasn't rolled out everywhere, or
• the CI is being identified as malicious usage by the server.

The latter may be impacted by the fact that we aren't currently running with an API key in CI (although that can't explain it fully, because I don't have an API key set up locally). I do plan at some point to get the API key working in CI, but since it hasn't affected anything I haven't set up the plumbing, so I can't easily test whether the situation would be improved by having a key (e.g., if the problem is tripping a malicious usage detection, maybe a key would avoid that?)

[stuartmorgan]

We should try to prioritize fixing this pretty highly, because we'll lose a lot of test coverage by disabling these.

[stuartmorgan]

A potential pattern: both tests using getScreenCoordinate are failing (and seemingly returning -1, -1), and getVisibleRegion presumably uses the same underlying info. (I don't have a connection for getZoomLevel though.)

[stuartmorgan]

In the app-facing package, testTakeSnapshot was already disabled for non-iOS because of lack of API key, interestingly.

[flutter-triage-bot[bot]]

This issue is assigned to @jmagman but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

[flutter-triage-bot[bot]]

This issue was assigned to @jmagman but has had no status updates in a long time. To remove any ambiguity about whether the issue is being worked on, the assignee was removed.