flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
166.26k stars 27.51k forks source link

integration_tests fail on a CI system targeting iOS 14, stuck on local network permission popup #65217

Open jmagman opened 4 years ago

jmagman commented 4 years ago

On iOS 14, an integration_test test will prompt the user to allow local network access when the observatory publishes:

The tests hang forever until you hit OK, so fails in a CI setup when there's no one to hit the button.

Steps to Reproduce

Fix up the iOS example to actually work.... https://github.com/flutter/plugins/tree/master/packages/integration_test/example/ios

  1. Check in the ios/Podfile with RunnerTests inherit! :search_paths set up (not sure why the README says to add a use_framework!...)
    
    # Uncomment this line to define a global platform for your project
    # platform :ios, '9.0'

CocoaPods analytics sends network stats synchronously affecting flutter build latency.

ENV['COCOAPODS_DISABLE_STATS'] = 'true'

project 'Runner', { 'Debug' => :debug, 'Profile' => :release, 'Release' => :release, }

def flutter_root generated_xcode_build_settings_path = File.expand_path(File.join('..', 'Flutter', 'Generated.xcconfig'), FILE) unless File.exist?(generated_xcode_build_settings_path) raise "#{generated_xcode_build_settings_path} must exist. If you're running pod install manually, make sure flutter pub get is executed first" end

File.foreach(generated_xcode_build_settings_path) do |line| matches = line.match(/FLUTTER_ROOT\=(.*)/) return matches[1].strip if matches end raise "FLUTTER_ROOT not found in #{generated_xcode_build_settings_path}. Try deleting Generated.xcconfig, then run flutter pub get" end

require File.expand_path(File.join('packages', 'flutter_tools', 'bin', 'podhelper'), flutter_root)

flutter_ios_podfile_setup

target 'Runner' do flutter_install_all_ios_pods File.dirname(File.realpath(FILE))

target 'RunnerTests' do inherit! :search_paths end end

post_install do |installer| installer.pods_project.targets.each do |target| flutter_additional_ios_build_settings(target) end end

2. Update the [import to the right `integration_test` casing](https://github.com/flutter/plugins/blob/master/packages/integration_test/example/ios/RunnerTests/RunnerTests.m#L1) to 

import <integration_test/IntegrationTestIosTest.h>


3. Run the tests.

### Recommended fix
1. Convert the XCTest to XCUITests.  This would be a breaking change for existing tests, including the one's in Flutter's repo.
2.  In `+setUp`/`class func setUp` (not `-setUp` or `func setUp`) detect the popup and hit it when the app launches in the `INTEGRATION_TEST_IOS_RUNNER` macro
```swift
let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard")
let button = springboard.buttons['OK']
if button.exists {
  button.tap()
}

See cocoon for working example.

  1. Update the example app to a XCUITests target.

  2. Change the README to

    Create a UI Testing Bundle target (navigating File > New > Target... and set up the values)

jmagman commented 4 years ago

See https://github.com/flutter/flutter/issues/65207 and https://github.com/flutter/flutter/issues/60634 for context.

jmagman commented 4 years ago

\cc @dnfield based on CODEOWNERS. I would work on this myself but I'm about to be on vacation for a week, so I want to leave a breadcrumb in case iOS 14 gets released in the meantime.