Test using SCNetworkReachability fails under xctool

[ewanmellor]

I have a test that first checks if gmail.com is reachable on the network, using SCNetworkReachability* from SystemConfiguration. This fails under xctool, but passes under xcodebuild.

git clone https://github.com/tipbit/tidbits
git clone https://github.com/tipbit/CocoaLumberjack
git clone https://github.com/facebook/xctool
cd tidbits

#
# xcodebuild all tests pass
#
xcodebuild  -scheme Tidbits -sdk iphonesimulator8.1 -configuration Debug -destination "platform=iOS Simulator,name=iPhone 6,OS=8.1" test

#
# xctool fails in 4 tests, 1 of which is in DNSQueryTests (the rest are Issue #450).
#
../xctool/xctool.sh  -scheme Tidbits -sdk iphonesimulator8.1 -configuration Debug -destination "platform=iOS Simulator,name=iPhone 6,OS=8.1" test

The logging is below. Note the "registration failed" message, and that the test took 35 seconds (a timeout, presumably). Under xcodebuild this test passes in 0.045 seconds.

  run-test TidbitsTests.xctest (8.1, iPhone 6, logic-test)
      -[DNSQueryTests testMXQuery]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
nwi_state: registration failed (1000000)
2014-11-22 12:53:36.555 xctest[99207:14288985] -[DNSQueryTests testMXQuery]: Skipping test; gmail.com not reachable.
━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    ✓ -[DNSQueryTests testMXQuery] (35007 ms)

$ xcodebuild -version
Xcode 6.1
Build version 6A1052d
$ (cd tidbits ; git reflog HEAD)
78e13d9 HEAD@{0}: clone: from https://github.com/tipbit/tidbits
$ (cd xctool ; git reflog HEAD)
092c6a5 HEAD@{0}: clone: from https://github.com/facebook/xctool

[ewanmellor]

This may be the same issue as #367, but since that issue is so old, and on Xcode 5.1, I figured opening a new issue would be best.

[ewanmellor]

See #450 for the same test suite, but discussing the other failures.

[LegNeato]

Thanks for the repro! Will try to take a look in the next week.

[foscomputerservices]

I'm having similar problems using SCNetworkReachability. When the tests start to run, I see the following output:

Dec 17 22:01:46 david.foscomputerservices.com xctest[79852] <Error>: DNS configuration server not available
Dec 17 22:01:46 david.foscomputerservices.com xctest[79852] <Error>: com.apple.SystemConfiguration.DNSConfiguration_sim: server not available nwi_state: registration failed (1000000)
Dec 17 22:02:21 david.foscomputerservices.com xctest[79852] <Error>: com.apple.SystemConfiguration.NetworkInformation_sim: server not available
Dec 17 22:02:21 david.foscomputerservices.com xctest[79852] <Error>: Network information server not available
Dec 17 22:02:21 david.foscomputerservices.com xctest[79852] <Debug>: Reachability Flag Status: -- ------- networkStatusForFlags

The output of the Reachability Flag Status is mine, but the rest are from somewhere not in my code (iOS internals maybe?).

I have no problems running these in XCode or via xcodebuild.

[foscomputerservices]

So I did a bit more digging on this and I verified that the above error output is coming from the call to SCNetworkReachabilityGetFlags().

So, I decided to just turn off the reachability code and see what would happen then. It seems that I'm simply just not able to access the network at all when running under xctool:

2014-12-19 17:26:02.344 xctest[1477:9076207] NSURLConnection/CFURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9807)
Dec 19 17:26:02 david.foscomputerservices.com xctest[1477] <Error>:     FOSWebService ERROR -- Message: Error Domain=NSURLErrorDomain Code=-1202 "The certificate for this server is invalid. You might be connecting to a server that is pretending to be “api.parse.com” which could put your confidential information at risk." UserInfo=0x7b02f280 {NSLocalizedDescription=The certificate for this server is invalid. You might be connecting to a server that is pretending to be “api.parse.com” which could put your confidential information at risk., NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFStreamErrorCodeKey=-9807, NSErrorFailingURLStringKey=https://api.parse.com/1/events/AppOpened, _kCFStreamErrorDomainKey=3, NSURLErrorFailingURLPeerTrustErrorKey=<SecTrustRef: 0x7b03bc70>, NSUnderlyingError=0x7b4dfa00 "The certificate for this server is invalid. You might be connecting to a server that is pretending to be “api.parse.com” which could put your confidential information at risk.", NSErrorFailingURLKey=https://api.parse.com/1/events/AppOpened}

Again, I have no problems running these tests in Xcode or xcodebuild.

[yoiang]

Likewise, I'm having similar issues with my tests that include network activity. Any clarity into where the issue would be, perhaps we can solve this ourselves?

[ExtremeMan]

This is fixed since we switched to simctl to run logic tests. Your test isn't failing since commit 6b59d2005b5edd24582aceb2601b698dae980736.