One Failing Integration Test

[tiagomartinho]

Expected behavior

All tests in the integration Test Suite to pass.

Actual behavior

1 Failing integration test:

test_16_tcp_client_ip.sh

Logs:

Running test 'test_16_tcp_client_ip.sh'... FAILURE (0)
--- OUTPUT BEGIN ---
+ set -o pipefail
+ test=/Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
+ tmp=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t
+ root=/Users/martinho/Development/swift-nio/IntegrationTests/..
+++ dirname /Users/martinho/Development/swift-nio/IntegrationTests/run-single-test.sh
++ cd /Users/martinho/Development/swift-nio/IntegrationTests
++ pwd
+ here=/Users/martinho/Development/swift-nio/IntegrationTests
+ source /Users/martinho/Development/swift-nio/IntegrationTests/test_functions.sh
+ source /Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
++ source defines.sh
+++ create_token
+++ mktemp /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_XXXXXX
++ token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ start_server /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi tcp
++ local token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ local type=--uds
++ local port=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/port.sock
++ local tmp_server_pid
++ local tok_type=--unix-socket
++ local curl_port=80
++ maybe_host=
++ maybe_nio_host=
++ [[ tcp == \t\c\p ]]
++ type=
++ port=0
++ tok_type=
++ maybe_host=localhost
+++ host -4 -t a localhost
+++ tr ' ' '\n'
+++ tail -1
++ maybe_nio_host='3(NXDOMAIN)'
--- OUTPUT  END  ---

Steps to reproduce

1. Run: swift build
2. Run: swift test
3. Run: scripts/integration_tests.sh

SwiftNIO version/commit hash

1.0.0 / 5ff6d9ba7241b1f4d8121d84531bf87c901d930b

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2) Target: x86_64-apple-macosx10.9 Darwin Tiagos-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

[tomerd]

@tiagomartinho the key here is maybe_nio_host='3(NXDOMAIN)' which means host was not able to correctly retrieve the host info which sometimes happens with VPN or proxies, i'll see if we can fail more clearly here

/cc @weissi

fwiw, all pull requests go through ci that runs these tests

[tiagomartinho]

@tomerd I'm not currently using a VPN or proxy, but maybe is a VPN configuration I have? Even if it's not currently active?

I also tried to run the integration test suite from Docker but I got: FAILURE (oks: 3, failures: 14) Is this expected?

Moreover I had 4 other questions:

• There is a particular reason not to use XCTest for the integration tests?
• Why the integration tests are not invoked with the command swift test?
• Why are the tests 12, 13, and 14 skipped?
• I didn't found the CI documented in the project, is it public or private?

[tomerd]

I also tried to run the integration test suite from Docker but I got:

how did you try to run it? in any case the work i am doing towards #83 should help make this run in docker correctly

Moreover I had 4 other questions:

• There is a particular reason not to use XCTest for the integration tests?
• Why the integration tests are not invoked with the command swift test?
• Why are the tests 12, 13, and 14 skipped?
• I didn't found the CI documented in the project, is it public or private?

i'll let @weissi answer 1-3. our ci is private atm

[weissi]

@tiagomartinho Sorry for the late reply, I was on holiday the last week:

There is a particular reason not to use XCTest for the integration tests?

Yes: These tests are end-to-end tests so they make sure everything works in an actual binary without mocking anything. This can't be done using XCTest. That said, we obviously have XCTest tests too. Some of those tests also test assertions (ie. make sure the program crashes) or compile binaries in a special mode. Also they for example test that we're compatible with a 3rd party HTTP client like cURL.

Why the integration tests are not invoked with the command swift test?

Because they're slower (they're not unit tests) and I'd also argue it'd be odd to shell out to an external program through swift test

Why are the tests 12, 13, and 14 skipped?

• SKIP_test_13_http_pipelining.sh: fixed with #62
• SKIP_test_12_headers_too_large.sh & SKIP_test_14_strict_mode_assertion.sh: because I didn't get to them yet, apologies, I;ll file an issue against me to do that.

[weissi]

filed as #132

[Lukasa]

With #132 present I think we can close this.