Test case failures on rhel 7.6 ppc64le platform

[SantoshMagdum]

Hi All,

I had build the nginx binary on rhel 7.6 ppc64le (version 1.17.1.1rc0) from source code - https://github.com/openresty/openresty. Please note that, I had copied and used ppc64le compiled LuaJIT code while building openresty (nginx). Below command I used to compile the openresty -

./configure --with-cc-opt="-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC" --with-http_image_filter_module --with-http_dav_module --with-http_auth_request_module --with-poll_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module --with-http_iconv_module --with-http_drizzle_module --with-http_postgres_module --with-http_addition_module --add-module=/usr/openresty/openresty_test_modules/nginx-eval-module --add-module=/usr/openresty/openresty_test_modules/replace-filter-nginx-module

And then tried to execute the test cases for 'lua-resty-upstream-healthcheck' like below -

[root]# pwd /usr/openresty/openresty/openresty-1.17.1.1rc0/build/lua-resty-upstream-healthcheck-0.06 [root]# prove -r t

NOTE: The 'lua-resty-upstream-healthcheck' module version used was 0.06 which was downoaded with openresty bundle.

But I am getting below kind of repeated errors -

    [root  lua-resty-upstream-healthcheck-0.06]#
    [root  lua-resty-upstream-healthcheck-0.06]# prove -r t/
    t/sanity.t .. 1/99
    #   Failed test 'TEST 7: peers version upgrade (make up peers down) - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'warn(): healthcheck: peer 127.0.0.1:12355 is turned up after 2 success(es)
    # warn(): healthcheck: peer 127.0.0.1:12356 is turned up after 2 success(es)
    # '
    #     doesn't match '(?^:^upgrading peers version to 1
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12355 is turned up after 2 success\(es\)
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12356 is turned up after 2 success\(es\)
    # publishing peers version 2
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){2,4}$)'
    TEST 15: peers > concurrency - WARNING: killing the child process 12933 with force... at /usr/local/share/perl5/Test/Nginx/Util.pm line 609.
    t/sanity.t .. 8/99
    #   Failed test 'TEST 15: peers > concurrency - response_body_like - response is expected (Upstream foo.com Primary Peers 127.0.0.1:12354 up 127.0.0.1:12355 up 127.0.0.1:12356 up 127.0.0.1:12357 up 127.0.0.1:12358 up Backup Peers 127.0.0.1:12359 up)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1635.
    #                   'Upstream foo.com
    #     Primary Peers
    #         127.0.0.1:12354 up
    #         127.0.0.1:12355 up
    #         127.0.0.1:12356 up
    #         127.0.0.1:12357 up
    #         127.0.0.1:12358 up
    #     Backup Peers
    #         127.0.0.1:12359 up
    # '
    #     doesn't match '(?^s:Upstream foo.com
    #     Primary Peers
    #         127.0.0.1:12354 DOWN
    #         127.0.0.1:12355 \S+
    #         127.0.0.1:12356 \S+
    #         127.0.0.1:12357 \S+
    #         127.0.0.1:12358 \S+
    #     Backup Peers
    #         127.0.0.1:12359 \S+
    # )'
    t/sanity.t .. 10/99
    #   Failed test 'TEST 15: peers > concurrency - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'healthcheck: failed to receive status line from 127.0.0.1:12354
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12357 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12358 was checked to be not ok
    # healthcheck: failed to receive status line from 127\.0\.0\.1:12354
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12359 was checked to be not ok
    # $)'
    t/sanity.t .. 14/99
    #   Failed test 'TEST 9: concurrency == 2 (odd number of peers) - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   ''
    #     doesn't match '(?^:^(?:spawn a thread checking primary peers 0 to 2
    # check primary peers 3 to 4
    # check backup peer 0
    # ){4,6}$)'
    t/sanity.t .. 25/99
    #   Failed test 'TEST 3: health check (bad case), no listening port in a primary peer - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'warn(): healthcheck: peer 127.0.0.1:12355 is turned down after 2 failure(s)
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12355 is turned down after 2 failure\(s\)
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){2,4}$)'
    t/sanity.t .. 31/99
    #   Failed test 'TEST 5: health check (bad case), timed out - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'warn(): healthcheck: peer 127.0.0.1:12354 is turned down after 2 failure(s)
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12354 is turned down after 2 failure\(s\)
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){0,2}$)'

    #   Failed test 'TEST 8: peers version upgrade (make down peers up) - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'warn(): healthcheck: peer 127.0.0.1:12354 is turned down after 2 failure(s)
    # '
    #     doesn't match '(?^:^upgrading peers version to 1
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12354 is turned down after 2 failure\(s\)
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # publishing peers version 2
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){3,5}$)'
    t/sanity.t .. 43/99
    #   Failed test 'TEST 4: health check (bad case), bad status - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'healthcheck: bad status code from 127.0.0.1:12355: 404
    # healthcheck: bad status code from 127.0.0.1:12355: 404
    # warn(): healthcheck: peer 127.0.0.1:12355 is turned down after 2 failure(s)
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: bad status code from 127\.0\.0\.1:12355: 404
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: bad status code from 127\.0\.0\.1:12355: 404
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12355 is turned down after 2 failure\(s\)
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){1,4}$)'
    t/sanity.t .. 48/99
    #   Failed test 'TEST 11: health check (good case), status ignored by default - tcp_query ok'
    #   at /usr/local/share/perl5/Test/Nginx/Util.pm line 188.
    #          got: ''
    #     expected: 'GET /status HTTP/1.0
    # Host: localhost
    #
    # '
    t/sanity.t .. 60/99
    #   Failed test 'TEST 1: health check (good case), status ignored by default - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   ''
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){3,5}$)'

    #   Failed test 'TEST 6: health check (bad case), bad status, and then rise again - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'healthcheck: bad status code from 127.0.0.1:12355: 403
    # warn(): healthcheck: peer 127.0.0.1:12355 is turned down after 1 failure(s)
    # warn(): healthcheck: peer 127.0.0.1:12355 is turned up after 2 success(es)
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: bad status code from 127\.0\.0\.1:12355: 403
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12355 is turned down after 1 failure\(s\)
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # publishing peers version 1
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12355 is turned up after 2 success\(es\)
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # publishing peers version 2
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be ok
    # ){1,3}$)'
    t/sanity.t .. 71/99
    #   Failed test 'TEST 10: concurrency == 3 (odd number of peers) - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   ''
    #     doesn't match '(?^:^(?:spawn a thread checking primary peer 0
    # spawn a thread checking primary peer 1
    # check primary peer 2
    # check backup peer 0
    # ){4,6}$)'
    t/sanity.t .. 86/99
    #   Failed test 'TEST 2: health check (bad case), no listening port in the backup peer - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   'warn(): healthcheck: peer 127.0.0.1:12356 is turned down after 2 failure(s)
    # '
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be not ok
    # healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be not ok
    # warn\(\): healthcheck: peer 127\.0\.0\.1:12356 is turned down after 2 failure\(s\)
    # publishing peers version 1
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12355 was checked to be ok
    # healthcheck: peer 127\.0\.0\.1:12356 was checked to be not ok
    # ){2,4}$)'

    #   Failed test 'TEST 14: health check with ipv6 backend (good case), status ignored by default - response_body - response is expected (repeated req 0, req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1589.
    # @@ -3,6 +3,6 @@
    #          127.0.0.1:12354 up
    #          [::1]:12355 up
    #      Backup Peers
    # -        [0:0::1]:12356 up
    # +        [::1]:12356 up
    #  upstream addr: 127.0.0.1:12354
    #  upstream addr: [::1]:12355
    t/sanity.t .. 94/99
    #   Failed test 'TEST 14: health check with ipv6 backend (good case), status ignored by default - grep_error_log_out (req 0)'
    #   at /usr/local/share/perl5/Test/Nginx/Socket.pm line 1145.
    #                   ''
    #     doesn't match '(?^:^healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer \[::1\]:12355 was checked to be ok
    # healthcheck: peer \[0:0::1\]:12356 was checked to be ok
    # (?:healthcheck: peer 127\.0\.0\.1:12354 was checked to be ok
    # healthcheck: peer \[::1\]:12355 was checked to be ok
    # healthcheck: peer \[0:0::1\]:12356 was checked to be ok
    # ){3,7}$)'
    # Looks like you failed 15 tests of 99.
    t/sanity.t .. Dubious, test returned 15 (wstat 3840, 0xf00)
    Failed 15/99 subtests

    Test Summary Report
    -------------------
    t/sanity.t (Wstat: 3840 Tests: 99 Failed: 15)
      Failed tests:  3, 9-10, 16, 27, 33, 39, 45, 57, 60, 68
                    79, 88, 93-94
      Non-zero exit status: 15
    Files=1, Tests=99, 37 wallclock secs ( 0.04 usr  0.01 sys +  0.60 cusr  0.25 csys =  0.90 CPU)
    Result: FAIL
    [root  lua-resty-upstream-healthcheck-0.06]#

Please help suggest if I need to export any specific environment/setup any additional service or should try any compiler flag/somehow increase timeout value to make these test cases pass?

nginx version (compiled with libdrizzle 1.0 and radius, mariadb, postgresql services setup) -

# nginx -V
nginx version: openresty/1.17.1.1rc0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC' --add-module=../ngx_devel_kit-0.3.1rc1 --add-module=../iconv-nginx-module-0.14 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../drizzle-nginx-module-0.1.11 --add-module=../ngx_postgres-1.0 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.15 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.7 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --with-http_image_filter_module --with-http_dav_module --with-http_auth_request_module --with-poll_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module --with-http_addition_module --add-module=/usr/openresty/openresty_test_modules/nginx-eval-module --add-module=/usr/openresty/openresty_test_modules/replace-filter-nginx-module --with-stream --with-stream_ssl_preread_module

[SantoshMagdum]

Hi All, Please share any pointers you have on way to resolve the above failures. Thanks in advance.