search

Unidata / netcdf-c

Official GitHub repository for netCDF-C libraries and utilities.
BSD 3-Clause "New" or "Revised" License
513 stars 262 forks source link

'ncdap_tst_remote3' test failure in Linux #2242

Open abhibaruah opened 2 years ago

abhibaruah commented 2 years ago

NetCDF version: 4.8.1 HDF5 version: 1.10.8 OS: Debian 10

Recently, we changed the upstream dependency of netcdf from hdf5 1.8.12 to 1.10.8. After this, on building netcdf, the test 'ncdap_tst_remote3' fails with a 'Permission Denied' error message.

The test log \netcdf\Testing\Temporary\LastTest.log records the following:

179/199 Testing: ncdap_tst_remote3
179/199 Test: ncdap_tst_remote3
Command: "/usr/local/bin/bash" "-c" "export srcdir=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test;export TOPSRCDIR=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf;/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh "
Directory: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test
"ncdap_tst_remote3" start time: Mar 09 12:51 EST
Output:
----------------------------------------------------------
/usr/local/bin/bash: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh: Permission denied
<end of output>
Test time =   0.02 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 09 12:51 EST
"ncdap_tst_remote3" time elapsed: 00:00:00

Is this a known issue or a server issue? Or can anything be done on the software side to fix this test failure?

WardF commented 2 years ago

If you run ls -alh /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh, what do you see? This seems like a pretty straightforward linux filesystem permissions issue, and it doesn't appear to be reflecting anything specific to the release.

abhibaruah commented 2 years ago

Hello Ward, Thank you for looking into this. I agree that the error message is not much helpful. Here is the output that I get on executing the ls -alh command

-rw-r--r--+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

Thanks

WardF commented 2 years ago

Hi @abhibaruah it looks like the 'execute' bit is not set on this file. I'm not sure how that would have happened, although it might be something to do with the underlying filesystem, or if this is a network share. Try setting the executable bit with the following command:

$ chmod 755 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

After running this command, run ls -alh /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh again, and we should see that the executable bit is set.

abhibaruah commented 2 years ago

Hello Ward, After I executed the chmod command, I did see the permissions change.

-rwxr-xr-x+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh*

But, when I build netcdf again after that, the permissions on tst_remote3.sh revert back to what I showed initially, and I see the same 'Permission denied' error.

The ls -alh return this now:

-rw-r--r--+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

Meanwhile, I am also trying to see if anything in our makefile or build harness is messing up the permissions.

Thanks, Abhi

abhibaruah commented 2 years ago

Hello @WardF, I was able to resolve the 'Permission Denied' error. However now, the same test fails with the following log:

179/199 Testing: ncdap_tst_remote3
179/199 Test: ncdap_tst_remote3
Command: "/usr/local/bin/bash" "-c" "export srcdir=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test;export TOPSRCDIR=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf;/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh "
Directory: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test
"ncdap_tst_remote3" start time: Mar 11 13:02 EST
Output:
----------------------------------------------------------
XXXXXXXXXXXXXXXXXXXX
YYYYYYYYYYYYYYYYYYYY
*** Testing DAP to netCDF-3 translation using remote server 
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457, 
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
<end of output>
Test time =  11.60 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 11 13:02 EST
"ncdap_tst_remote3" time elapsed: 00:00:11
----------------------------------------------------------
DennisHeimbigner commented 2 years ago 
<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457,

The problem is the middle value differs in the last digit: 3 vs 4. This occasionally occurs depending on the platform.

abhibaruah commented 2 years ago

Thank you for taking a look Dennis. Is there a way to fix this, or can this be treated as an outlier and filtered for now?

DennisHeimbigner commented 2 years ago

If the data is being accesses with ncdump, then you might be able to use the -p flag.

DennisHeimbigner commented 2 years ago

Aince it is clear that there is no essential problem, I recommend that you build netcdf-c with the --disable-dap-remote-tests option.

abhibaruah commented 2 years ago

Thank you Dennis. I have filtered out the corresponding test by commenting out the line add_sh_test(ncdap tst_remote3) in ncdap_test/CMakeLists.txt

ArchangeGabriel commented 2 years ago

Just wanted to say that I’m seeing the same failure on Arch Linux (while it wasn’t showing before):

199/236 Testing: ncdap_tst_remote3
199/236 Test: ncdap_tst_remote3
Command: "/usr/bin/bash" "-c" "export srcdir=/build/netcdf-openmpi/src/netcdf-c-4.8.1/ncdap_test;export TOPSRCDIR=/build/netcdf-openmpi/src/netcdf-c-4.8.1;/build/netcdf-openmpi/src/build/ncdap_test/tst_remote3.sh "
Directory: /build/netcdf-openmpi/src/build/ncdap_test
"ncdap_tst_remote3" start time: Mar 20 22:45 UTC
Output:
----------------------------------------------------------
*** Testing DAP to netCDF-3 translation using remote server 
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457, 
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
<end of output>
Test time =  26.18 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 20 22:45 UTC
"ncdap_tst_remote3" time elapsed: 00:00:26
----------------------------------------------------------
Dave-Allured commented 2 years ago

The test.67 error is a duplicate of #2188.

serbinsh commented 2 years ago

I am not sure if this is the same or related but I am also running into a testing error on 4.8.1, but only the DAP test

I am compiling with HDF5 1.12.1 with gcc 8.5.0-4 (system gcc on RHEL 8.5)

All other tests passed.

FAIL: tst_remote3.sh
============================================================================
Testsuite summary for netCDF 4.8.1
============================================================================
# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ncdap_test/test-suite.log
Please report to support-netcdf@unidata.ucar.edu
============================================================================
make[5]: *** [Makefile:1090: test-suite.log] Error 1
make[5]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[4]: *** [Makefile:1198: check-TESTS] Error 2
make[4]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[3]: *** [Makefile:1346: check-am] Error 2
make[3]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[2]: *** [Makefile:977: check-recursive] Error 1
make[2]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[1]: *** [Makefile:1348: check] Error 2
make[1]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make: *** [Makefile:729: check-recursive] Error 1
=============================================
   netCDF 4.8.1: ncdap_test/test-suite.log
=============================================

# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_remote3
=================

*** Testing DAP to netCDF-3 translation using remote server
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457,
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457,
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
FAIL tst_remote3.sh (exit status: 1)
Dave-Allured commented 2 years ago

@serbinsh, yes, your error log clearly shows this is the same single-digit failure of test.67. This is now an expected failure because of a presumed behavior change on a remote test server. This is being analyzed in #2188.