search

DOI-USGS / ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
https://isis.astrogeology.usgs.gov
Other
200 stars 168 forks source link

ctest failures on MacOSX 10.15 #5034

Closed KrisBecker closed 1 year ago

KrisBecker commented 2 years ago

ISIS version(s) affected: dev

Description
There are a rather significant number of tests failing on my Mac after a successful build of the current dev branch. I am getting 115 (~5%) of the 2490 tests failing. Some of them look concerning.

I have updated both the ISISTESTDATA and ISISDATA (via rsync) data areas. I see three new files in ISISTESTDATA that look to be extraneous to the actual test data:

(base) zion2[1143]: grep Copied rclone_transfer.log 
2022/08/17 09:01:45 INFO  : test.txt: Copied (new)
2022/08/17 09:01:45 INFO  : print.prt: Copied (new)
2022/08/17 09:01:52 INFO  : ~?: Copied (new)

Here is the report generated from ctest:


95% tests passed, 115 tests failed out of 2490

Label Time Summary:
apollo          =   3.97 sec*proc (4 tests)
app             = 4448.84 sec*proc (768 tests)
cassini         =   1.12 sec*proc (5 tests)
chandrayaan1    =   0.38 sec*proc (1 test)
clementine      =   0.89 sec*proc (4 tests)
dawn            =   1.49 sec*proc (2 tests)
galileo         =   0.23 sec*proc (1 test)
hayabusa        =   0.85 sec*proc (2 tests)
hayabusa2       =   0.28 sec*proc (1 test)
isis            = 153.04 sec*proc (321 tests)
juno            =   0.27 sec*proc (1 test)
kaguya          =   0.91 sec*proc (2 tests)
lo              =   0.75 sec*proc (3 tests)
lro             =   0.97 sec*proc (3 tests)
mariner         =   0.22 sec*proc (1 test)
messenger       =   0.53 sec*proc (1 test)
mex             =   0.51 sec*proc (2 tests)
mgs             =   2.75 sec*proc (4 tests)
mro             =   1.94 sec*proc (6 tests)
newhorizons     =   1.14 sec*proc (4 tests)
odyssey         =   0.41 sec*proc (2 tests)
rosetta         =   1.42 sec*proc (2 tests)
tgo             =   0.28 sec*proc (1 test)
unit            = 174.79 sec*proc (375 tests)
viking          =   0.21 sec*proc (1 test)
voyager         =   0.26 sec*proc (1 test)

Total Test time (real) = 9399.13 sec

The following tests did not run:
    1732 - TempTestingFiles.FunctionalTestJitterfitDefault (Disabled)
    1763 - TempTestingFiles.UnitTestImageImporterTestJpeg (Disabled)

The following tests FAILED:
     58 - isis_unit_test_CameraStatistics (Failed)
     63 - isis_unit_test_Cube (Failed)
    109 - isis_unit_test_IdealCamera (Failed)
    128 - isis_unit_test_JP2Importer (Failed)
    164 - isis_unit_test_NaifDskShape (Failed)
    210 - isis_unit_test_ProcessGroundPolygons (Failed)
    308 - isis_unit_test_UniversalGroundMap (Failed)
    371 - cam2map_app_test_mpp (Failed)
    372 - cam2map_app_test_rngmap (Failed)
    373 - camtrim_app_test_case01 (Failed)
    374 - camtrim_app_test_case02 (Failed)
    375 - camtrim_app_test_lev2 (Failed)
    376 - camtrim_app_test_maptemplate (Failed)
    480 - footprintmerge_app_test_360Boundary (Failed)
    532 - isis2gml_app_test_case01 (Failed)
    533 - isis2gml_app_test_case02 (Failed)
    534 - isis2gml_app_test_lincsinc (Failed)
    601 - map2cam_app_test_case01 (Failed)
    602 - map2cam_app_test_case02 (Failed)
    661 - photrim_app_test_base (Failed)
    662 - photrim_app_test_emission (Failed)
    663 - photrim_app_test_incidence (Failed)
    664 - photrim_app_test_phase (Failed)
    666 - pixel2map_app_test_bandSelect (Failed)
    668 - pixel2map_app_test_errors (Failed)
    671 - pixel2map_app_test_mpp (Failed)
    672 - pixel2map_app_test_multipleInputCubes (Failed)
    715 - skymap_app_test_user (Failed)
    716 - skypt_app_test_default (Failed)
    717 - skypt_app_test_flat (Failed)
    718 - skyrange_app_test_default (Failed)
    734 - spicefit_app_test_default (Failed)
    810 - deltack_app_test_directOffTarget (Failed)
    858 - socetframesettings_app_test_viking (Failed)
    882 - cassini_unit_test_IssNACamera (Failed)
    883 - cassini_unit_test_IssWACamera (Failed)
    884 - cassini_unit_test_VimsCamera (Failed)
    902 - vimscal_app_test_irdark (Failed)
    903 - vimscal_app_test_nodark (Failed)
    904 - cassini_module_test_vims (Failed)
    906 - clementine_unit_test_HiresCamera (Failed)
    907 - clementine_unit_test_LwirCamera (Failed)
    908 - clementine_unit_test_NirCamera (Failed)
    919 - clementine_module_test_clementine (Failed)
    927 - galileo_unit_test_SsiCamera (Failed)
    950 - hayabusa_module_test_fullframe (Failed)
    951 - hayabusa_module_test_subframe (Failed)
    972 - lo_unit_test_LoCameraFiducialMap (Failed)
    973 - lo_unit_test_LoHighCamera (Failed)
    974 - lo_unit_test_LoMediumCamera (Failed)
    977 - lro_unit_test_LroNarrowAngleCamera (Failed)
    979 - lro_unit_test_MiniRF (Failed)
    983 - lronaccal_app_test_nacl-full (Failed)
    984 - lronaccal_app_test_nacl-summed (Failed)
    985 - lronaccal_app_test_nacr-full (Failed)
    986 - lronaccal_app_test_nacr-summed (Failed)
    1001 - mariner_unit_test_Mariner10Camera (Failed)
    1035 - mdisedrinfo_app_test_kernelchk (Failed)
    1040 - mex_unit_test_HrscCamera (Failed)
    1042 - mex_module_test_hrsc (Failed)
    1044 - mgs_unit_test_MocNarrowAngleCamera (Failed)
    1057 - mgs_module_test_moc (Failed)
    1058 - mro_unit_test_CTXCamera (Failed)
    1062 - mro_unit_test_HiriseCamera (Failed)
    1063 - mro_unit_test_MarciCamera (Failed)
    1076 - hideal2pds_app_test_notJitterCorrected (Failed)
    1108 - mro_module_test_hirise (Failed)
    1109 - near_module_test_msi (Failed)
    1115 - odyssey_unit_test_ThemisIrCamera (Failed)
    1116 - odyssey_unit_test_ThemisVisCamera (Failed)
    1128 - odyssey_module_test_themis (Failed)
    1132 - osirisrex_module_test_mapcam (Failed)
    1146 - viking_unit_test_VikingCamera (Failed)
    1156 - voyager_unit_test_VoyagerCamera (Failed)
    1421 - DefaultCube.FunctionalTestCaminfoCsv (Timeout)
    1422 - DefaultCube.FunctionalTestCaminfoDefault (Failed)
    1425 - DefaultCube.FunctionalTestCaminfoCamStatsTable (Failed)
    1436 - DefaultCube.FunctionalTestCamrangeMeta (Failed)
    1437 - DefaultCube.FunctionalTestCamrangeUniversalGround (Failed)
    1438 - DefaultCube.FunctionalTestCamrangeLatitude (Failed)
    1439 - DefaultCube.FunctionalTestCamrangeCardinals (Failed)
    1441 - DefaultCube.FunctionalTestCamstatsDefaultParameters (Failed)
    1443 - DefaultCube.FunctionalTestCamstatsFlat (Failed)
    1497 - DefaultCube.FunctionalTestNoprojDefault (Failed)
    1498 - DefaultCube.FunctionalTestNoprojExpand (Failed)
    1499 - DefaultCube.FunctionalTestNoprojFromInput (Failed)
    1500 - DefaultCube.FunctionalTestNoprojFromUser (Failed)
    1501 - DefaultCube.FunctionalTestNoprojSpecs (Failed)
    1502 - DefaultCube.FunctionalTestPhocubeDefault (Failed)
    1503 - DefaultCube.FunctionalTestPhocubeAllBands (Failed)
    1524 - DefaultCube.UnitTestImagePolygonBoundary (Failed)
    1525 - DefaultCube.UnitTestImagePolygonOutlier (Failed)
    1528 - LineScannerCube.FunctionalTestCamdevDefault (Failed)
    1529 - LineScannerCube.FunctionalTestNoprojLineScanner (Failed)
    1759 - TempTestingFiles.FunctionalTestStd2isisJp2 (Failed)
    1765 - TempTestingFiles.UnitTestImageImporterStd2IsisJp2 (Failed)
    1767 - TempTestingFiles.UnitTestImagePolygonCross (Failed)
    1863 - MroHiriseCube.FunctionalTestHicolormosDefault (Failed)
    1879 - MroHiriseCube.FunctionalTestHimosDefault (Failed)
    1912 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KGray (Failed)
    1913 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KU16 (Failed)
    1914 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KS16 (Failed)
    1924 - SmallARGBCube.FunctionalTestsIsis2StdJpeg2KRGB (Failed)
    1925 - SmallARGBCube.FunctionalTestsIsis2StdJpeg2KARGB (Failed)
    1940 - ApolloNetwork.FunctionalTestJigsawApollo (Failed)
    1951 - LidarNetwork.FunctionalTestJigsawLidar (Failed)
    1995 - LidarObservationPair.FunctionalTestLrolola2isisTwoImage (Failed)
    1996 - LidarObservationPair.FunctionalTestLrolola2isisMultipleCsv (Failed)
    2282 - TgoCassisModuleKernels.TgoCassisSingleFrameletProjection (Failed)
    2283 - TgoCassisModuleKernels.TgoCassisTestColorMosaic (Failed)
    2284 - TgoCassisModuleKernels.TgoCassisMapProjectedReingested (Failed)
    2285 - TgoCassisModuleKernels.TgoCassisSingleColorMosaicReingest (Failed)
    2286 - TgoCassisModuleKernels.TgoCassisTestProjSingleStitchedFrame (Failed)
    2287 - TgoCassisModuleTests.TgoCassisIngestReingest (Failed)
    2288 - TgoCassisModuleTests.TgoCassisUncontrolledSingleColorMosaic (Failed)
Errors while running CTest

I checked the nightly dev builds on Jenkins and I am seeing 40 errors in the last Ubuntu build, but no Mac build data to compare. In fact, I could not find any Mac builds since at least back to January 17, 2022. Is there no nightly dev Mac (or other supported systems) test data to compare with?

Just a few examples of the test failures:

58/2492 Testing: isis_unit_test_CameraStatistics
<snip>
------------------ DIFFERENCES ------------------
2,3c2,96
< **ERROR** Unable to initialize camera model in Camera Factory.
< **ERROR** The camera is requesting spice data [BODY_FRAME_CODE] that was not attached, please re-run spiceinit.
---
>
> User Parameters:
>   Linc = 1
>   Sinc = 1
>
63/2492 Testing: isis_unit_test_Cube
<snip>
------------------ DIFFERENCES ------------------
279c279,281
< **PROGRAMMER ERROR** Sorry you can't do a SetMethod after the cube is opened.
---
> **I/O ERROR** Failed to open [...] with read/write access.
> **ERROR** Cannot create the cube [shouldntExist.cub] with a pixel type set to None.
> **PROGRAMMER ERROR** Cannot add a group to the label of cube [IsisCube_06.cub] because it is opened read-only.

There appear to be a lot of these errors, particularly related to cameras/projections:

371/2492 Testing: cam2map_app_test_mpp
<snip>
Output:
----------------------------------------------------------
SOURCE FOLDER $PROJECT_ROOT/ISIS3/isis/src/base/apps/cam2map/tsts/mpp
libc++abi.dylib: terminating with uncaught exception of type std::invalid_argument: The cache can only be minimized if velocity is provided.
bash: line 5: 49466 Abort trap: 6           $PROJECT_ROOT/ISIS3/build/bin/cam2map -preference=$PROJECT_ROOT/ISIS3/build/TestPreferences from= /opt/isis4/testData/isis/src/base/apps/cam2map/tsts/mpp/input/ab102401.cub to=output/cam2mapTruth.cub map=/opt/isis4/testData/isis/src/base/apps/cam2map/tsts/mpp/input/sinusoidal.map pixres=mpp defaultrange=camera resolution=1000 > /dev/null
make[2]: *** [Makefile:9: commands] Error 134

How to reproduce

Possible Solution

Additional context

acpaquette commented 2 years ago

@KrisBecker Can you run conda list ale and give the version of ALE that you built with? This could easily be from a recent ALE release that was not properly tested against ISIS.

And yes, due to changes internal to the USGS getting proper nightly builds for mac has been delayed but should be remedied soon.

KrisBecker commented 2 years ago

Here is the complete package list:

(CometsAug17) zion2[1031]: conda list
# packages in environment at /Users/kbecker/miniconda3/envs/CometsAug17:
#
# Name                    Version                   Build  Channel
ale                       0.8.6            py39hf018cea_0    conda-forge
aom                       3.3.0                h96cf925_1    conda-forge
armadillo                 11.2.3               ha02a57f_0    conda-forge
arpack                    3.7.0                hefb7bc6_2    conda-forge
atk-1.0                   2.36.0               he69c4ee_4    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
blas                      2.116                  openblas    conda-forge
blas-devel                3.9.0           16_osx64_openblas    conda-forge
boost                     1.72.0           py39hb64e6f8_1    conda-forge
boost-cpp                 1.72.0               hf3dc895_6    conda-forge
bullet                    2.86.1                        0    usgs-astrogeology
bz2file                   0.98                       py_0    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.18.1               h0d85af4_0    conda-forge
ca-certificates           2022.6.15            h033912b_0    conda-forge
cairo                     1.16.0            h9e0e54b_1010    conda-forge
cmake                     3.24.0               h9c7736b_0    conda-forge
csm                       3.0.3.3                       0    conda-forge
cspice                    67                   hac89ed1_2    conda-forge
curl                      7.83.1               h23f1065_0    conda-forge
cyrus-sasl                2.1.27               hbdf3212_5    conda-forge
doxygen                   1.9.3                h51faa4e_1    conda-forge
eigen                     3.3.9                h926bf3e_1    conda-forge
embree                    2.17.7               h694c41f_2    conda-forge
expat                     2.4.8                h96cf925_0    conda-forge
ffmpeg                    4.4.1                h2579ba5_2    conda-forge
flann                     1.9.1             h30321d8_1010    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               h676cef8_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               h3f81eb7_0    conda-forge
fribidi                   1.0.10               hbcb3906_0    conda-forge
gdk-pixbuf                2.42.8               hb161b9c_0    conda-forge
geos                      3.7.3                h4a8c4bd_0    conda-forge
geotiff                   1.7.1                ha1a2aeb_3    conda-forge
gettext                   0.19.8.1          hd1a6beb_1008    conda-forge
giflib                    5.2.1                hbcb3906_2    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
gnutls                    3.6.13               h756fd2b_1    conda-forge
graphite2                 1.3.13            h2e338ed_1001    conda-forge
graphviz                  3.0.0                h6f567cf_1    conda-forge
gsl                       2.7                  h93259b0_0    conda-forge
gtk2                      2.24.33              h7c1209e_2    conda-forge
gts                       0.7.6                hccb3bdf_2    conda-forge
harfbuzz                  4.2.0                h48644e2_0    conda-forge
hdf5                      1.12.1          nompi_h0aa1fa2_104    conda-forge
icu                       69.1                 he49afe7_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
inja                      3.3.0                he49afe7_0    conda-forge
jama                      125                  h694c41f_0    conda-forge
jasper                    2.0.33               h013e400_0    conda-forge
jpeg                      9e                   hac89ed1_2    conda-forge
kakadu                    1                             0    usgs-astrogeology
krb5                      1.19.3               hb98e516_0    conda-forge
lame                      3.100             h35c211d_1001    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libblas                   3.9.0           16_osx64_openblas    conda-forge
libcblas                  3.9.0           16_osx64_openblas    conda-forge
libclang                  13.0.1          default_he082bbe_0    conda-forge
libcurl                   7.83.1               h23f1065_0    conda-forge
libcxx                    14.0.6               hce7ea42_0    conda-forge
libdeflate                1.13                 h775f41a_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libevent                  2.1.10               h7d65743_4    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgd                     2.3.3                h02d8a21_2    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_23    conda-forge
libgfortran5              9.3.0               h6c81a4c_23    conda-forge
libglib                   2.72.1               hfbcb929_0    conda-forge
libiconv                  1.16                 haf1e3a3_0    conda-forge
liblapack                 3.9.0           16_osx64_openblas    conda-forge
liblapacke                3.9.0           16_osx64_openblas    conda-forge
libllvm13                 13.0.1               h64f94b2_2    conda-forge
libnghttp2                1.47.0               h5aae05b_1    conda-forge
libntlm                   1.4               h35c211d_1002    conda-forge
libopenblas               0.3.21          openmp_h1027af7_0    conda-forge
libopencv                 4.5.5            py39hc2bf5a6_7    conda-forge
libpng                    1.6.37               h5481273_4    conda-forge
libpq                     14.5                 h2ca9b15_0    conda-forge
libprotobuf               3.19.4               hcf210ce_0    conda-forge
librsvg                   2.52.5               h5e5d7d9_3    conda-forge
libsqlite                 3.39.2               h5a3d3bf_1    conda-forge
libssh2                   1.10.0               h47af595_3    conda-forge
libtiff                   4.4.0                h5e0c7b4_3    conda-forge
libtool                   2.4.6             he49afe7_1008    conda-forge
libuv                     1.44.2               hac89ed1_0    conda-forge
libvpx                    1.11.0               he49afe7_3    conda-forge
libwebp                   1.2.4                hfa4350a_0    conda-forge
libwebp-base              1.2.4                h775f41a_0    conda-forge
libxcb                    1.13              h0d85af4_1004    conda-forge
libxml2                   2.9.12               h7e28ab6_1    conda-forge
libzlib                   1.2.12               hfe4f2af_2    conda-forge
llvm-openmp               14.0.4               ha654fa7_0    conda-forge
lz4-c                     1.9.3                he49afe7_1    conda-forge
macports-legacy-support   1.0.6                h5eb16cf_0    conda-forge
make                      4.3                  h22f3db7_1    conda-forge
mesalib                   21.2.5               h2df1e00_3    conda-forge
metis                     5.1.0             h2e338ed_1006    conda-forge
mpfr                      4.1.0                h0f52abe_1    conda-forge
mysql                     8.0.28               h88f4db0_2    conda-forge
mysql-client              8.0.28               h7ddd48c_2    conda-forge
mysql-common              8.0.28               hdd8d184_2    conda-forge
mysql-connector-c         6.1.6                hfc679d8_1    conda-forge
mysql-devel               8.0.28               hdd8d184_2    conda-forge
mysql-libs                8.0.28               h353f102_2    conda-forge
mysql-server              8.0.28               h6edde1b_2    conda-forge
nanoflann                 1.3.2                h694c41f_0    conda-forge
ncurses                   6.3                  h96cf925_1    conda-forge
nettle                    3.6                  hedd7734_0    conda-forge
networkx                  2.8.5              pyhd8ed1ab_0    conda-forge
ninja                     1.7.2                         0    conda-forge
nlohmann_json             3.10.5               hb486fe8_1    conda-forge
nn                        1.86.0            h1de35cc_2003    conda-forge
nspr                      4.32                 hcd9eead_1    conda-forge
nss                       3.78                 ha8197d3_0    conda-forge
numpy                     1.23.2           py39h62c883e_0    conda-forge
openblas                  0.3.21          openmp_hf6976e5_0    conda-forge
opencv                    4.5.5            py39h6e9494a_7    conda-forge
openh264                  2.1.1                hfd3ada9_0    conda-forge
openssl                   3.0.5                hb81d4ab_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pango                     1.50.7               hc4a7b6d_0    conda-forge
pcl                       1.11.1               h7984e4d_1    conda-forge
pcre                      8.45                 he49afe7_0    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               hbcb3906_0    conda-forge
pluggy                    1.0.0            py39h6e9494a_3    conda-forge
proj                      9.0.1                h05f0992_1    conda-forge
protobuf                  3.19.4           py39h9fcab8e_0    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
pvl                       1.3.2              pyhd8ed1ab_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
py-opencv                 4.5.5            py39h71a6800_7    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pytest                    7.1.2            py39h6e9494a_0    conda-forge
python                    3.9.13          hf8d34f4_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.2.1           pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py39h63b48b0_4    conda-forge
qhull                     2020.2               h940c156_2    conda-forge
qt                        5.12.9               h2a607e2_5    conda-forge
qwt                       6.2.0                h0f3c874_2    conda-forge
rclone                    1.59.1               h1762f63_0    conda-forge
readline                  8.1.2                h3899abd_0    conda-forge
rhash                     1.4.3                hac89ed1_0    conda-forge
scipy                     1.9.0            py39h29d19b3_0    conda-forge
setuptools                65.0.2           py39h6e9494a_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
spiceypy                  5.1.1              pyhd8ed1ab_0    conda-forge
sqlite                    3.39.2               hd9f0692_1    conda-forge
suitesparse               5.10.1               h7aff33d_1    conda-forge
superlu                   5.2.2                h1f0f902_0    conda-forge
svt-av1                   0.9.1                h96cf925_0    conda-forge
swig                      4.0.2                hce5123c_2    conda-forge
tbb                       2021.5.0             hbb4e6a2_1    conda-forge
texlive-core              20210325             h03edc0b_2    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
tnt                       126                  h694c41f_1    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022c                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
x264                      1!161.3030           h0d85af4_1    conda-forge
x265                      3.5                  hbb4e6a2_3    conda-forge
xalan-c                   1.11                 h4a8c4bd_0    conda-forge
xerces-c                  3.2.3                h6564042_4    conda-forge
xorg-damageproto          1.2.1             h0d85af4_1002    conda-forge
xorg-fixesproto           5.0               h0d85af4_1002    conda-forge
xorg-glproto              1.4.17            h0d85af4_1002    conda-forge
xorg-inputproto           2.3.2             h35c211d_1002    conda-forge
xorg-kbproto              1.0.7             h35c211d_1002    conda-forge
xorg-libice               1.0.10               h0d85af4_0    conda-forge
xorg-libsm                1.2.3             h0d85af4_1000    conda-forge
xorg-libx11               1.7.2                h0d85af4_0    conda-forge
xorg-libxau               1.0.9                h35c211d_0    conda-forge
xorg-libxdamage           1.1.5                h0d85af4_1    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xorg-libxext              1.3.4                h0d85af4_1    conda-forge
xorg-libxfixes            5.0.3             h0d85af4_1004    conda-forge
xorg-libxi                1.7.10               h0d85af4_0    conda-forge
xorg-libxrandr            1.5.2                h0d85af4_1    conda-forge
xorg-libxrender           0.9.10            h0d85af4_1003    conda-forge
xorg-randrproto           1.5.0             h0d85af4_1001    conda-forge
xorg-renderproto          0.11.1            h0d85af4_1002    conda-forge
xorg-util-macros          1.19.3               h35c211d_0    conda-forge
xorg-xextproto            7.3.0             h35c211d_1002    conda-forge
xorg-xf86vidmodeproto     2.3.1             h0d85af4_1002    conda-forge
xorg-xproto               7.0.31            h35c211d_1007    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
zlib                      1.2.12               hfe4f2af_2    conda-forge
zstd                      1.5.2                hb844be6_4    conda-forge
Kelvinrr commented 2 years ago

@KrisBecker Yeah, i've been aware of these recently but have not had the time to really look into them. We might not be able to really look at these until the next support sprint. Some might be data related but the missing keyword stuff doesn't make sense in light of recent changes.

These are also on Linux so they're not necessarily mac specific. New AWS Jenkins is being setup which will replace the current instance with low availability so hopefully we can get over some of out CI nightmares sooner than later.

KrisBecker commented 2 years ago

This commit on August 3 seems to be the culprit for, at least, the camera (like ctest no. 58) and potentially the projection errors.

What concerns me greatly is it seems to have severely broken backward compatibility of Cameras (and projections). (Is this API breaking and/or a bug?)

Another concern is that these errors are showing up in your Jenkins nightly dev build tests, but not in the final report. See the most recent console logs of Ubuntu build no. 1052. If you scroll to the unit tests, you will find the same unit test failures I am seeing, but the final ctest report at the end ignores/does not list them.

I can also trace ctest no. 164 to a recent NAIF DSK commit, which has a typo causing inconsistency in unitTest.cpp and the truth file.

jlaura commented 2 years ago

To help address the is this API breaking question, here is the definition of what is consider in scope for the API.

KrisBecker commented 2 years ago

FYI, here is the ctest results for 7.1.0_RC that I installed and built on July 29 using the same ISISTESTDATA and ISISDATA:

98% tests passed, 38 tests failed out of 2490

Label Time Summary:
apollo          =   3.78 sec*proc (4 tests)
app             = 4803.94 sec*proc (768 tests)
cassini         =   3.69 sec*proc (5 tests)
chandrayaan1    =   0.31 sec*proc (1 test)
clementine      =   0.81 sec*proc (4 tests)
dawn            =   0.74 sec*proc (2 tests)
galileo         =   0.30 sec*proc (1 test)
hayabusa        =   0.85 sec*proc (2 tests)
hayabusa2       =   0.37 sec*proc (1 test)
isis            = 163.34 sec*proc (321 tests)
juno            =   0.31 sec*proc (1 test)
kaguya          =   0.94 sec*proc (2 tests)
lo              =   0.88 sec*proc (3 tests)
lro             =   1.22 sec*proc (3 tests)
mariner         =   1.38 sec*proc (1 test)
messenger       =   0.50 sec*proc (1 test)
mex             =   0.70 sec*proc (2 tests)
mgs             =   2.46 sec*proc (4 tests)
mro             =   3.41 sec*proc (6 tests)
newhorizons     =   1.14 sec*proc (4 tests)
odyssey         =   0.53 sec*proc (2 tests)
rosetta         =   1.36 sec*proc (2 tests)
tgo             =   0.33 sec*proc (1 test)
unit            = 190.38 sec*proc (375 tests)
viking          =   0.53 sec*proc (1 test)
voyager         =   0.48 sec*proc (1 test)

Total Test time (real) = 9842.91 sec

The following tests did not run:
    1746 - TempTestingFiles.FunctionalTestJitterfitDefault (Disabled)
    1777 - TempTestingFiles.UnitTestImageImporterTestJpeg (Disabled)

The following tests FAILED:
     63 - isis_unit_test_Cube (Failed)
    128 - isis_unit_test_JP2Importer (Failed)
    164 - isis_unit_test_NaifDskShape (Failed)
    810 - deltack_app_test_directOffTarget (Failed)
    904 - cassini_module_test_vims (Failed)
    919 - clementine_module_test_clementine (Failed)
    950 - hayabusa_module_test_fullframe (Failed)
    951 - hayabusa_module_test_subframe (Failed)
    983 - lronaccal_app_test_nacl-full (Failed)
    984 - lronaccal_app_test_nacl-summed (Failed)
    985 - lronaccal_app_test_nacr-full (Failed)
    986 - lronaccal_app_test_nacr-summed (Failed)
    1042 - mex_module_test_hrsc (Failed)
    1057 - mgs_module_test_moc (Failed)
    1108 - mro_module_test_hirise (Failed)
    1109 - near_module_test_msi (Failed)
    1128 - odyssey_module_test_themis (Failed)
    1132 - osirisrex_module_test_mapcam (Failed)
    1435 - DefaultCube.FunctionalTestCaminfoCsv (Timeout)
    1773 - TempTestingFiles.FunctionalTestStd2isisJp2 (Failed)
    1779 - TempTestingFiles.UnitTestImageImporterStd2IsisJp2 (Failed)
    1808 - CombineNetworks.FunctionalTestCnetcombineptList (Failed)
    1822 - CSMPluginFixture.CSMInitDefault (Failed)
    1893 - MroHiriseCube.FunctionalTestHimosDefault (Failed)
    1926 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KGray (Failed)
    1927 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KU16 (Failed)
    1928 - IsisTruthCube.FunctionalTestsIsis2StdJpeg2KS16 (Failed)
    1938 - SmallARGBCube.FunctionalTestsIsis2StdJpeg2KRGB (Failed)
    1939 - SmallARGBCube.FunctionalTestsIsis2StdJpeg2KARGB (Failed)
    2009 - LidarObservationPair.FunctionalTestLrolola2isisTwoImage (Failed)
    2010 - LidarObservationPair.FunctionalTestLrolola2isisMultipleCsv (Failed)
    2282 - TgoCassisModuleKernels.TgoCassisSingleFrameletProjection (Failed)
    2283 - TgoCassisModuleKernels.TgoCassisTestColorMosaic (Failed)
    2284 - TgoCassisModuleKernels.TgoCassisMapProjectedReingested (Failed)
    2285 - TgoCassisModuleKernels.TgoCassisSingleColorMosaicReingest (Failed)
    2286 - TgoCassisModuleKernels.TgoCassisTestProjSingleStitchedFrame (Failed)
    2287 - TgoCassisModuleTests.TgoCassisIngestReingest (Failed)
    2288 - TgoCassisModuleTests.TgoCassisUncontrolledSingleColorMosaic (Failed)
Errors while running CTest
acpaquette commented 2 years ago

@KrisBecker I have a fix for the failures that are related to the BODY_FRAME_CODE, but the other change that impacted many tests was changing the precision that we read doubles in from json files. Many of our gtests are showing a change in values of about e-6 which makes sense as the original precision of doubles being read in from json was a precision of 6.

The original behavior was likely incorrect and has been change to 16 digit precision in #5021 and further changed in #5030. This change will likely require that many of our tests will require the truth values to be updated