search

isce-framework / isce3

InSAR Scientific Computing Environment ***NOTICE - This repository is currently a mirror of an internally hosted project and is in the process of migrating the development to this public repository. In near future we will accept external pull requests.***
Other
124 stars 32 forks source link

isce3 minor build/ test questions - v 0.4 #5

Open piyushrpt opened 3 years ago

piyushrpt commented 3 years ago

Thanks for the new release. It pretty much built out of the box except for some minor issues.

  1. There appears to be some discrepancy in the way isce3 and pyre discover python versions to use. I eventually had to use both these settings: -DPython_EXECUTABLE=path-to-python3 -DPython3_EXECUTABLE=path-to-python3 to get everything aligned.

  2. Some the tests fail with ctest possibly because they are trying to reach out to a s3 bucket to get data. Is there a flag / option to disable these tests.

  3. yamale is an undocumented dependency. Would it be useful to add this test at cmake stage or have it documented.

rtburns-jpl commented 3 years ago

Regarding your first point, pyre is now using find_package(Python) just like isce3 so once we bump pyre you can get by with just -DPython_EXECUTABLE.

Also, it sounds like you are letting cmake fetch pyre at build time - you can speed up the build a bit using prebuilt pyre 1.9.9 from conda-forge, and won't have to worry about pyre configuration parameters.

piyushrpt commented 3 years ago

I was building this on my mac. I will switch to using pyre from conda-forge on linux.

gmgunter commented 3 years ago

Hi @piyushrpt, thanks for testing out the new release and keeping us honest!

Some the tests fail with ctest possibly because they are trying to reach out to a s3 bucket to get data.

I think currently the only test that should require s3 bucket access is test.python.pkg.pybind_nisar.workflows.stage_dem. If any other tests are failing unexpectedly, please let us know.

Is there a flag / option to disable these tests

It's on our TODO list for some future release to add unit test markers so that you can easily filter out these tests, but there's nothing like this right now, sorry.

For now, you could use

$ ctest -E "stage_dem"

to exclude the stage_dem test, which is expected to fail. Hopefully, the relevant S3 bucket will be made public sometime soon anyway.

yamale is an undocumented dependency. Would it be useful to add this test at cmake stage or have it documented.

Oops, I forgot to update the public docs pages. While trying to resolve this, I realized that our CI bot, which is supposed to be generating the internal documentation pages for the develop branch, got removed in the most recent round of CI cleanup. Let me see if we can resolve this & update the public documentation shortly.

piyushrpt commented 3 years ago

These were the other tests that failed. Digging deeper to see why

The following tests FAILED:
     44 - test.cxx.isce3.geometry.bbox.geoperimeter_equator (Failed)
    140 - test.python.pkg.pybind_nisar.workflows.geocode_insar (Failed)
    146 - test.python.pkg.pybind_nisar.workflows.rubbersheet (Failed)
piyushrpt commented 3 years ago

The test.cxx.isce3.geometry.bbox.geoperimeter_equator is due to ogr->getNumPoints() counting duplicate vertex for closed polygons and reporting 42 points instead of 41, here: https://github.com/isce-framework/isce3/blob/59cdd2c659a4879367db5537604b0ca93d26b372/tests/cxx/isce3/geometry/bbox/geoperimeter_equator.cpp#L153

piyushrpt commented 3 years ago

The other failures seem to be a transient issue - different order of tests running might resolve it. Maybe dependency between tests?

hfattahi commented 3 years ago

The other failures seem to be a transient issue - different order of tests running might resolve it. Maybe dependency between tests?

any more details for the rubbersheet and geocode_insar?

hfattahi commented 3 years ago

sorry that was closed by mistake

hfattahi commented 3 years ago

Just saw that it was on your mac. rubbersheet test should not be run on CPU since we have only GPU ampcor.

vbrancat commented 3 years ago

Give the dependency in the CMakeFile (https://github.com/isce-framework/isce3/blob/59cdd2c659a4879367db5537604b0ca93d26b372/tests/python/packages/CMakeLists.txt#L23), the rubbersheet unit test should not run if the machine has no GPU :s

vbrancat commented 3 years ago

Found why the rubbersheet unit test gets executed also when no GPU is available. This line https://github.com/isce-framework/isce3/blob/59cdd2c659a4879367db5537604b0ca93d26b372/tests/python/packages/CMakeLists.txt#L18 should not be here. I will submit a PR to solve the issue :s

hfattahi commented 3 years ago

Found why the rubbersheet unit test gets executed also when no GPU is available. This line

https://github.com/isce-framework/isce3/blob/59cdd2c659a4879367db5537604b0ca93d26b372/tests/python/packages/CMakeLists.txt#L18

should not be here. I will submit a PR to solve the issue :s

Great that should take care of one failed test on CPU. I suspect geocode_insar is looking for geocoding the offsets which don't exist on CPU! need to take a closer look.

piyushrpt commented 3 years ago

Is there a shortcut to determine if CUDA support is included in a build? Either via isce3/ pyre - something like

isce3.hasCudaSupport or pyre.hasCudaSupport

Or maybe this can be determined by trying to import a cuda-based module and catching an import error.

geocode also is related to offsets from a quick glance at the code.

gmgunter commented 3 years ago

Is there a shortcut to determine if CUDA support is included in a build?

Yup, on the python side, you can do

import isce3
hasattr(isce3, "cuda") # True if CUDA support is enabled

On the C++ side, you can use

#include <isce3/config.h>

#if ISCE3_WITH_CUDA
// CUDA support is enabled
#else
// no CUDA support
#endif
Narayana-Rao commented 2 years ago

Is it possible to build and install ISCE3 by manually disabling CUDACXX ? The GPU I am using doesn't support CUDA>9.0. 

CMake Error at /usr/local/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:180 (message):
  Failed to find nvcc.

  Compiler requires the CUDA toolkit.  Please set the CUDAToolkit_ROOT
  variable.
Call Stack (most recent call first):
  CMakeLists.txt:47 (enable_language)
rtburns-jpl commented 2 years ago

Yes, by default cmake is detecting your existing cuda installation, but you can disable this entirely by configuring with the cmake arg -DWITH_CUDA=OFF.

Narayana-Rao commented 2 years ago

Thank you @rtburns-jpl, with -DWITH_CUDA=OFF arg, it is generating the following error (error_log.txt)


CMake Error at /usr/local/share/cmake-3.23/Modules/Internal/CheckFlagCommonConfig.cmake:58 (message):
  check_compiler_flag: CUDA: needs to be enabled before use.
Call Stack (most recent call first):
  /usr/local/share/cmake-3.23/Modules/Internal/CheckCompilerFlag.cmake:13 (cmake_check_flag_common_init)
  /usr/local/share/cmake-3.23/Modules/CheckCompilerFlag.cmake:40 (cmake_check_compiler_flag)
  .cmake/Warnings.cmake:67 (check_compiler_flag)
  .cmake/ConfigISCE.cmake:58 (set_warnings)
  CMakeLists.txt:86 (CheckCXX)
rtburns-jpl commented 2 years ago

Can you try deleting your build directory and starting again with a clean cache?

Narayana-Rao commented 2 years ago

Thank you @rtburns-jpl. After trying this I got 30 tests failed (Ubuntu 20.04; GDAL=3.0.4; Python=3.8). I have noticed that the majority of the failures are due to corrupted/incorrect paths of *.h5 files (I am not sure).
LastTest.log 

0% tests passed, 30 tests failed out of 30

Total Test time (real) =  30.61 sec

The following tests FAILED:
     87 - test.python.pybind.core.attitude (Failed)
     98 - test.python.pybind.core.orbit (Failed)
    104 - test.python.pybind.focus.backproject (Failed)
    108 - test.python.pybind.geocode.geocodeSlc (Failed)
    109 - test.python.pybind.geocode.geocodeCov (Failed)
    110 - test.python.pybind.geocode.radar_grid_cube (Failed)
    111 - test.python.pybind.geometry.bbox (Failed)
    113 - test.python.pybind.geometry.geo2rdr (Failed)
    114 - test.python.pybind.geometry.rdr2geo (Failed)
    115 - test.python.pybind.geometry.rtc (Failed)
    116 - test.python.pybind.geometry.metadata_cubes (Failed)
    136 - test.python.pybind.geometry.look_inc_from_sr (Failed)
    139 - test.python.pybind.antenna.el_pattern_est (Failed)
    148 - test.python.pkg.nisar.products.readers.raw (Failed)
    149 - test.python.pkg.nisar.workflows.crossmul (Failed)
    150 - test.python.pkg.nisar.workflows.doppler_lut_from_raw (Failed)
    151 - test.python.pkg.nisar.workflows.focus (Failed)
    152 - test.python.pkg.nisar.workflows.gen_doppler_range_product (Failed)
    153 - test.python.pkg.nisar.workflows.stage_dem (Failed)
    154 - test.python.pkg.nisar.workflows.gcov (Failed)
    155 - test.python.pkg.nisar.workflows.geo2rdr (Failed)
    156 - test.python.pkg.nisar.workflows.geocode_insar (Failed)
    157 - test.python.pkg.nisar.workflows.gslc (Failed)
    158 - test.python.pkg.nisar.workflows.insar (Failed)
    159 - test.python.pkg.nisar.workflows.filter_interferogram (Failed)
    160 - test.python.pkg.nisar.workflows.bandpass_insar (Failed)
    162 - test.python.pkg.nisar.workflows.rdr2geo (Failed)
    163 - test.python.pkg.nisar.workflows.resample_slc (Failed)
    164 - test.python.pkg.nisar.workflows.unwrap (Failed)
    166 - test.python.pkg.nisar.antenna.beamformer (Failed)
Errors while running CTest
gmgunter commented 2 years ago

Hi @Narayana-Rao, isce3 uses two different software packages to interact with HDF5 files: the h5py package is used by isce3's Python code, and the underlying libhdf5 library is used directly by isce3's C++ code. Due to the (somewhat awkward) way that isce3 currently passes references to H5Objects between its C++ and Python layers, it's very important for these two packages to be compatible in an ABI sense.

I suspect that the issue in this case is that these two packages (h5py and libhdf5) in your environment are binary incompatible. How did you install these packages? Did you install them using conda or some other package manager?

The packages should be reliably compatible if they were both installed from conda-forge, e.g. via

$ conda install --channel=conda-forge h5py hdf5

However, if, for example, you installed libhdf5 using the APT package manager

$ apt-get install libhdf5-dev

Then you should install h5py from source rather than installing a binary package. Otherwise, you're likely to get a binary that links against a different libhdf5, which will cause issues like what you experienced. In that case, you can try installing h5py from source using pip:

$ pip install --no-binary=h5py h5py

Hope that helps.

Narayana-Rao commented 2 years ago

Thank you @gmgunter, it has solved issues with the majority of tests but not all. Here is the list of tests that failed, mostly with E OSError: Unable to open file (file is not already open for SWMR writing) . I am using h5py=3.6.0.

LastTest.log

The following tests FAILED:
    139 - test.python.pybind.antenna.el_pattern_est (Failed)
    152 - test.python.pkg.nisar.workflows.gen_doppler_range_product (Failed)
    153 - test.python.pkg.nisar.workflows.stage_dem (Failed)
    165 - test.python.pkg.nisar.workflows.antenna_parser (Failed)
    166 - test.python.pkg.nisar.antenna.beamformer (Failed)
Errors while running CTest
Narayana-Rao commented 2 years ago

After a fresh installation of Ubuntu, now, I am able to successfully install and run tests (except stage_dem ) (Ubuntu 20.04; GDAL 3.0.4; Python 3.8.5). Thanks a lot, @gmgunter @rtburns-jpl for the support.

Narayana-Rao commented 2 years ago

I am trying to install and use plant-isce3. Therefore, I have installed latest isce3 ('0.8.0-dev+04b17713'). The following test are failed (Ubuntu 20.04; GDAL 3.0.4; Python 3.8.5) LastTest.log

140:test.python.pkg.isce3.ionosphere.ionosphere
153:test.python.pkg.nisar.workflows.stage_dem
162:test.python.pkg.nisar.workflows.ionosphere

Still I have installed plant, plant-isce3. However, while running plant_isce3_geocode.py, I have encountered the following error (plant_isce3.log). Could anyone please suggest any fix for this. 

AttributeError: 'isce3.ext.isce3.geocode.GeocodeCFloat32' object has no attribute 'dem_block_margin'