search

isce-framework / isce3

InSAR Scientific Computing Environment ***NOTICE - This repository is currently a mirror of an internally hosted project and is in the process of migrating the development to this public repository. In near future we will accept external pull requests.***
Other
130 stars 35 forks source link

alos2_to_nisar_l1.py can't convert alos2 ceos to h5 #12

Open zyxgis opened 1 year ago

zyxgis commented 1 year ago

os : centos 7 python: 3.9 cxx: gcc11

i use alos2_to_nisar_l1.py to convert alos2 ceos to h5, but i get the follow error:

$ /python/bin/python3 /home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py -l HH -i /home/testUser/test/ceos/0000471428_001001_ALOS2441290770-220725/ -o /home/testUser/test/ceos/output-hh2.h5 looking for available polarizations... found polarization: HH HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:

000: H5O.c line 1175 in H5Ovisit_by_name3(): invalid location identifier

major: Invalid arguments to routine
minor: Inappropriate type

001: H5VLint.c line 1741 in H5VL_vol_object(): invalid identifier

major: Invalid arguments to routine
minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:

000: H5D.c line 132 in H5Dcreate2(): invalid location identifier

major: Invalid arguments to routine
minor: Inappropriate type

Traceback (most recent call last): File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 720, in process(args=args) File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 139, in process orbit = construct_nisar_hdf5(args.outh5, leader) File "/home/testUser/isce/share/nisar/examples/alos2_to_nisar_l1.py", line 253, in construct_nisar_hdf5 set_h5_orbit(orbit_group, orbit) File "/home/testUser/isce/share/nisar/examples/alos_to_nisar_l0b.py", line 141, in set_h5_orbit orbit.save_to_h5(group) RuntimeError: Caught an unknown exception!

bhawkins commented 1 year ago

My guess is that your h5py and isce3 are linked against different versions of the HDF5 library. You might be able to confirm this with ldd. For example, figure out where your h5py is installed with

python3 -c "import h5py; print(h5py.__file__)"

and there should be a bunch of *.so files in the same directory. You can run ldd on one of those .so files and libisce3.so to see if they link to the same libhdf5.so file (or if it was statically linked then it won't show up). My usual workaround is to install h5py from source, manually making sure to link with the same HDF5 library I used to compile isce3.

The isce3 developers plan to refactor the code to avoid sharing HDF5 objects between the C++ and Python layers, but that effort will take some time.

zyxgis commented 1 year ago 

python3 -c "import h5py; print(h5py.__file__)"
/python/lib/python3.9/site-packages/h5py/__init__.py


ldd /home/testUser/isce/lib64/libisce3.so
        linux-vdso.so.1 =>  (0x00007ffd03f41000)
        libgdal.so.31 => /home/testUser/gdal/lib/libgdal.so.31 (0x00007fb3d9492000)
        libpyre.so => /python/compile/lib64/libpyre.so (0x00007fb3db258000)
        libhdf5_cpp.so.200 => /home/testUser/gdal/lib/libhdf5_cpp.so.200 (0x00007fb3db1da000)
        libhdf5.so.200 => /home/testUser/gdal/lib/libhdf5.so.200 (0x00007fb3d9054000)
        libhdfs.so.0.0.0 => /home/testUser/gdal/lib/libhdfs.so.0.0.0 (0x00007fb3d8e41000)
        libcrypto.so.3 => /home/testUser/gdal/lib64/libcrypto.so.3 (0x00007fb3d89d9000)
        libcurl.so.4 => /home/testUser/gdal/lib/libcurl.so.4 (0x00007fb3db14d000)
        libsz.so.2 => /home/testUser/gdal/lib/libsz.so.2 (0x00007fb3db138000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb3d87c3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb3d85bf000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb3d82bd000)
        libjournal.so => /python/compile/lib64/libjournal.so (0x00007fb3db0a4000)
        libfftw3f.so.3 => /home/testUser/isce/lib/libfftw3f.so.3 (0x00007fb3d819d000)
        libfftw3f_threads.so.3 => /home/testUser/isce/lib/libfftw3f_threads.so.3 (0x00007fb3db09a000)
        libfftw3_threads.so.3 => /home/testUser/isce/lib/libfftw3_threads.so.3 (0x00007fb3db091000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fb3d7f77000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb3d7d5b000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb3d7be7000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb3d79d1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb3d7603000)
        libSFCGAL.so.1 => /home/testUser/gdal/lib64/libSFCGAL.so.1 (0x00007fb3d6b42000)
        libgeos_c.so.1 => /home/testUser/gdal/lib64/libgeos_c.so.1 (0x00007fb3d6afa000)
        libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fb3d68cf000)
        libopenjp2.so.7 => /lib64/libopenjp2.so.7 (0x00007fb3d667c000)
        libnetcdf.so.19 => /home/testUser/gdal/lib/libnetcdf.so.19 (0x00007fb3d6427000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fb3d61fc000)
        libpq.so.5 => /home/testUser/gdal/lib/libpq.so.5 (0x00007fb3d5fac000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fb3d5d9d000)
        libzstd.so.1 => /home/testUser/gdal/lib/libzstd.so.1 (0x00007fb3d5cb5000)
        liblzma.so.5 => /home/testUser/gdal/lib/liblzma.so.5 (0x00007fb3d5c8d000)
        libproj.so.25 => /home/testUser/gdal/lib64/libproj.so.25 (0x00007fb3d588f000)
        libsqlite3.so.0 => /home/testUser/gdal/lib/libsqlite3.so.0 (0x00007fb3d574c000)
        libtiff.so.5 => /home/testUser/gdal/lib/libtiff.so.5 (0x00007fb3d56d3000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fb3d547e000)
        libdeflate.so.0 => /home/testUser/gdal/lib/libdeflate.so.0 (0x00007fb3db077000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fb3d5276000)
        libpcre2-8.so.0 => /home/testUser/gdal/lib/libpcre2-8.so.0 (0x00007fb3d51da000)
        libiconv.so.2 => /home/testUser/gdal/lib/libiconv.so.2 (0x00007fb3d50f4000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fb3d4d8a000)
        libjvm.so => /home/testUser/java/jdk1.8.0_211/jre/lib/amd64/server/libjvm.so (0x00007fb3d3da1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb3db04d000)
        libssl.so.3 => /home/testUser/gdal/lib64/libssl.so.3 (0x00007fb3d3cf8000)
        libfftw3.so.3 => /home/testUser/isce/lib/libfftw3.so.3 (0x00007fb3d3bd1000)
        libgmpxx.so.4 => /lib64/libgmpxx.so.4 (0x00007fb3d39ca000)
        libboost_thread.so.1.80.0 => /home/testUser/gdal/lib/libboost_thread.so.1.80.0 (0x00007fb3d39af000)
        libboost_system.so.1.80.0 => /home/testUser/gdal/lib/libboost_system.so.1.80.0 (0x00007fb3d39aa000)
        libboost_serialization.so.1.80.0 => /home/testUser/gdal/lib/libboost_serialization.so.1.80.0 (0x00007fb3d3954000)
        libmpfr.so.6 => /home/testUser/gdal/lib/libmpfr.so.6 (0x00007fb3d36aa000)
        libgmp.so.10 => /home/testUser/gdal/lib/libgmp.so.10 (0x00007fb3d3633000)
        libgeos.so.3.11.0 => /home/testUser/gdal/lib64/libgeos.so.3.11.0 (0x00007fb3d3391000)
        libhdf5_hl.so.200 => /home/testUser/gdal/lib/libhdf5_hl.so.200 (0x00007fb3d336c000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007fb3d30fa000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fb3d2c97000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fb3d2a4a000)
        libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007fb3d27eb000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fb3d2502000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fb3d22fd000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fb3d20ca000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fb3d1eba000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb3d1cb5000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb3d1a9b000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fb3d188c000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fb3d166e000)
        libssl3.so => /lib64/libssl3.so (0x00007fb3d1409000)
        libsmime3.so => /lib64/libsmime3.so (0x00007fb3d11e1000)
        libnss3.so => /lib64/libnss3.so (0x00007fb3d0ea6000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007fb3d0c76000)
        libplds4.so => /lib64/libplds4.so (0x00007fb3d0a72000)
        libplc4.so => /lib64/libplc4.so (0x00007fb3d086c000)
        libnspr4.so => /lib64/libnspr4.so (0x00007fb3d062d000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb3d0405000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb3d01ce000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb3cff6b000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fb3cfd68000)
zyxgis commented 1 year ago

i compile hdf5-1.12.2.tar.gz i install h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to python3

bhawkins commented 1 year ago

Still need to see the output of ldd on one of the .so files in /python/lib/python3.9/site-packages/h5py. My guess is that it will be using a different libhdf5.so than your isce3 library uses, which is the problem. If that's the case, my usual workaround is to install h5py from source (not a binary .whl).