search

stcorp / harp

Data harmonization toolset for scientific earth observation data
http://stcorp.github.io/harp/doc/html/index.html
BSD 3-Clause "New" or "Revised" License
55 stars 19 forks source link

Cannot compile or build harp #242

Closed whositwhatnow closed 3 years ago

whositwhatnow commented 3 years ago

I cannot compile or build the application.

I meet all requirements and have all dependencies required but I keep on getting the error:

/bin/grep: /mnt/scr1/pre-release/hdf4/v4.2.15/thg-builds/moohan/lib/libmfhdf.la: No such file or directory /bin/sed: can't read /mnt/scr1/pre-release/hdf4/v4.2.15/thg-builds/moohan/lib/libmfhdf.la: No such file or directory libtool: error: '/mnt/scr1/pre-release/hdf4/v4.2.15/thg-builds/moohan/lib/libmfhdf.la' is not a valid libtool archive make[1]: *** [libharp.la] Error 1

I've indicated in the configure my libs/includes directories for all hdf4/5/cuda and it recognizes them

$ ./configure --prefix=/cm/shared/apps/harp/1.13 --enable-python PYTHON=/cm/shared/apps/python/3.8.7/bin/python3.8 --with-hdf4 HDF4_LIB=/cm/shared/apps/hdf4/4.2.15/lib/ HDF4_INCLUDE=/cm/shared/apps/hdf4/4.2.15/include/ --with-hdf5 HDF5_LIB=/cm/shared/apps/hdf5_18/1.8.17/lib/ HDF5_INCLUDE=/cm/shared/apps/hdf5_18/1.8.17/include/ CODA_LIB=/cm/shared/apps/coda/2.22/lib/ CODA_INCLUDE=/cm/shared/apps/coda/2.22/include/

svniemeijer commented 3 years ago

Does that /mnt/scr1/pre-release/hdf4/v4.2.15/thg-builds/moohan/lib/libmfhdf.la file actually exist? Can you let me know how you have build HDF4 (configure + make steps)?

whositwhatnow commented 3 years ago

The files exist but not in that directory, I built HDF4 using the CMake instructions

https://portal.hdfgroup.org/display/support/Building+HDF4+with+CMake

and used the .build-unix.sh script

No idea where that location came from.

svniemeijer commented 3 years ago

These .la files are generated when a package is build using a configure script that was created using the autotools (autoconf/automake/libtool). They should not appear when building with CMake. Are you sure those .la files were generated as part of a CMake build of HDF4?

The .la files contain (among other things) references to dependent libraries (and their locations). So if these files have invalid content, then you will get build errors when building with the autotools approach.

You might want to try removing the .la files and see if that works. Alternatively, you could try building CODA and HARP using CMake as well (that type of build will ignore .la files by default).

whositwhatnow commented 3 years ago

I built coda and hdf4 using cmake Still failing building harp using cmake.

svniemeijer commented 3 years ago

Are you getting the same error? Please provide sufficient context.

whositwhatnow commented 3 years ago

Yes correct same error, sorry was using the .configure command not cmake.

What are the options for cmake I cannot find info on building with cmake. It cannot find the HDF libraries.

Just using cmake . -Bbuild

svniemeijer commented 3 years ago

You can use ccmake for an interactive version of cmake that will show you what options can/should be set. You can also have a look at https://github.com/stcorp/harp-conda-recipe/blob/master/build.sh for some common options.

Furthermore, make sure that you remove any previous instance of your install directories (e.g. /cm/shared/apps/hdf4/4.2.15) before you build+install again. Otherwise artefacts of the autoconf build may still be present in the install directory.

whositwhatnow commented 3 years ago

So I was able to compile it using cmake, but it didn't apply the python portion (using python 3.8.7) This is my command i used to build it

cmake -DCMAKE_INSTALL_PREFIX="/cm/shared/apps/harp/1.13" -DHARP_BUILD_PYTHON=True -DCODA_INCLUDE_DIR="/cm/shared/apps/coda/2.22/include/" -DHDF4_INCLUDE_DIR="/cm/shared/apps/hdf4/4.2.15/include/" -DHDF5_INCLUDE_DIR="/cm/shared/apps/hdf5_18/1.8.17/include/" -DCODA_LIBRARY_DIR="/cm/shared/apps/coda/2.22/lib/" -DHDF4_LIBRARY_DIR="/cm/shared/apps/hdf4/4.2.15/lib/" -DHDF5_LIBRARY_DIR="/cm/shared/apps/hdf5_18/1.8.17/lib/"

svniemeijer commented 3 years ago

What do you mean that it "didn't apply the python portion"? Did it not install the python interfaces? Did you get any errors?

Also make sure that you remove the existing CMakeCache.txt file when you change any parameters in the cmake command. The cmake approach sometimes keeps things in cache that it shouldn't when you want to change the build configuration.

whositwhatnow commented 3 years ago

correct it did not install the python interfaces. No errors

Also, correct I have done that removing the Cache.txt file

svniemeijer commented 3 years ago

Can you show me the output of grep -i python CMakeCache.txt?

The python interface is just a bunch of python files based on cffi. There is nothing build for the python interface. So as long as HARP_BUILD_PYTHON:BOOL=ON is set, then your python files should be available after installation in /cm/shared/apps/harp/1.13/lib/python3.8/site-packages.

whositwhatnow commented 3 years ago

ok so thats where the issue is I have multiple versions of python it is installing it in the default 2.7 site packages. Is there anyway to add the different location

svniemeijer commented 3 years ago

Specifying your python version is a matter of pointing to the right python executable with a -DPYTHON_EXECUTABLE=/path/to/your/python3.8 cmake option.

whositwhatnow commented 3 years ago

So for some reason it is putting it in a new directory in the harp dir

Installing: /cm/shared/apps/harp/1.13/lib/python3.8/site-packages/harp/init.py -- Installing: /cm/shared/apps/harp/1.13/lib/python3.8/site-packages/harp/_harpc.py -- Installing: /cm/shared/apps/harp/1.13/lib/python3.8/site-packages/harp/_harppy.py

Not installing it in my python instance.

cmake -DCMAKE_INSTALL_PREFIX="/cm/shared/apps/harp/1.13" -DHARP_BUILD_PYTHON=True -DPYTHON_EXECUTABLE=/cm/shared/apps/python/3.8.7/bin/python3.8 -DCODA_INCLUDE_DIR="/cm/shared/apps/coda/2.22/include/" -DHDF4_INCLUDE_DIR="/cm/shared/apps/hdf4/4.2.15/include/" -DHDF5_INCLUDE_DIR="/cm/shared/apps/hdf5_18/1.8.17/include/" -DCODA_LIBRARY_DIR="/cm/shared/apps/coda/2.22/lib/" -DHDF4_LIBRARY_DIR="/cm/shared/apps/hdf4/4.2.15/lib/" -DHDF5_LIBRARY_DIR="/cm/shared/apps/hdf5_18/1.8.17/lib/"

whositwhatnow commented 3 years ago

guess I can add location to python libs -DPYTHON_LIBRARY_DIR="/cm/shared/apps/python/3.8.7/lib/"

svniemeijer commented 3 years ago

I am not sure I understand your approach. On one hand you seem to want to have each package in its own directory. But then when it comes to the python modules itself, you expect them to all end up in /cm/shared/apps/python/3.8.7? That makes no sense. If you use a prefix (similar like with python setup.py install --prefix or pip install --prefix) everything gets installed under that prefix: command line scripts, modules, etc. So if you are fine having python command line scripts installed under /cm/shared/apps/python/3.8.7/bin (which is what the setup.py/pip install would do), then why not the other software components.

I think you should maybe start thinking of using 'environments' (like venv or conda), because your current approach is not viable for managing different version of python modules.

In any case, for the HARP python interface to work, the python modules need to be placed correctly relative to the libraries (in <prefix>/lib) and data files (in <prefix>/share). So even moving the harp python modules to /cm/shared/apps/python/3.8.7 will not work unless you install HARP fully in /cm/shared/apps/python/3.8.7.

Since this doesn't seem to be a HARP issue any longer, I am closing this ticket.