Native install error message on low RAM machine -- Configuring incomplete, errors occurred!

ivangayton commented 5 years ago

This is a Native install.

What's your browser and operating system?

Ubuntu 16.04.6 x64

1 GB RAM
25 GB Disk It's in fact the basic $5/month Digital Ocean droplet.

By the way, I'm not trying to run real ODM datasets with such a small server, I normally use this small instance to do the install, then resize to 192GB RAM and 32 processors—but that's a little expensive to keep running while doing an install!

What is the problem?

The install script throws an error: -- Configuring incomplete, errors occurred!. Subsequently I find myself in the good old "no module named ecto" situation.

Expected behavior?

I've done a few native installs (though not for a number of months), and usually I get a happy conclusion from this process, without the error message.

How can we reproduce this?

The creation of the server is as follows:

Create the Digital Ocean droplet (Ubuntu 16.04.06 x64, 1GB RAM, 25 GB SSD, 1 processor)
Connect to it via ssh as root ssh root@xxx.xxx.xxx.xxx
Create a new user using adduser odm
Give that user sudo privileges using usermod -aG sudo odm

Copy the Authorized Keys file to the new user using:

mkdir /home/odm/.ssh
cp .ssh/Authorized_keys /home/odm/.ssh/
chown odm /home/odm/.ssh/Authorized_keys
exit

Then log in as the new user using ssh odm@xxx.xxx.xxx.xxx

and execute sudo apt update && sudo apt upgrade -y

That's all I do (other than install a terminal emacs with sudo apt install emacs-nox). So it's a very vanilla Ubuntu environment, with nothing installed over and above the OS itself (other than emacs).

Install is as follows:

git clone https://github.com/opendronemap/opendronemap
cd opendronemap
sudo bash configure.sh install

The full terminal output is here, but the end of the story is:

c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
common/CMakeFiles/pcl_common.dir/build.make:422: recipe for target 'common/CMakeFiles/pcl_common.dir/src/range_image.cpp.o' failed
make[5]: *** [common/CMakeFiles/pcl_common.dir/src/range_image.cpp.o] Error 4
CMakeFiles/Makefile2:117: recipe for target 'common/CMakeFiles/pcl_common.dir/all' failed
make[4]: *** [common/CMakeFiles/pcl_common.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/pcl.dir/build.make:113: recipe for target 'pcl/stamp/pcl-build' failed
make[2]: *** [pcl/stamp/pcl-build] Error 2
CMakeFiles/Makefile2:327: recipe for target 'CMakeFiles/pcl.dir/all' failed
make[1]: *** [CMakeFiles/pcl.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
Compiling build
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- No build type selected, default to Release
-- The imported target "vtkRenderingPythonTkWidgets" references the file
   "/usr/lib/x86_64-linux-gnu/libvtkRenderingPythonTkWidgets.so"
but this file does not exist.  Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
   "/usr/lib/cmake/vtk-6.2/VTKTargets.cmake"
but not all the files it references.

-- The imported target "vtk" references the file
   "/usr/bin/vtk"
but this file does not exist.  Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
   "/usr/lib/cmake/vtk-6.2/VTKTargets.cmake"
but not all the files it references.

CMake Warning at modules/odm_georef/CMakeLists.txt:16 (find_package):
  Could not find a package configuration file provided by "PCL" (requested
  version 1.8) with any of the following names:

    PCLConfig.cmake
    pcl-config.cmake

  Add the installation prefix of "PCL" to CMAKE_PREFIX_PATH or set "PCL_DIR"
  to a directory containing one of the above files.  If "PCL" provides a
  separate development package or SDK, be sure it has been installed.

CMake Error at modules/odm_georef/CMakeLists.txt:19 (find_package):
  Could not find a package configuration file provided by "OpenCV" with any
  of the following names:

    OpenCVConfig.cmake
    opencv-config.cmake

  Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
  "OpenCV_DIR" to a directory containing one of the above files.  If "OpenCV"
  provides a separate development package or SDK, be sure it has been
  installed.

-- Configuring incomplete, errors occurred!
See also "/home/odm/opendronemap/build/CMakeFiles/CMakeOutput.log".
Configuration Finished

There's another red-highlighted error in middle of the console output that may be relevant:

Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-dDSL9J-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JJnQ51-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-dDSL9J-build/

While we're at it, here are the CMake logs.

Nevertheless, we persist. Now I add the path variables to the shell by using emacs to append the following lines into /home/odm/.bashrc:

export PYTHONPATH=$PYTHONPATH:/home/odm/opendronemap/SuperBuild/install/lib/python2.7/dist-packages
export PYTHONPATH=$PYTHONPATH:/home/odm/opendronemap/SuperBuild/src/opensfm
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/odm/opendronemap/SuperBuild/install/lib

Note that the suggested path in the documentation is /your/path/OpenDroneMap (capitalized), but a git clone produces a /whatever/path/opendronemap (uncapitalized) directory. This has tripped me up in the past, but not this time—I've checked and the path is correct.

In the process of verifying the paths, the second hint of trouble appears: the first addition to the PYTHONPATH doesn't seem to exist! The command ls /home/odm/opendronemap/SuperBuild/install/lib/ returns:

cmake                   liblaszip.so                               libpdal_plugin_reader_pgpointcloud.so.8.0.0
libceres.a              liblaszip.so.3                             libpdal_plugin_writer_pgpointcloud.so
libexiv2.so             liblaszip.so.3.1.1                         libpdal_plugin_writer_pgpointcloud.so.7
libexiv2.so.0.27.0      libpdal_base.so                            libpdal_plugin_writer_pgpointcloud.so.8.0.0
libexiv2.so.27          libpdal_base.so.7                          libpdal_util.so
libgflags.a             libpdal_base.so.8.0.0                      libpdal_util.so.7
libgflags_nothreads.a   libpdalcpp.so                              libpdal_util.so.8.0.0
libhexer.so             libpdal_plugin_kernel_fauxplugin.so        libtex.a
libhexer.so.1.1.0       libpdal_plugin_kernel_fauxplugin.so.7      libxmp.a
liblaszip_api.so        libpdal_plugin_kernel_fauxplugin.so.8.0.0  pdal
liblaszip_api.so.3      libpdal_plugin_reader_pgpointcloud.so      pkgconfig
liblaszip_api.so.3.1.1  libpdal_plugin_reader_pgpointcloud.so.7

There's no .../python2.7/dist-packages in there.

Ok, at this point, I don't know if the error is fatal or not, or if that seemingly missing library is actually needed. So I download the smallest test dataset using git clone https://github.com/merkato/odm_boruszyn_kap.git.

In order to ensure that the shell variables are loaded from the .bashrc file, which requires a logout, I exit and ssh back in, then execute:

cd opendronemap
python run.py -i ~/odm_boruszyn_kap/images/ test-project

And: hooray! Here's our old friend:

ImportError: No module named ecto.

Since I'm fairly confident that I've got the path variables exported exactly as last time I successfully installed, I suspect it's not the usual error on my part, there's something in the install that's not working!

Hope this helps!

Thanks, as always, for the amazing software.

Ivan

pierotofy commented 5 years ago

Hey Ivan, thanks for the detailed error report!

The issue is that it takes a bit of memory to compile ODM natively, in particular as you found out the compiler will give up with the error: c++: internal compiler error: Killed (program cc1plus) if you don't have enough memory.

From experimentation I usually have a VM with ~3GB of RAM allocated to build our docker images (and have had the process fail with 2GB), so my recommendation would be to either:

Use the docker image which requires no compilation.
Compile ODM on a 4GB machine, take a backup and restore it to a smaller droplet (or in your case, restore it directly to your bigger droplet). https://www.digitalocean.com/docs/droplets/resources/downsize/

Cheers!

ivangayton commented 5 years ago

D'oh! Thanks @pierotofy , I'm an idiot! Of course that's the problem.

smathermather commented 5 years ago

Ha! That's ok Ivan. I've done this too. And I thus I should have caught your issue too... .

OpenDroneMap / ODM