search

mapillary / OpenSfM

Open source Structure-from-Motion pipeline
https://www.opensfm.org/
BSD 2-Clause "Simplified" License
3.3k stars 848 forks source link

Script errors/aborts when running Berlin example #959

Open hengfranklin opened 1 year ago

hengfranklin commented 1 year ago

Hi all,

I successfully installed all the dependencies and built openSFM following the ubuntu instructions. I'm currently running on Ubuntu 22.04.

However, when I run: bin/opensfm_run_all data/berlin, I get the following error right away:

line 17: 91044 Aborted (core dumped) "$PYTHON" "$DIR"/opensfm_main.py "$@"

I also printed PYTHON and DIR to make sure it's all the correct paths. Thanks!

hengfranklin commented 1 year ago

Essentially getting Aborted (core dumped) when python3 bin/opensfm_main.py extract_metadata data/berlin runs

ishaniis commented 1 year ago

@hengfranklin : Were you able to solve this issue?

kip-carr commented 1 year ago

I am seeing the same errors. It seems to happen at the end of every step. Running with python 3.8 if that helps.

vankeer commented 1 year ago

Same here; when running on the test data $ bin/opensfm_run_all data/berlin I am getting:

/path/to/OpenSfM/bin/opensfm: line 17: 70945 Aborted
    (core dumped) "$PYTHON" "$DIR"/opensfm_main.py "$@"

Ubuntu 22.04 LTS 64-bit with 128 GB RAM and a i9-12900k CPU. Python 3.10.6 OpenCV 4.7 Ceres Solver 2.1 and all dependencies in the requirements.txt were installed:

cloudpickle==0.4.0
exifread==2.1.2
flask==2.1.2
fpdf2==2.4.6
joblib==0.14.1
matplotlib
networkx==2.5
numpy
Pillow>=8.1.1
pyproj>=1.9.5.1
pytest==3.0.7
python-dateutil>=2.7
pyyaml==5.4
scipy
Sphinx==4.2.0
six
xmltodict==0.10.2
wheel
opencv-python==4.5.1.48 ; sys_platform == "win32"
opencv-python ; sys_platform == "linux"

Not sure how to fix this?

$ gdb -ex r --args python3-dbg bin/opensfm_main.py extract_metadata data/berlin
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
/home/laurens/.gdbinit:1: Error in sourced command file:
Undefined command: "zcat".  Try "help".
Reading symbols from python3-dbg...
Starting program: /usr/bin/python3-dbg bin/opensfm_main.py extract_metadata data/berlin
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffea3ac640 (LWP 76290)]
[New Thread 0x7fffe9bab640 (LWP 76291)]
[New Thread 0x7fffe13aa640 (LWP 76292)]
[New Thread 0x7fffd8ba9640 (LWP 76293)]
[New Thread 0x7fffd03a8640 (LWP 76294)]
[New Thread 0x7fffc7ba7640 (LWP 76295)]
[New Thread 0x7fffbf3a6640 (LWP 76296)]
[New Thread 0x7fffaeba5640 (LWP 76297)]
[New Thread 0x7fffae3a4640 (LWP 76298)]
[New Thread 0x7fffa5ba3640 (LWP 76299)]
[New Thread 0x7fff9d3a2640 (LWP 76300)]
[New Thread 0x7fff94ba1640 (LWP 76301)]
[New Thread 0x7fff8c3a0640 (LWP 76302)]
[New Thread 0x7fff83b9f640 (LWP 76303)]
[New Thread 0x7fff7b39e640 (LWP 76304)]
[New Thread 0x7fff6ab9d640 (LWP 76305)]
[New Thread 0x7fff6a39c640 (LWP 76306)]
[New Thread 0x7fff59b9b640 (LWP 76307)]
[New Thread 0x7fff5139a640 (LWP 76308)]
[New Thread 0x7fff48b99640 (LWP 76309)]
[New Thread 0x7fff48398640 (LWP 76310)]
[New Thread 0x7fff37b97640 (LWP 76311)]
[New Thread 0x7fff2f396640 (LWP 76312)]
../Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x1533240
object refcount : 10
object type     : 0x9aeb40
object type name: type
object repr     : <class 'cv2.utils.nested.ExportClassName'>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ffff7c42000 (most recent call first):
  Garbage-collecting
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1184 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/usr/lib/python3/dist-packages/pyproj/crs/__init__.py", line 7 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/usr/lib/python3/dist-packages/pyproj/__init__.py", line 57 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/path/to/OpenSfM/opensfm/io.py", line 11 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/path/to/OpenSfM/opensfm/large/metadataset.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/path/to/OpenSfM/opensfm/actions/align_submodels.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/path/to/OpenSfM/opensfm/commands/align_submodels.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/path/to/OpenSfM/opensfm/commands/__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/path/to/OpenSfM/bin/opensfm_main.py", line 9 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg.lapack_lite, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyproj._compat, pyproj._datadir, pyproj._network, pyproj._crs (total: 18)

Thread 1 "python3-dbg" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350213632) at ./nptl/pthread_kill.c:44
44  ./nptl/pthread_kill.c: No such file or directory.
(gdb)
vankeer commented 1 year ago

Update to my last comment: it seems my Nvidia drivers weren't installed properly (it's a new machine). I re-installed the 525 driver, made sure to enroll in MOK after reboot, and now CUDA support works, as well as OpenSfM... up until a certain point.

Now I'm getting an intel-mkl related error - even though I installed intel-mkl and libmkl-full-dev. Also tried pip install mkl.

$LD_PRELOAD is set to /usr/lib/x86_64-linux-gnu/libmkl_def.so:/usr/lib/x86_64-linux-gnu/libmkl_avx2.so:/usr/lib/x86_64-linux-gnu/libmkl_core.so:/usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so:/usr/lib/x86_64-linux-gnu/libmkl_intel_thread.so:/usr/lib/x86_64-linux-gnu/libiomp5.so

Still...

$ ./bin/opensfm_run_all data/berlin/
python3
/path/to/OpenSfM/bin
/path/to/OpenSfM/bin/opensfm
extract_metadata
extract_metadata data/berlin/
2023-01-15 13:33:12,555 INFO: Loading existing EXIF for 03.jpg
2023-01-15 13:33:12,559 INFO: Loading existing EXIF for 02.jpg
2023-01-15 13:33:12,559 INFO: Loading existing EXIF for 01.jpg
python3
/path/to/OpenSfM/bin
/path/to/OpenSfM/bin/opensfm
detect_features
detect_features data/berlin/
2023-01-15 13:33:13,004 INFO: Planning to use 111679.2 MB of RAM for both processing queue and parallel processing.
2023-01-15 13:33:13,004 INFO: Scale-space expected size of a single image : 23.8125 MB
2023-01-15 13:33:13,004 INFO: Expecting to queue at most 200 images while parallel processing of 8 images.
2023-01-15 13:33:13,006 INFO: Reading data for image 03.jpg (queue-size=0)
2023-01-15 13:33:13,006 INFO: Reading data for image 02.jpg (queue-size=0)
2023-01-15 13:33:13,006 INFO: Reading data for image 01.jpg (queue-size=0)
2023-01-15 13:33:13,082 INFO: Skip recomputing ROOT_HAHOG features for image 02.jpg
2023-01-15 13:33:13,083 INFO: Skip recomputing ROOT_HAHOG features for image 01.jpg
2023-01-15 13:33:13,083 INFO: Finished reading images
2023-01-15 13:33:13,083 INFO: Skip recomputing ROOT_HAHOG features for image 03.jpg
python3
/path/to/OpenSfM/bin
/path/to/OpenSfM/bin/opensfm
match_features
match_features data/berlin/
2023-01-15 13:33:13,513 INFO: Matching 3 image pairs
2023-01-15 13:33:13,515 INFO: Computing pair matching with 8 processes
2023-01-15 13:33:13,548 DEBUG: Masking 0 / 4744 (0.00) features for 03.jpg
2023-01-15 13:33:13,549 DEBUG: Masking 0 / 4659 (0.00) features for 01.jpg
2023-01-15 13:33:13,557 DEBUG: Masking 0 / 4659 (0.00) features for 01.jpg
2023-01-15 13:33:13,579 DEBUG: Masking 0 / 4432 (0.00) features for 02.jpg
2023-01-15 13:33:13,580 DEBUG: Masking 0 / 4432 (0.00) features for 02.jpg
2023-01-15 13:33:13,737 DEBUG: Matching 01.jpg and 03.jpg.  Matcher: FLANN (symmetric) T-desc: 0.220 T-robust: 0.001 T-total: 0.221 Matches: 334 Robust: 299 Success: True
2023-01-15 13:33:13,765 DEBUG: Matching 03.jpg and 02.jpg.  Matcher: FLANN (symmetric) T-desc: 0.246 T-robust: 0.002 T-total: 0.249 Matches: 794 Robust: 739 Success: True
2023-01-15 13:33:13,766 DEBUG: Matching 01.jpg and 02.jpg.  Matcher: FLANN (symmetric) T-desc: 0.248 T-robust: 0.002 T-total: 0.250 Matches: 1210 Robust: 1179 Success: True
2023-01-15 13:33:13,766 INFO: Matched 3 pairs (perspective-perspective: 3) in 0.2529736200000343 seconds (0.08432462566668164 seconds/pair).
python3
/path/to/OpenSfM/bin
/path/to/OpenSfM/bin/opensfm
create_tracks
create_tracks data/berlin/
2023-01-15 13:33:14,212 INFO: reading features
2023-01-15 13:33:14,227 DEBUG: Merging features onto tracks
2023-01-15 13:33:14,242 DEBUG: Good tracks: 1656
python3
/path/to/OpenSfM/bin
/path/to/OpenSfM/bin/opensfm
reconstruct
reconstruct data/berlin/
2023-01-15 13:33:14,670 INFO: Starting incremental reconstruction
2023-01-15 13:33:14,684 INFO: Starting reconstruction with 01.jpg and 02.jpg
2023-01-15 13:33:14,689 INFO: Two-view 5-points reconstruction inliers (transposed=False): 1189 / 1189
INTEL MKL ERROR: /lib/x86_64-linux-gnu/libmkl_avx2.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

EDIT: never mind; fixed it! Must have been some dynamic linking problem, as the problem was gone after exporting LD_PRELOAD like this:

export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libmkl_core.so:/usr/lib/x86_64-linux-gnu/libmkl_sequential.so'

WangYu0611 commented 1 year ago

so , python 3.8 can fix this question？

WangYu0611 commented 1 year ago

I am seeing the same errors. It seems to happen at the end of every step. Running with python 3.8 if that helps.

this idea can fix the error