search

openMVG / openMVG

open Multiple View Geometry library. Basis for 3D computer vision and Structure from Motion.
Mozilla Public License 2.0
5.66k stars 1.67k forks source link

Non-deterministic behavior (Apple vs Ubuntu) #2079

Open dbeef opened 2 years ago

dbeef commented 2 years ago

Hello,

To begin with, great respect for fantastic work which is OpenMVG.

While using SIFT_Anatomy_Image_describer I discovered a discrepancy when it comes to the output it produces - the results are consistent on the same platform, however, they start to differ when running on a different device.

The problem can be reproduced by running a particular test - openMVG_test_sift which yields slightly different output image (gaussian_octave_*.png) on an M1 Mac and on a generic Ubuntu PC, with the same image on the input.

I include my Mac/Ubuntu build scripts here: https://github.com/migr8/openmvg-build-scripts/ The README in the root of the repository describes the exact usage. The test.sh script will run this specific test case and print MD5 hash of the output gaussian_octave images - they are different based on different platforms:

# Ubuntu:
Image hashes & sizes:
7237fd0531cf3af860b70070048ef55b  gaussian_octave_0_0.png
959bbb3f5f6507914bff16129d3c39e1  gaussian_octave_0_1.png
e959d017a202bd195452843d7a03e871  gaussian_octave_0_2.png
-rw-rw-r-- 1 dbeef dbeef 68131 cze  7 11:22 gaussian_octave_0_0.png
-rw-rw-r-- 1 dbeef dbeef 61927 cze  7 11:22 gaussian_octave_0_1.png
-rw-rw-r-- 1 dbeef dbeef 56697 cze  7 11:22 gaussian_octave_0_2.png

# Mac M1 (Apple Clang):
Image hashes & sizes:
7522636b1b967f43e5403d1b46b0c5ee  gaussian_octave_0_0.png
d78d803126c5425c3c2ebbadcf7236bb  gaussian_octave_0_1.png
1fa2913d3d41f89f68a3e7fce93602b1  gaussian_octave_0_2.png
-rw-r--r-- 1 dbeef staff 68133 Jun  7 10:58 gaussian_octave_0_0.png
-rw-r--r-- 1 dbeef staff 61927 Jun  7 10:58 gaussian_octave_0_1.png
-rw-r--r-- 1 dbeef staff 56705 Jun  7 10:58 gaussian_octave_0_2.png

# Mac M1 (GNU GCC 12)
Image hashes & sizes:
d0467e3aa5b6352889dc43c7ed384d39  gaussian_octave_0_0.png
ef2c73ecfe0a3cb28888840bbbfc42ad  gaussian_octave_0_1.png
ae5417914c6fa43630e501e31806f4ae  gaussian_octave_0_2.png
-rw-r--r-- 1 dbeef staff 68154 Jun  7 11:42 gaussian_octave_0_0.png
-rw-r--r-- 1 dbeef staff 61929 Jun  7 11:42 gaussian_octave_0_1.png
-rw-r--r-- 1 dbeef staff 56733 Jun  7 11:42 gaussian_octave_0_2.png

# MacBook Pro 2018 Intel Core i7 (Apple Clang)
Image hashes & sizes:
7012158e0075c48553538c765fa71a3a gaussian_octave_0_0.png
ce810921eac49822a967c48adc43d40f gaussian_octave_0_1.png
e959d017a202bd195452843d7a03e871 gaussian_octave_0_2.png
-rw-r--r-- 1 me staff 68135 Jun 7 11:49 gaussian_octave_0_0.png
-rw-r--r-- 1 me staff 61929 Jun 7 11:49 gaussian_octave_0_1.png
-rw-r--r-- 1 me staff 56697 Jun 7 11:49 gaussian_octave_0_2.png

I traced similar reports on the OpenMVG issue page:

https://github.com/openMVG/openMVG/issues/1707 https://github.com/openMVG/openMVG/issues/1966 https://github.com/openMVG/openMVG/issues/471 https://github.com/openMVG/openMVG/issues/1707 https://github.com/openMVG/openMVG/issues/2030 https://github.com/openMVG/openMVG/issues/1928 https://github.com/openMVG/openMVG/issues/1840 https://github.com/openMVG/openMVG/issues/1128

Most of them conclude that this is due to OMP parallelization. I made sure no OMP is used through:

However, the result is still the same, that is - they differ.

As a note, contents of the patches directory on the build scripts repo could be upstreamed to the OpenMVG repository - I solved the problem of building on an M1 Mac (running OpenMVG on it right now :)) and even targeting iOS through ios-cmake toolchain file - which solves relevant issues:

https://github.com/openMVG/openMVG/issues/2024 https://github.com/openMVG/openMVG/issues/1954 https://github.com/openMVG/openMVG/issues/1864 https://github.com/openMVG/openMVG/issues/1820

In total, I ran OpenMVG on x86_64 Windows, x86_64 Ubuntu, ARM64 Android, ARM64 iOS, ARM64 Mac and x86_64 Mac - all to see how the output differs across the platforms.

Results are, that all Apple results are consistent (scratch that, see Intel vs ARM Mac above), and all non-Apple results are consistent - which would suggest this has something to do specifically with Apple.

Another path I explored is compiling OpenMVG on Mac using GCC 12 - different results than using clang, but not the same as on non-Apple devices.

Also, I tested how setting constant seed through std::srand will affect the test - it does not.

Regards, Daniel

pmoulon commented 1 year ago

Thank you for your report. What do you propose for actions?