Segmentation Fault at Startup

[alexmillane]

Hi. Really enjoyed reading your paper. Outstanding work!

Unfortunately, I've so far been thwarted trying to run the code. I am able to compile everything and run the tests (which succeed). However, if I try to run the examples I get an immediate segmentation fault, no output, just "'core dumped".

I get this behaviour on ubuntu 16.04, ubuntu 18.04, and I even get the same behaviour trying to run an example on your docker image (mrpt/mrpt-build-env:full-bionic).

One commonality with all these systems is me. Any ideas?

[jlblancoc]

Great, thanks!

Hmm... not a real long term solution, but, if you built gtsam from sources, try disabling the cmake option to "optimize native", in both gtsam and mola and let's see if that fixes the crash...

[dibet]

Hi, i have the same problem...

[dibet]

It is necessary install MOLA?

[alexmillane]

I compiled with:

MOLA_BUILD_MARCH_NATIVE=OFF

and get the same behavior.

I could compile with the debug symbols and see if I can get some more info on where things go wrong?

[dibet]

I compiled with MOLA_BUILD_MARCH_NATIVE=OFF and the same problem. I'am trying to debug...

[alexmillane]

Also as a note I'm using the system versions of the deps (yaml_cpp, gtsam and mrpt).

[dibet]

Hi again, In my case the problem is when loading the library libmola-slam-gtsam-dbg.so in the line

void* handle = dlopen(lib.wholePath.c_str(), RTLD_LAZY);

I see it in the build/lib/ path.

[jlblancoc]

@dibet :

It is necessary install MOLA?

Nope. Actually, I would strongly recommend not doing it

@alexmillane :

Also as a note I'm using the system versions of the deps (yaml_cpp, gtsam and mrpt).

For gtsam and mrpt, you mean the system versions, from their respective PPA, right?

In my case the problem is when loading the library libmola-slam-gtsam-dbg.so in the line

Looks like an Eigen / AVX2 optimization problem... Can you run grep flags /proc/cpuinfo just to see what's different with my test systems? (Since I cannot reproduce the sigsegv)

[dibet]

I'am using Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d

[dibet]

For gtsam and mrpt, you mean the system versions, from their respective PPA, right?

I follow this tutorial https://docs.mola-slam.org/latest/building.html I'am using kubuntu 16.04

[jlblancoc]

Hmm... what's your version of gcc? Perhaps for Ubuntu 16.04 you have to manually upgrade to gcc-7 even after installing mrpt from the PPA. Instructions here. Let me know if upgrading gcc and then recompiling mola fixes the problem.

[jlblancoc]

Just pushed a potential workaround (not actual solution, though...) to this problem. Try it with:

cd MOLA_SOURCE_ROOT_DIR
git pull
modules/pull-all-master.sh

and rebuilding mola.

Note that the last sh script is new, and referenced here in the docs.

[alexmillane]

For gtsam and mrpt, you mean the system versions, from their respective PPA, right?

Yep.

Perhaps for Ubuntu 16.04 you have to manually upgrade to gcc-7 even after installing mrpt from the PPA. Instructions here. Let me know if upgrading gcc and then recompiling mola fixes the problem.

I manually upgraded to gcc 7.4. I think this is required to build at all.

Just pushed a potential workaround (not actual solution, though...) to this problem. Try it with:

This seems to get further! But it still crashes, unfortunately. I get some output now:

MOLA_INPUT_RAWLOG=/media/millanea/datasets/mola/map1_test1_cut.rawlog  mola-cli -c rawlog_odom_and_lidar.yml -p
[11:14:02.8925|INFO |MolaLauncherApp] Setting up system from YAML config... (set DEBUG verbosity level to see full config)
[11:14:03.0373|INFO |MolaLauncherApp] Instantiating module `backend` of type `mola::ASLAM_gtsam`
[11:14:03.0377|INFO |MolaLauncherApp] Instantiating module `map` of type `mola::WorldModel`
[11:14:03.0379|INFO |WorldModel] === Using map name: `mola_map_2019_08_12,11_14_03.037950` ===
[11:14:03.0379|INFO |WorldModel] === Setting map storage base directory: `/tmp/mola_map_2019_08_12,11_14_03.037950/` ===
[11:14:03.0382|INFO |MolaLauncherApp] Instantiating module `lidar_fe` of type `mola::LidarOdometry`
[11:14:03.0385|INFO |MolaLauncherApp] Instantiating module `rawlog_input` of type `mola::RawlogDataset`
[11:14:03.0389|INFO |MolaLauncherApp] Entering main SLAM/localization loop...(CTRL+C from mola-cli to stop)
[11:14:03.0389|INFO |mola::ASLAM_gtsam:backend] Attached to WorldModel module `mola::WorldModel:map`
[11:14:03.0394|INFO |mola::LidarOdometry:lidar_fe] Attached to SLAM backend module `mola::ASLAM_gtsam:backend`
[11:14:03.0394|INFO |mola::LidarOdometry:lidar_fe] Attached to WorldModel module `mola::WorldModel:map`
[11:14:03.0396|INFO |mola::LidarOdometry:lidar_fe] Number of ICP working threads: 8 (determined automatically)
11:14:03: Warning: Mismatch between the program and library build versions detected.
The library used 3.0 (wchar_t,compiler with C++ ABI 1009,wx containers,compatible with 2.8),
and your program used 3.0 (wchar_t,compiler with C++ ABI 1011,wx containers,compatible with 2.8).
[11:14:03.0798|WARN |mola::ASLAM_gtsam:backend] Creating KF #1 without a good initial guess.
Segmentation fault (core dumped)

[jlblancoc]

Ok... I'll add more unit tests, and possibly CircleCI tests in different Ubuntu distros (docker images) to find out whether this is an issue of gtsam...

[jlblancoc]

Ok, let's try this: update again:

cd MOLA_SOURCE_ROOT_DIR
git pull
git submodule update --init

run cmake to realize of the changes:

cd build
cmake .. 

and build the new gtsam unit test in VERBOSE to see all the flags:

VERBOSE=1 make test-gtsam-basic-graph

finally, run all tests to see if the gtsam one crahses here or not: make test In any case, please, copy the result of the VERBOSE make command to see if it helps.

You can try make test with both configurations, cmake option MOLA_BUILD_MARCH_NATIVE to ON and OFF.

[jlblancoc]

I could reproduce this crash in a u16.04 system. It seems to be related to ABI incompatible changes between gcc5 and gcc7. Bottom line is: remove the libgtsam-dev package and build it manually, apparently without march=native.

Please, refer to the updated instructions for 16.04 distributed along this page.

I'll close this one, feel free of reopening if it still crashes after following all those instructions.