Cmake error concerning Qt

[PascalYIN]

Hello, I am a beginner of TSDF fusion 3D reconstruction and I was trying to build your DynamicFusion code to see its performance. However I got some errors when cmaking your code :

CMake Error at cmake/SetCUDAAppTarget.cmake:18 (add_executable): Target "InfiniTAM" links to target "Qt4::QtCore" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Call Stack (most recent call first): Apps/InfiniTAM/CMakeLists.txt:51 (INCLUDE)

There were about 20 similar errors.

Do you have any idea about why I have these errors or how can I fix them ? Thanks for your help ! :)

[Algomorph]

Hi, @PascalYIN This branch is both WIP and somewhat "abandoned", so before you try anything, please take a look at this thread here: #137 In short,

• The multithreaded version has a bug and the CUDA implementation is incomplete.
• Some effort would be required to get it back up and working as well as it did.

If you're willing to try to get it working again, then keep in mind: there's nothing wrong with the CMake, as tested on Ubuntu 18.04, as I and another developer from #137 just verified yesterday.

My code doesn't use Qt directly, but VTK uses it. There was a bug in Qt, so I have some code that conditionally works around that bug, but it doesn't add the Qt as target.

Here is what I have in my cmake-gui for Qt:

Build result:

[PascalYIN]

Thank you @Algomorph for your kind responses. I followed your parameters and successfully built the project. Though the executable file looks like a shared lib file, weird.
I didn't build the file with OpenMP and it seems the optimization process takes quite a long time, is that normal? I haven't finished downloading the Snoopy dataset so I'm not yet able to compare with your reconstruction result, but I still want to ask you what have you done to cut down the depth information that don't belong to the Snoopy's body ? It seems that background depth pixels will also be reconstructed.

[Algomorph]

@PascalYIN ,

Though the executable file looks like a shared lib file, weird.

InfiniTAM builds as a shared library, while the executables are all in the Apps folder and they simply link to it.

I didn't build the file with OpenMP and it seems the optimization process takes quite a long time, is that normal?

The optimization isn't fully optimized (pun intended), but yes, even with the voxel block hash on a machine with a mediocre CPU it would take awhile being single-threaded. I think the OpenMP / parallelized version might have a race condition somewhere (see #137 ). In my last month working on this repo (August 2018) before switching to building my own from scratch, I have focused on just getting the thing to work as well as I could on the CPU first.

What have you done to cut down the depth information that don't belong to the Snoopy's body?

The Snoopy dataset has mask binary images included. They're not stellar (esp. for the first few frames), but they are what you would use to benchmark against Mira Slavcheva's results. My implementation has support for these masks built in, just take a look at this example run from #137 and you'll know how to use them.

[PascalYIN]

@Algomorph Thanks. Is it possible to save the reconstructed model ? I tried to have a look at the back of the reconstructed model but the interface of InfiniTAM is not very easy to use. I mean, I found it difficult to "walk around" in the interface. And also there seems to be some strange distortions if you look from the back of the reconstructed surface.

[Algomorph]

@PascalYIN Yeah, I think there should be a keyboard shortcut there to run marching cubes and dump the 3D model to disk. It should be labeled on the GUI, if it's not, look at the InfiniTAM_bpo code that handles the keyboard.

[PascalYIN]

@Algomorph I have managed to save the reconstructed model, it seems that the program is able to well reconstruct the very first part of surface, however, as the object turns (like rigid rotation), the program is not able to track the rotation correctly. Which means if we look at the back of the model we can see a lot of fragmented slices. By the way, is it possible to increase voxel density?

[Algomorph]

@PascalYIN I see, it sounds similar to the results I got. KillingFusion / SobolevFusion both use the SDF-2-SDF algorithm for the rigid optimization part, but it is mentioned that any alternative fusion-based rigid tracker should work, hence here I just use one of the trackers already included in InfiniTAM. Manipulating its parameters in the settings might improve the result, but something makes me doubt that the rigid tracking other than SDF-2-SDF can suffice to produce the same-quality result.

You can manipulate the voxel density and other TSDF scene parameters by changing this line here: https://github.com/Algomorph/InfiniTAM/blob/ac2f3f7db90dac14469e5e3744827078e97cb906/InfiniTAM/ITMLib/Utils/ITMLibSettings.cpp#L14

The scene constructor is documented here: https://github.com/Algomorph/InfiniTAM/blob/ac2f3f7db90dac14469e5e3744827078e97cb906/InfiniTAM/ITMLib/Utils/ITMSceneParams.h#L56

I'm surprised you were able to get a decent reconstruction on the first few frames -- I tested the code recently and were not able to reproduce my better results from before, I thought it's either due to a bug that creeped in or due to some differences in system / compiler. Out of curiosity, which operating system / version are you using?

[PascalYIN]

@Algomorph In case that we weren't talking about the same performance, I show here the screen shot of my reconstruction result at frame 65. The result is obtained by using the same command lines as in https://github.com/Algomorph/InfiniTAM/issues/137#issuecomment-476378501, I didn't use the calibration file of the Snoopy dataset.

It's obtained on Ubuntu 18.04.2 LTS with Qt-5.10.0 VTK-8.1.1(gcc 4.8.5), OpenC-3.4.5(gcc 4.8.5) and boost-1.65.1(gcc 5.0). InfiniTAM_bpo compiled with gcc 5.0 as well.

It's not exactly same as the result that you have shown here (https://github.com/Algomorph/InfiniTAM/issues/137#issuecomment-476631396) though.

For the rigid rotation I used my own depth video, I also tried the video with the original InfiniTAM which worked pretty well, but since the movement of my object is not purely rigid, the original InfiniTAM can't handle all its motion.

I have been reading your code for a while but since I can't debug the code line by line, it's difficult for me to locate the tracker setting :(

[Algomorph]

@PascalYIN , the tracker settings are in this same file here: https://github.com/Algomorph/InfiniTAM/blob/ac2f3f7db90dac14469e5e3744827078e97cb906/InfiniTAM/ITMLib/Utils/ITMLibSettings.cpp#L97 I've inherited this from the original InfiniTAM authors, so it isn't quite "my code".

The results look much better than I got presumably with the same code a few weeks back... My curiosity is piqued now... Unfortunately I don't have the time right now to investigate. I'm working on a different codebase. Which, by the way, will feature a full SDF-2-SDF rigid pipeline in the near future. My colleague has already written the 2D version of the algorithm, I just recently closed that PR, actually.

https://github.com/Algomorph/LevelSetFusion-CPP https://github.com/Algomorph/LevelSetFusion-Python https://github.com/Algomorph/LevelSetFusion-Python/projects/1

[PascalYIN]

@Algomorph I see. Anyway, if you are interested I can give you more details. I compiled your code in an entirely new Ubuntu 18.04 and I have noted down every operation until the code was compiled in case that I needed to rebuild the environment.

I am willing to make some changes in your code for my own experiments, would you mind telling me how did you construct the data term and all by telling me where to find the implementation ?

By the way, I just looked through your python version implementation for the LevelSetFusion and it seems that PyCharm couldn't find level_set_fusion_optimization.

[Algomorph]

I am willing to make some changes in your code for my own experiments, would you mind telling me how did you construct the data term and all by telling me where to find the implementation ?

Sure, any improvements / experimental changes are welcome. To begin, take a look at ITMDenseDynamicMapper, ITMSceneMotionTracker, and ITMDynamicSceneReconstructionEngine classes, in that order. They pretty much entail the whole algorithm. ITMDenseDynamicMapper is the top-level overseer for the non-rigid optimization (some other things, like rigid reconstruction, are called from ITMDynamicEngine), most of what you need is in InitializeProcessing, ProcessFrame, and FinalizeProcessing, which are called on every frame. The ITMSceneMotionTracker is responsible for the iterative optimization, while the ITMDynamicSceneReconstructionEngine is responsible for the fusion part itself. The design may be a little convoluted and non-optimal, since I was trying to fit into the existing InfiniTAM framework as much as possible, which wasn't designed with non-rigid fusion in mind.

By the way, I just looked through your python version implementation for the LevelSetFusion and it seems that PyCharm couldn't find level_set_fusion_optimization.

Yeah, you need to do git submodule init and 'git pull --recur' in the root directory of the checked-out repo, and then build & install the CPP module, unless you want to prune off all the C++ stuff. I suppose I should add that into the README for newcomers. Everything is somewhat raw since we've been just focusing on our own development to begin with, before the library has obvious utility to others.

[Algomorph]

CMake Error at cmake/SetCUDAAppTarget.cmake:18 (add_executable): Target "InfiniTAM" links to target "Qt4::QtCore" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

I have encountered a similar issue just now when I was trying to statically link VTK, I fiddled around with the VTK components specified in CMake's find_package, and that seemed to resolve the issue. I pushed that to head.

I compiled the app w/ gcc-5 (other things built with gcc-7, mostly), still gives me result somewhat worse than yours:

I'll investigate further and write some tests to see if I can catch that CPU parallelization bug.

[PascalYIN]

I just think about one thing, were you using the Snoopy calibration test file? I didn’t have this calibration file so I was using the default calibration file of the code, maybe that’s where the difference came from. By the way I have been wondering if different calibration information can make remarkable difference of the reconstruction results.

[Algomorph]

I just think about one thing, were you using the Snoopy calibration test file?

The one Mira used, supposedly, during taking of the sequence is [here on her website].(http://campar.in.tum.de/personal/slavcheva/deformable-dataset/data/intrinsics_kinect1.txt) Seems like a pretty rough calibration to me, perhaps read-in from Kinect factory defaults (unlikely that round integer values are high precision), but that's the best we have...

I just changed it into the native InfiniTAM format like so: snoopy_calib.txt

Yes, the calibration can make a huge difference. Hell, a colleague of mine even wrote a depth-calibration utility for the Kinect that compiles a 3D lookup table to remove systematic error.

I'll try without calibration.

[Algomorph]

I've pushed another commit just now to the repo, it fixes up a few utils that should make testing possible (set BUILD_TESTS to ON in CMake). Now there is a way to make arbitrary-size PlainVoxelArray scenes (previously, the size was fixed to 512^3, which may be too large for adequate unit tests). Also, there are utilities for setting voxels manually & copying scenes, which is also critical for unit tests.

The only thing that should probably added for testing is implementation of the equality/unequality operators for ITMScene, and potentially some ways to compare the scene contents between PlainVoxelArray & VoxelBlockHash scenes.

Unfortunately, my advisor put a lid on this at the time, placing priority on another task. Hopefully, with these things you'll have enough to start writing more serious tests of your own to see if you can narrow down the parallelization bugs and such things.

[Algomorph]

I didn’t have this calibration file so I was using the default calibration file of the code, maybe that’s where the difference came from.

By the way, what exactly do you mean by "default calibration file of the code"? I see Files/PrimeSense/calib.txt and Files/Teddy/calib.txt, which one?

[Algomorph]

Good news, a build from Aug 13 / 2018 yielded a much better result with the following parameters:

./InfiniTAM_bpo "snoopy/snoopycalib.txt" "snoopy/frames/color%06i.png" "snoopy/frames/depth%06i.png" "snoopy/frames/omask%06i.png" --output debug_output/snoopy --start_from_frame_ix 16 --process_N_frames 50 --record_reconstruction_video --max_iterations 300 --weight_data_term 2.0

Code is in the branch I pushed as "Aug_13".

P.S. A slightly newer version in branch feature/DynamicFusion_Aug17 is a bit cleaner and doesn't generate pesky folders, but yields results that look just as good as Aug_13 version.

[PascalYIN]

@Algomorph I didn't specify any calibration file, which means ./InfiniTAM_bpo "" ~/data/colorfile ~/data/depthfile ~/data/maskfile in command line.

I see, I will try these 2 branches. Thanks for all the helps.

[Algomorph]

FYI: Just pushed to the main feature/DynamicFusion branch w/o the series of commits that messed things up, but with the revival of tests that I added. I pushed the experimental changes that were causing bad results to a separate branch just in case.

[PascalYIN]

I got these errors when I tried to compile the Aug_13 branch:

../../ITMLib/libITMLib.a(DenseDynamicMapperInstantiation.cpp.o): In function ITMLib::ITMDenseDynamicMapper<ITMVoxel_f_dynamic_canonical, ITMVoxel_f_dynamic_live, ITMLib::ITMPlainVoxelArray>::ResetCanonicalScene(ITMLib::ITMScene<ITMVoxel_f_dynamic_canonical, ITMLib::ITMPlainVoxelArray>*) const': DenseDynamicMapperInstantiation.cpp:(.text._ZNK6ITMLib21ITMDenseDynamicMapperI28ITMVoxel_f_dynamic_canonical23ITMVoxel_f_dynamic_liveNS_18ITMPlainVoxelArrayEE19ResetCanonicalSceneEPNS_8ITMSceneIS1_S3_EE[_ZNK6ITMLib21ITMDenseDynamicMapperI28ITMVoxel_f_dynamic_canonical23ITMVoxel_f_dynamic_liveNS_18ITMPlainVoxelArrayEE19ResetCanonicalSceneEPNS_8ITMSceneIS1_S3_EE]+0x40): undefined reference toITMLib::ITMSceneManipulationEngine_CUDA<ITMVoxel_f_dynamic_canonical, ITMLib::ITMPlainVoxelArray>::ResetScene(ITMLib::ITMScene<ITMVoxel_f_dynamic_canonical, ITMLib::ITMPlainVoxelArray>)' ../../ITMLib/libITMLib.a(DenseDynamicMapperInstantiation.cpp.o): In function `ITMLib::ITMDenseDynamicMapper<ITMVoxel_f_dynamic_canonical, ITMVoxel_f_dynamic_live, ITMLib::ITMPlainVoxelArray>::ResetLiveScene(ITMLib::ITMScene<ITMVoxel_f_dynamic_live, ITMLib::ITMPlainVoxelArray>) const': DenseDynamicMapperInstantiation.cpp:(.text._ZNK6ITMLib21ITMDenseDynamicMapperI28ITMVoxel_f_dynamic_canonical23ITMVoxel_f_dynamic_liveNS_18ITMPlainVoxelArrayEE14ResetLiveSceneEPNS_8ITMSceneIS2_S3_EE[_ZNK6ITMLib21ITMDenseDynamicMapperI28ITMVoxel_f_dynamic_canonical23ITMVoxel_f_dynamic_liveNS_18ITMPlainVoxelArrayEE14ResetLiveSceneEPNS_8ITMSceneIS2_S3_EE]+0x40): undefined reference to `ITMLib::ITMSceneManipulationEngine_CUDA<ITMVoxel_f_dynamic_live, ITMLib::ITMPlainVoxelArray>::ResetScene(ITMLib::ITMScene<ITMVoxel_f_dynamic_live, ITMLib::ITMPlainVoxelArray>*)' collect2: error: ld returned 1 exit status Apps/InfiniTAM/CMakeFiles/InfiniTAM.dir/build.make:224: recipe for target 'Apps/InfiniTAM/InfiniTAM' failed make[2]: *** [Apps/InfiniTAM/InfiniTAM] Error 1 CMakeFiles/Makefile2:148: recipe for target 'Apps/InfiniTAM/CMakeFiles/InfiniTAM.dir/all' failed

Is this normal ?

[Algomorph]

@PascalYIN try feature/DynamicFusion_Aug17, I just cleaned and rebuilt it from scratch, it worked for me.

[Algomorph]

FYI: Just pushed to the main feature/DynamicFusion branch w/o the series of commits that messed things up, but with the revival of tests that I added. I pushed the experimental changes that were causing bad results to a separate branch just in case.

My bad, forgot to push. Now this ^ is done, so you can just do a hard reset on the feature/DynamicFusion branch back something like 30 commits and then pull again, then rebuild. feature/DynamicFusion at this point is pointing to the same commit as feature/DynamicFusion_Aug17.

[PascalYIN]

I got the same reconstruction result with https://github.com/Algomorph/InfiniTAM/issues/138#issuecomment-490665725. But I still have no idea why it's so hard for your code to reconstruct the back of the object.

I also find an error of the sdf hessian computation at line 415, col 32 of ITMSceneMotionTracker_shared.h, should be x instead of y

May I ask you how the weight of each SDF voxel was decided?

[Algomorph]

@PascalYIN,

I got the same reconstruction result with #138 (comment). But I still have no idea why it's so hard for your code to reconstruct the back of the object.

1) My current intuition about the problems with reconstruction on the Snoopy sequence are as follows: the SDF-2-SDF rigid tracker produced better results on it, for whatever reason, then the built-in InfiniTAM tracker that I was last using. If the rigid tracking is off, it starts to wreak havoc with the non-rigid / dynamic alignment, and eventually the effects compound over time. It may very well be that different areas of the object are affected differently, depending on how well they are aligned with the rigid tracker.

I recommend to run the CPU / hash version on the whole Snoopy sequence to see how bad things really are.

In order to determine how bad the rigid tracking is, I can think of a few things. One is to output heatmaps of TSDF grid slices at around the middle of snoopy, in the XY plane (horizontal plane), of canonical TSDF grid and live TSDF grid, before non-rigid reconstruction. Preferably, you'd want to see also the TSDF grid build from depth frame before rigid alignment, i.e. "Tracking" in InfiniTAM takes place, and for that you'd need to use the reconstruction engine ahead of time somehow.

Then you can visually compare how well the canonical is rigidly aligned to the "live"/incoming frame.

2) Using the built-in bilateral filter might actually help with the back areas. The back of Snoopy is reconstructed using very scarce, bad data that we get from around silhouettes on the depth image. Kinect-like sensors (and moth range sensors) give notoriously bad quality in those areas. If you've read the original KinectFusion paper from '11, you might recall that the authors simply filtered those regions out before using the rest of the data (which also isn't hard, and a lot of people do this).

I also find an error of the sdf hessian computation at line 415, col 32 of ITMSceneMotionTracker_shared.h, should be x instead of y

thanks for catching the bug, I'll take a look soon!

May I ask you how the weight of each SDF voxel was decided?

What do you mean by "weight of each SDF voxel"?

[Algomorph]

P.S. regarding (1), I wrote are a bunch of utilities for exporting TSDF slices as images already, as well as some for exporting 3D TSDF data. Those can be used for visual debugging of the rigid alignment.

[PascalYIN]

@Algomorph in the function fuseLivevoxelIntoCanonical, the weight of each voxel is used to update the voxel's new SDF value: `int oldWDepth = canonicalVoxel.w_depth; float oldSdf = TVoxelCanonical::valueToFloat(canonicalVoxel.sdf);

float newSdf = oldWDepth * oldSdf + liveSdf;
float newWDepth = oldWDepth + 1.0f;
newSdf /= newWDepth;
newWDepth = MIN(newWDepth, maximumWeight);`

But I don't know how the beginning value of this weight is decided

[PascalYIN]

Regarding (1) I think I do have seen some of the utilities somewhere in the repository, can't remember where to find though. To be exact, what kind of data do these utilities export?

[Algomorph]

But I don't know how the beginning value of this weight is decided

The w_depth value is always set to zero using the ResetScene method of the SceneManipulationEngine after the scene is constructed. Previously this method was in the reconstruction engines, I think, and I'm not sure I delegated it to the new SceneManipulationEngine code everywhere, but the code is the same. It just goes through the voxels in the scene, however it happens to be indexed, and sets them to zero.

[Algomorph]

Regarding (1) I think I do have seen some of the utilities somewhere in the repository, can't remember where to find though. To be exact, what kind of data do these utilities export?

Take a look at ITMSceneVisualizer2D and its uses. There are two sets of methods -- one set for just rasterizing a scene slice on an OpenCV image (scaled up) and saving it, another (using the first) to save the entire scene as 2D slices along one or all of the axes. I think adapting the first set could be useful for this. Output is just a 255-bit image.

There is a "focus coordinates" thing in the settings singleton, I used that to highlight a specific voxel to see what's happening with it.

[PascalYIN]

Hi @Algomorph , I have a question about your Killing term gradient implementation, https://github.com/Algomorph/InfiniTAM/blob/e55edbb12b8d299a27158c774dd285eaec150e07/InfiniTAM/ITMLib/SceneMotionTrackers/Shared/ITMCalculateWarpGradientBasedOnWarpedLiveFunctor.h#L265

It doesn't seem like the given formula(9) in the KillingFusion parper though,

Why is the Killing term gradient implemented like this ?

[Algomorph]

@PascalYIN , very good question! I had a long back-and-forth with Mira about that. They did their implementation initially by combining the conventional Tikhonov term with a whole separate Killing term -- and, somehow, it worked, I'm not sure how -- then they wrote the paper and discovered they're doing unnecessary work (see where they say "the first part is just the regular Tikhonov" or something like that). And then, they forgot to fix up the formulas for the energy gradients.

So if you acutally use Euler-Lagrange equations to find the functional derivatives of their energy, it won't equal to what's in the paper or in the supplementary materials.

So yeah, my implementation features the correct solution to what's in the paper, as confirmed by Mira herself. I ran some experiments back in the day, it seemed like it actually worked. You're welcome to tweak and experiment, though.

Lemme see if I can dig up the PDF of the solution for just one of the directions of the gradient for you... Here you go (the 2s can be factored out from the bottom line): EulerLagrange.pdf

[Algomorph]

@PascalYIN , there are also several other people who expressed interest in continuing this implementation very recently, you can check around in different issues / discussions on this tracker. I highly recommend that all of you guys get in touch and collaborate on this -- if you organize well, you can avoid much communication overhead.

For my part, I intend to get back and fix a few things up here some time in the second half of July. I have my PhD proposal defense coming up in 3 days and then a long-awaited vacation. In the meantime, I'll continue to be in touch as much as possible before I leave on my road-trip, if I can be of any assistance.