Failed to display segmentation result in the process of reconstruction

[CharlieLeee]

@martinruenz Hi Martin, thanks for the great work and clear instructions, I successfully build the system. However, when I ran the command: ./MaskFusion -run -l /home/charlie/Downloads/teddy-handover.klg

There are no segmentation results displayed on the screen. Could you help to point out the potential cause of this issue?

• output of the command

  
  Calibration set to resolution: 640x480, [fx: 528 fy: 528, cx: 320 cy: 240]
  Reading log file: /home/charlie/Downloads/teddy-handover.klg which has 528 frames. 
  Initialised MainController. Frame resolution is set to: 640x480
  Exporting results to: /home/charlie/Downloads/teddy-handover.klg-export//

• Initialising MaskRCNN (thread: 140646186743552) ...
  • Loading module... /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Created model with max number of vertices: 9437184 Initialised multi-object fusion (main-thread: 140647066766720)
• The background model can have up to 9437184 surfel (3072x3072)
• Object models can have up to 1048576 surfel (1024x1024)
• Using GPU unspecified for SLAM system and GPU 0 for MaskRCNN
• Using frame-queue of size: 30 2020-06-27 01:31:23.576753: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA Using TensorFlow backend. Your GPU "GeForce RTX 2060" isn't in the ICP Step performance database, please add it Your GPU "GeForce RTX 2060" isn't in the RGB Step performance database, please add it Your GPU "GeForce RTX 2060" isn't in the RGB Res performance database, please add it Your GPU "GeForce RTX 2060" isn't in the SO3 Step performance database, please add it

Configurations: BACKBONE resnet101 BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] COMPUTE_BACKBONE_SHAPE None DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.7 DETECTION_NMS_THRESHOLD 0.3 FPN_CLASSIF_FC_LAYERS_SIZE 1024 GPU_COUNT 1 GRADIENT_CLIP_NORM 5.0 IMAGES_PER_GPU 1 IMAGE_CHANNEL_COUNT 3 IMAGE_MAX_DIM 1024 IMAGE_META_SIZE 93 IMAGE_MIN_DIM 800 IMAGE_MIN_SCALE 0 IMAGE_RESIZE_MODE square IMAGE_SHAPE [1024 1024 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME coco NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 PRE_NMS_LIMIT 6000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (32, 64, 128, 256, 512) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TOP_DOWN_PYRAMID_SIZE 256 TRAIN_BN False TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001

• Initialised MaskRCNN
• MaskRCNN got first data -- starting loop. Segmentation fault (core dumped)

I also tried the Core/Segmentation/MaskRCNN/offline_runner.py script, and it worked pretty well. So the issue might not be related to mask rcnn and tf.

Btw, I would appreciate it if you could tell me a way to generate a semantic point cloud just like the one in your impressive video?

[martinruenz]

Hi Charlie

• There is a macro in MfSegmentation.cpp to enable debug visualizations. I recommend that you enable the visualizations in order to see which data goes in and out of the segmentation method. See: https://github.com/martinruenz/maskfusion/blob/02b232298d245fa5027ec60f5030ed7fab5e3c15/Core/Segmentation/MfSegmentation.cpp#L29
• Your log shows a segfault, did this happen when shutting down the software or before? This could be a potential cause of the issue.
• To run the segmentation online you need 2 GPUs, otherwise the performance will be ba (see readme).
• Check savePly to export objects: https://github.com/martinruenz/maskfusion/blob/7b8dd3252dc1dcf8a1d01e0a263cb4bcce0972da/Core/MaskFusion.cpp#L733

[CharlieLeee]

@martinruenz Hi Martin, thank you for replying!

The segmentation fault happened when I closed the GUI window.

I tried to run maskfusion offline, but seems like the system still runs the online mask rcnn since MaskRCNN got first data -- starting loop. still shows.

But I somehow got it working offline correctly by pressing ctrl +c in the terminal after mask rcnn starts running (it may not always work but it worked).

Sure, thank you for helping! I'll get back to you once I turn on the debug visualization and run again!

[ZhenghaoL]

@CharlieLeee Hi, i have the same problem with you, have you solved it now?

[CharlieLeee]

@ZhenghaoL Sadly, I haven't actually solved the problem, as I've said, the way I made this work is by pressing ctrl+c during the running of the program

[1171257311]

@CharlieLeee 你好，我也尝试了一下离线进行分割之后再运行而不是在线运行，但是发现效果还是不是很理想，比如物体移动的路径上都会留下来残影，如下图。不知道作者是如何达到视频里面的效果的。请问你有想法吗？？感谢。