segmentation

[HJMGARMIN]

Great work! How can I make the same segmentation results on TUM dataset? Which param should I change in the yaml file? The second picture is my result on "rgbd_dataset_freiburg3_long_office_household" with PointCloudMapping.type: "octree_point" and Segmentation.on: 1

[luigifreda]

Hi, I am glad you liked PLVS. Thanks.

As you understood, to enable segmentation, you need to set PointCloudMapping.type: "octree_point" and Segmentation.on: 1 These are the parameters I normally use

This is what I normally get (in an intermediate frame) as shown in the videos

[HJMGARMIN]

Thanks for your reply, but it seems does not work on my device. And I notice that the color of the same objects in "connected components" is changing. My yaml file of TUM3 is below: `%YAML:1.0

--------------------------------------------------------------------------------------------

Viewer Parameters

--------------------------------------------------------------------------------------------

Viewer.on: 1 is ON, 0 is OFF

Viewer.on: 1

Viewer.KeyFrameSize: 0.05 Viewer.KeyFrameLineWidth: 1.0 Viewer.GraphLineWidth: 0.9 Viewer.PointSize: 2.0 Viewer.LineSize: 1.0 Viewer.CameraSize: 0.08 Viewer.CameraLineWidth: 3.0 Viewer.ViewpointX: 0.0 Viewer.ViewpointY: -0.7 Viewer.ViewpointZ: -1.8 Viewer.ViewpointF: 500.0

Viewer.ViewpointYtop: -4.0

--------------------------------------------------------------------------------------------

Camera Parameters. Adjust them!

--------------------------------------------------------------------------------------------

File.version: "1.0"

Camera.type: "PinHole"

Camera calibration and distortion parameters (OpenCV)

Camera1.fx: 535.4 Camera1.fy: 539.2 Camera1.cx: 320.1 Camera1.cy: 247.6

Camera1.k1: 0.0 Camera1.k2: 0.0 Camera1.p1: 0.0 Camera1.p2: 0.0

Camera.width: 640 Camera.height: 480

Camera frames per second

Camera.fps: 30

Color order of the images (0: BGR, 1: RGB. It is ignored if images are grayscale)

Camera.RGB: 1

Close/Far threshold. Baseline times.

Stereo.ThDepth: 40.0 Stereo.b: 0.0747

Depth map values factor

RGBD.DepthMapFactor: 5000.0

--------------------------------------------------------------------------------------------

ORB Parameters

--------------------------------------------------------------------------------------------

ORB Extractor: Number of features per image

ORBextractor.nFeatures: 1000

ORB Extractor: Scale factor between levels in the scale pyramid

ORBextractor.scaleFactor: 1.2

ORB Extractor: Number of levels in the scale pyramid

ORBextractor.nLevels: 8

ORB Extractor: Fast threshold

Image is divided in a grid. At each cell FAST are extracted imposing a minimum response.

Firstly we impose iniThFAST. If no corners are detected we impose a lower value minThFAST

You can lower these values if your images have low contrast

ORBextractor.iniThFAST: 20 ORBextractor.minThFAST: 7

--------------------------------------------------------------------------------------------

Line Segments Tracking

--------------------------------------------------------------------------------------------

Line.on: 1 is ON, 0 is OFF

Line.on: 1

set to 0 if keeping all lines

Line.nfeatures: 100

number of levels in the scale pyramid

Line.nLevels: 3

scale factor between levels in the scale pyramid

Line.scaleFactor: 1.2

noise standard deviation on line extraction (in pixel) CHANGED!

Line.sigma: 2

min. line length (relative to img size)

Line.minLineLength: 0.02

weighting of lines w.r.t. points in success/failure tracking conditions

Line.lineTrackWeigth: 2

min. line length 3D (meters)

Line.minLineLength3D: 0.01

mu weight for endpoint-endpoint dist in line 3D dist

Line.muWeightForLine3dDist: 0.3

for using LSD extractor (much slower) instead of EDLine lines extractor

Line.LSD.on: 0

--------------------------------------------------------------------------------------------

Map Object

--------------------------------------------------------------------------------------------

MapObject.on: 1 is ON, 0 is OFF

MapObject.on: 0

Images of the planar objects (separated by empty spaces)

MapObject.imgs: "teddy.png" MapObject.numMinInliers: 20 MapObject.maxReprojectionError: 1 MapObject.maxSim3Error: 0.01

--------------------------------------------------------------------------------------------

KeyFrame Generation

--------------------------------------------------------------------------------------------

on: 1 is ON, 0 is OFF

KeyFrame.fovCentersBasedGeneration.on: 0 KeyFrame.maxFovCentersDistance: 0.4

--------------------------------------------------------------------------------------------

Sparse Mapping

--------------------------------------------------------------------------------------------

SparseMapping.filename: "sparse_map_rgbd.atlas"

use saved map on start: 1 is ON, 0 is OFF

SparseMapping.reuseMap: 0

freeze map during optimizations: 1 is ON, 0 is OFF

SparseMapping.freezeMap: 0

save map on shutdown: 1 is ON, 0 is OFF

SparseMapping.saveMap: 0

force immediate relocalization (or wait for loop-closing thread for relocalization): 1 is ON, 0 is OFF

SparseMapping.forceRelocalization: 1

--------------------------------------------------------------------------------------------

Depth Noise Model

--------------------------------------------------------------------------------------------

used for scaling depth noise model

Depth.sigmaZfactor: 6

--------------------------------------------------------------------------------------------

PointCloud Mapping

--------------------------------------------------------------------------------------------

PointCloudMapping.on: 1 is ON, 0 is OFF

PointCloudMapping.on: 1

PointCloudMapping.type: voxelgrid, octomap, octree_point, chisel, fastfusion, voxblox

NOTE: chisel and fastfusion require rectified images if you do not use pointcloud generation

PointCloudMapping.type: "octree_point"

PointCloudMapping.resolution: 0.01 PointCloudMapping.numKeyframesToQueueBeforeProcessing: 1 PointCloudMapping.downSampleStep: 2 PointCloudMapping.minDepth: 0.1 PointCloudMapping.maxDepth: 5 PointCloudMapping.removeUnstablePoints: 1

PointCloudMapping.resetOnSparseMapChange: 1 PointCloudMapping.cloudDeformationOnSparseMapChange: 0

[octree_point] specific params

PointCloudMapping.pointCounterThreshold: 3

[voxblox] specific params

PointCloudMapping.voxbloxIntegrationMethod: "fast"

[octree_point, chisel, voxblox] specific params

PointCloudMapping.useCarving: 1 is ON, 0 is OFF

PointCloudMapping.useCarving: 0

depth filtering

PointCloudMapping.filterDepth.on: 0 PointCloudMapping.filterDepth.diameter: 7 PointCloudMapping.filterDepth.sigmaDepth: 0.02 PointCloudMapping.filterDepth.sigmaSpace: 5

for loading

load dense map on start: 1 is ON, 0 is OFF

PointCloudMapping.loadMap: 0

voxblox also accepts its .proto file!

PointCloudMapping.loadFilename: "volumetric_map_0.ply"

--------------------------------------------------------------------------------------------

Segmentation

--------------------------------------------------------------------------------------------

Segmentation.on: 1 is ON, 0 is OFF

working only with [octree_point]

Segmentation.on: 1

Segmentation.maxDepth: 2

the following params work well with PointCloudMapping.downSampleStep: 2

Segmentation.erosionDilationOn: 1 Segmentation.minFi: 0.92 Segmentation.maxDelta: 0.01 Segmentation.singleDepth.minComponentArea: 80 Segmentation.lineDrawThinckness: 2 Segmentation.maxAngleForNormalAssociation: 20 Segmentation.labelsMatchingMinOverlapPerc: 0.2 Segmentation.labelsMatchingMinOverlapPoints: 0 Segmentation.globalLabelsMatchingMinOverlapPerc: 0.2

--------------------------------------------------------------------------------------------

Gl Objects

--------------------------------------------------------------------------------------------

pre-load static GL objects to be visualized in the map

GlObjects.on: 0

GlObjects.list:

- { filename: "map_0.ply", type: "cloudmesh", pose: [ 0, 0, 0, 0, 0, 0] }

- { filename: "sassi.ply", type: "cloudmesh", pose: [ 0, 0, 0, 0, 0, 0] }

` and my results are below. The segmentation results are worse than yours. The colors split on the same object are very messy.

[luigifreda]

Hi, Thanks for sharing your pics. I have some questions for you:

• What is your exact computer configuration? (OS, CPU, GPU, GCC version)
• Are you under docker, virtual box, or in a native OS?
• Which GPU are you using?
• How did you exactly install PLVS?
• How are you exactly running the test? Could you please attach the used script file and configuration file? (please, attach the file and do not copy-paste the text)
• Did you change any part of the code? Could you please run git diff >& /tmp/log.txt and share the generated log file as an attachment?
• Did you touch any GUI slider before making the snapshots?

I tested PLVS on different laptops and desktop computers (with NVIDIA GPUs). I have never seen such a result ... unless you change the parameters.

It seems you are also getting a different volumetric map with much fewer points than normal. Can you share a pic of the volumetric map you are getting?

[HJMGARMIN]

OS: Ubuntu 18.04 CPU: i7-11700 GPU: RTX2080 I installed PLVS using the script you perovided. Because I can not upload .sh file, I comvert them to .txt. run_tum_rgbd.txt config.txt I did not change any part of the main code. log.txt I just open "Show Segments" in the GUI and do not touch other GUI slider. The volumetric map on my computer is : When I run PLVS with segments, the state of GPU is as below.

[luigifreda]

I see you are using Ubuntu 18. I tested the procedures and the code under Ubuntu 20.

Can you test the code under docker with Ubuntu 20? For instance, you can use https://github.com/luigifreda/rosdocker

[luigifreda]

Update. I got PLVS working back on Ubuntu 18. Indeed, i was not able to build PLVS with C++17 under Ubuntu 18. I am wondering how you got it built with C++17?? (You didn't reply to my GCC question) Moreover, you are not using a local opencv install. Please, install a local opencv.

What I recommend on Ubuntu 18:

• Revert your changes in config.sh at root level of the repository (we want to use the local opencv install)
• Go into config.sh at root level of the repository and set CPP_STANDARD_VERSION=11 (here https://github.com/luigifreda/plvs/blob/master/config.sh#L14)
• Clean everything by running ./clean.sh
• Install a local opencv by running ./install_local_opencv.sh (as suggested in the main README)
• Run ./build.sh (as suggested in the main README)

[D1st3f]

I have bad result form segmentation:

And when segmentation off:

Also: PointCloudMapping.type: "octree_point" Segmentation.on: 1

[D1st3f]

Another GUI screenshots: P.S. Tried without touching sliders in GUI, same result.

[D1st3f]

Look's like something broken i normals

[CQnancy]

In file included from /home/techinao/catkin_ws/src/plvs/Thirdparty/volumetric_mapping/volumetric_mapping/volumetric_map_base/src/world_base.cc:40:0: /usr/include/pcl_conversions/pcl_conversions.h:42:10:fatal error: ros/ros.h: No such file or directory。Do you have this question? If so, how was it resolved?

[luigifreda]

Did you run the script install_dependencies.sh? What is your OS and configuration?

[CQnancy]

Did you run the script install_dependencies.sh? What is your OS and configuration?

I ran it one by one./clean. sh, ./install_dependency. sh, ./Install_local_opencv.sh and./build.sh, Ubuntu 18.04.6

[luigifreda]

Hey @CQnancy, thanks for your feedback. I worked to improve PLVS support under Ubuntu 18.04. You can now pull, run clean.sh, install deps, and repeat the build procedure (no need to reinstall local opencv if you already did it).