Closed bigbigpark closed 2 years ago
MaxChanger
commented
2 years ago Thanks, I think a simple way is write scripts to convert your own data into the same file structure as SemanticKITTI, .pcd to .bin, and provide the corresponding intrinsic parameters and trajectory file.
bigbigpark
commented
2 years ago Thanks for replying, in the below image,
You mean I should put my data(as a format of .bin) into scan_folder in the line 18.
My data has a number of pcd file not including camera so is the intrinsic parameter identity matrix at calib_file?
Last, how about pose_file? I don't know the ground truth pose in my data
MaxChanger
commented
2 years ago Hi, the data structure folder is shown in the following. Sorry, I typed a wrong word, it should be the extrinsics of camera_lidar in calib.txt, and the intrinsic parameter of camera is not needed here. If I understand correctly, you can simply run a LiDAR SLAM to get the pose/trajectory, and then give the camera_lidar's extrinsics parameter to a identity matrix.
DATAROOT
├── sequences
│ └── 08
│ ├── calib.txt # calibration file provided by KITTI
│ ├── poses.txt # ground truth poses file provided by KITTI
│ ├── velodyne # velodyne 64 LiDAR scans provided by KITTI
│ │ ├── 000000.bin
│ │ ├── 000001.bin
│ │ └── ...
│ ├── labels # ground truth labels provided by SemantiKITTI
│ │ ├── 000000.label
│ │ ├── 000001.label
│ │ └── ...
│ └── residual_images_1 # the proposed residual images
│ ├── 000000.npy
│ ├── 000001.npy
│ └── ...I'm actually pretty new about KITTI dataset
In calib.txt, I don't understand what is P0, P1, P2, P3, Tr
I have searched a lot, but I couldn't get it.
I think Tr is transformation from lidar to camera coordinates so shoud I put them a identity matrix [R | t] ?
like r11=1, r22=1, r33=1, tx=0, ty=0, tz=0 !
I can't understand the meaning about P0~P4
Could you help me?
MaxChanger
commented
2 years ago Hi @bigbigpark, you are right, and there are some docs to reference. https://github.com/alexkreimer/odometry/blob/master/devkit/readme.txt https://github.com/yanii/kitti-pcl/blob/master/KITTI_README.TXT
I have tested new datasets that I have collected before, and it doesn't matter without a camera. Therefore, you don't have to worry about P0-P3. If you want to know more, you can refer to the readme above. A simple way is pose.txt is the trajectory of LiDAR, and then let Tr be the identity matrix, as follow, do not need to fill in P*.
# calib.txt only need one line as follow
Tr: 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0Another thing to note is that the pose file is best to use the first frame as the world system (I haven't verified that not doing this will make a difference), that is, the first line in poses.txt should be R=identity t=0, 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0, I think you could know what this operation meaning and how to do this.
bigbigpark
commented
2 years ago Thank you for your prompt reply!
As you mentioned above, I prepared the all required data except for label file
Here is what I have done
# calib.txt only need one line as follow
Tr: 1.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0$ python3 utils/gen_residual_images.pySo I can get .npy file in residual_images_1 and .png file in visualization_1
$ cd mos_SalsaNext/train/tasks/semantic
$ python3 infer.py -d ../../../../data -m ../../../../data/model_salsanext_residual_1 -l ../../../../data/predictions_salsanext_residual_1_new -s validthe error occured. The log is here
----------
INTERFACE:
dataset ../../../../data
log ../../../../data/predictions_salsanext_residual_1_new
model ../../../../data/model_salsanext_residual_1
Uncertainty False
Monte Carlo Sampling 30
infering valid
----------
----------
Opening arch config file from ../../../../data/model_salsanext_residual_1
Opening data config file from ../../../../data/model_salsanext_residual_1
train 00
train 01
train 02
train 03
train 04
train 05
train 06
train 07
train 09
train 10
valid 08
test 11
test 12
test 13
test 14
test 15
test 16
test 17
test 18
test 19
test 20
test 21
model folder exists! Using model from ../../../../data/model_salsanext_residual_1
Sequences folder exists! Using sequences from ../../../../data/sequences
parsing seq 08
Traceback (most recent call last):
File "infer.py", line 144, in <module>
user = User(ARCH, DATA, FLAGS.dataset, FLAGS.log, FLAGS.model,FLAGS.split,FLAGS.uncertainty,FLAGS.monte_carlo)
File "../../tasks/semantic/modules/user.py", line 54, in __init__
shuffle_train=False)
File "../..//tasks/semantic/dataset/kitti/parser.py", line 470, in __init__
gt=self.gt)
File "../..//tasks/semantic/dataset/kitti/parser.py", line 196, in __init__
assert(len(scan_files) == len(label_files))
AssertionErrorI don't know how I get to the label_files!
MaxChanger
commented
2 years ago Hello @bigbigpark, Glad to hear positive feedback.
I'm guessing you put the new LiDAR data under the seq08 folder?
This happens because gt=True in __init__ of SemanticKITTI class, if self.split == 'valid'.
https://github.com/PRBonn/LiDAR-MOS/blob/c638e8b5cecc26b4b11c4421aa37f9c51acddc87/mos_SalsaNext/train/tasks/semantic/dataset/kitti/parser.py#L461-L491
If you run the command python infer.py ...... with flage -s valid, it should be checked whether the number of LiADR bins and the number of labels are equal.
If you just want to inference based on the checkpoint provided by the author, then I suggest setting the folder id of the new data store to 40 41 42 ..... (maybe others). And modify the data_cfg.yaml file in the corresponding checkpoint folder, replace the seq_id corresponding to the test key inside with 40 41 .... (replace the 11~21) , and then use python infer.py ...... with flage -s test.
MaxChanger
commented
2 years ago Hi, @bigbigpark. Is this solution possible? Hope to get your feedback.
bigbigpark
commented
2 years ago Hi, @MaxChanger. Sorry for being late. I have had difficulty with CUDA version and pytorch.
As you mentioned before, I modify data_cfg.yaml file like below picture.
And in 40 folder,
So when I type that command, an error occured even though I change the flage from -s vailid to -s test.
$ python3 infer.py -d ../../../../data -m ../../../../data/model_salsanext_residual_1 -l ../../../../data/predictions_salsanext_residual_1_new -s testThe error log is here,
----------
INTERFACE:
dataset ../../../../data
log ../../../../data/predictions_salsanext_residual_1_new
model ../../../../data/model_salsanext_residual_1
Uncertainty False
Monte Carlo Sampling 30
infering test
----------
----------
Opening arch config file from ../../../../data/model_salsanext_residual_1
Opening data config file from ../../../../data/model_salsanext_residual_1
train 00
train 01
train 02
train 03
train 04
train 05
train 06
train 07
train 09
train 10
valid 08
test 40
model folder exists! Using model from ../../../../data/model_salsanext_residual_1
Sequences folder exists! Using sequences from ../../../../data/sequences
parsing seq 40
Using 1999 scans from sequences [40]
Depth of backbone input = 6
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
********************************************************************************
Cleaning point-clouds with kNN post-processing
kNN parameters:
knn: 5
search: 5
sigma: 1.0
cutoff: 1.0
nclasses: 3
********************************************************************************
Infering in device: cuda
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py:166: RuntimeWarning: invalid value encountered in true_divide
pitch = np.arcsin(scan_z / depth)
Traceback (most recent call last):
File "infer.py", line 145, in <module>
user.infer()
File "/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../tasks/semantic/modules/user.py", line 112, in infer
self.infer_subset(loader=self.parser.get_test_set(),
File "/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../tasks/semantic/modules/user.py", line 134, in infer_subset
for i, (proj_in, proj_mask, _, _, path_seq, path_name, p_x, p_y, proj_range, unproj_range, _, _, _, _, npoints) in enumerate(loader):
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in __next__
data = self._next_data()
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1347, in _next_data
return self._process_data(data)
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
data.reraise()
File "/home/scpark/.local/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/scpark/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "../..//tasks/semantic/dataset/kitti/parser.py", line 285, in __getitem__
scan.open_scan(scan_file, index_pose, current_pose, if_transform=self.transform_mod)
File "/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py", line 105, in open_scan
self.set_points(points, remissions)
File "/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py", line 143, in set_points
self.do_range_projection()
File "/home/scpark/tools/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/../../common/laserscan.py", line 201, in do_range_projection
self.proj_range[proj_y, proj_x] = depth
IndexError: index -2147483648 is out of bounds for axis 0 with size 64What model is your LiDAR and how many lines are there?
Velodyne-64 is used in KITTI, so in salsanext_mos.yml, FOV is set to fov_up: 3, fov_down: -25and width and height are set to 64*2048, If the LiDAR model is different, please update here. In addition, if you need to retrain, you also need to update img_mean and img_std.
According to your error.log, it should happen during the projection process of LiDAR to RangeImage? This shouldn't have much to do with valid or test flag.
Make sure the projection is correct, it can be easily visualized. It is also worth mentioning that this projection relationship should also be required when data preprocessing generates residual image and visualization-residual-image. Is there any error there? You can simply check whether the visualization-residual-image visualization is correct
bigbigpark
commented
2 years ago I used VLP 16 lidar model.
There are no errors about generating residual images.
MaxChanger
commented
2 years ago ok, some releated issue https://github.com/PRBonn/LiDAR-MOS/issues/34 https://github.com/PRBonn/LiDAR-MOS/issues/19 (maybe most in chinese)
According to their experience, directly using the 64-line pre-training model, in 16-line inference, the results may not be very good. Please let me know if there is any progress. :-)
bigbigpark
commented
2 years ago Sure. I'll do.
You are one of the kindest people who ever answered me.
I have recently studied lidar-based segmantation and found your awesome work LiDAR-MOS.
Can you recommend some materials for studying about lidar-based segmentation ?
The purpose is mainly about removing dynamic object in point cloud.
Best regards.
MaxChanger
commented
2 years ago Hi @bigbigpark
Some recent paper (sorted by time):
For an earlier solution, you can actually refer to Sec. 2 Related Work of these work and LiDAR-MOS.
There are some open-source code I known:
bigbigpark
commented
2 years ago Thanks for replying as before :-)
I'll update my progress here!!
noobth1nker
commented
3 months ago @bigbigpark May I ask if you have continued with this job? I would like to know your latest progress.
Hi, I have read the paper and built-and-run your LiDAR-MOS.
Thanks for sharing your awesome projects here.
I have a question.
How can I use the code with my own dataset?
I'll use the pretrained model you uploaded, so I think all I need to do is make my data be appropriate format for the
LiDAR-MOS.The data I have consists of
.bagfile and.pcdfile.I'd appreciate for you to give me some advice.
Best regards.