Training issues - Githubissues

lzbushicai commented 9 months ago

Hello，Cao Anh Quan！！, I organized my data format into KITTI format and trained it, but I found that the relationship loss often appears as nan. sequences:['01'],frame_id:['002205'],**relation_loss: nan** sequences:['01'],frame_id:['002205'],loss_sem_scal: 5.660839080810547 sequences:['01'],frame_id:['002205'],loss_geo_scal: 2.1648528575897217 Warning: frustum_ nonempty is zero, and the division operation will be skipped or assigned a default value. sequences是:['01'],frame_id为:['002205'],**total_loss: nan** I guess this may be related to my data (I did not change the code for calculating the loss). In my tags, there are a particularly large number of 255, as shown in the following figure. May I ask if there is any special meaning or handling method for 255 in your design?
my dataset label distribution

Another issue is that due to the large size of my img, I do not have enough memory to train the network. Therefore, I resized the image to (450 720) here is my code , ` img = np.array(img, dtype=np.float32, copy=False) / 255.0 img_resized = Image.fromarray((img 255).astype(np.uint8)).resize((720, 450)) img = img[:450, :720, :] # crop image` I'm unsure whether this will substantially influence the outcomes of my training, and I would greatly appreciate your insights and guidance on this matter.

anhquancao commented 9 months ago

Hi @lzbushicai,

The 255 is automatically ignored in all the losses so I don't think they are the problem.
I suggest you to check it the loss is nan on some specific frames. Maybe there are problems with the labels in these frames.
When you resize your image, you need to change the cam_K accordingly. https://dsp.stackexchange.com/questions/6055/how-does-resizing-an-image-affect-the-intrinsic-camera-matrix

lzbushicai commented 9 months ago

Thank you for your suggestion. After resizing the image, I did not adjust the camera parameters.

lzbushicai commented 8 months ago

Hi!!Cao Anh Quan！！Hello, there are always many bugs in the training process. I re-examined the problem and found that there was always a problem in visualizing the voxel I made using the official kitti tool. When using the official kitti tool,, I fir /st converted the point cloud .bin file and label .label file. , processed into fixed-size .bin .label .invalid .occloud files. However, some sequences are always wrong. it is the problem of the lidar system in calib.txt turning to camera external parameters, but I can't get the correct result using the transformation matrix of my data set. Here are some error results： wrong_voxel1 wrong_voxel2

So I followed your suggestion and wrote a script to process the point cloud data .bin file and label data .label file. However, in this process, I did not use the camera’s internal and external parameters calib.txt and the pose of each scan. pose.txt, do I need to add any additional information to train the data processed by my script?Below is my script；

import numpy as np

# Load point cloud data and labels
points = np.fromfile(rellis_bin, dtype=np.float32).reshape(-1, 4)[:, :3]
labels = np.fromfile(rellis_label, dtype=np.uint32).astype(np.float32)

# Define the bounds and size of the voxels
x_min, x_max, y_min, y_max, z_min, z_max = 0, 51.2, -25.6, 25.6, -2, 4.4
voxel_size = 0.2

# Calculate the number of voxels along each axis
x_voxels = int((x_max - x_min) / voxel_size)
y_voxels = int((y_max - y_min) / voxel_size)
z_voxels = int((z_max - z_min) / voxel_size)

# Initialize the voxel label array
voxel_labels = np.zeros((x_voxels, y_voxels, z_voxels), dtype=np.uint32)

# Compute the voxel index for each point
indices = np.floor((points - np.array([x_min, y_min, z_min])) // voxel_size).astype(np.int32)

# Assign labels to each voxel
for i, (x, y, z) in enumerate(indices):
    if 0 <= x < x_voxels and 0 <= y < y_voxels and 0 <= z < z_voxels:
        label = labels[i]  # Corrected variable name for clarity
        voxel_labels[x, y, z] = label  # Modify here to implement label statistics and selection logic

# Save the voxel_labels array, which contains the label for each voxel; unassigned voxels have a label of 0
np.save("voxel_00_82.npy", voxel_labels)

anhquancao commented 8 months ago

Hello, your voxelization code appears to be correct. However, have you verified that your x, y, z axes align with those of Semantic KITTI?

Without using camera parameters, it's unclear how you would be able to aggregate the point cloud to construct a dense scene.

You will still require the calibration parameters to project the voxels onto the image.

lzbushicai commented 8 months ago

Hello, your voxelization code appears to be correct. However, have you verified that your x, y, z axes align with those of Semantic KITTI?

Without using camera parameters, it's unclear how you would be able to aggregate the point cloud to construct a dense scene.

You will still require the calibration parameters to project the voxels onto the image.

Thanks, I am projecting the point cloud with labels into the image to troubleshoot the problem. Check if the problem is with the KITTI script or with the Extrinsic Parameters

lzbushicai commented 8 months ago

Hi, I'm using my own script to voxelize the point cloud, but the points I get are particularly sparse (relative to the KITTI tool), can I aggregate the front and back frames to solve this problem? Here is the label distribution after processing by my script

   Label    Count
0      0  2092045
1      3     1120
2      4     1804
3     11     1832
4     17       31
5     18       52
6     19      268

Here is the label distribution after processing by KITTI tool

  Label    Count
0    0.0   148154
1    3.0    66652
2    4.0   135036
3   17.0      907
4   18.0     3499
5   19.0    12389
6  255.0  1730515

it's obvious that the file processed by my own script is very sparse. because the script processed by the kitti tool is problematic, and I don't think I can train it to produce good results due to the sparse points.

anhquancao commented 8 months ago

I think you need to aggregate the point clouds from many consecutive frames to have dense scenes not only the front and back.

lzbushicai commented 8 months ago

Hi!!Cao Anh Quan！ How long did it take you to train the KITTI model? Since my prediction categories went from 20 to 15, I couldn't use your pre-trained model, my model converged very slowly and had very low metrics。

epoch=000-val/:
'mIoU=0.01065.ckpt'
epoch=001-val/:
'mIoU=0.01568.ckpt'
epoch=002-val/:
'mIoU=0.01592.ckpt'
epoch=003-val/:
'mIoU=0.01336.ckpt'
epoch=004-val/:
'mIoU=0.01378.ckpt'
epoch=005-val/:
'mIoU=0.01774.ckpt'
epoch=006-val/:
'mIoU=0.01647.ckpt'
epoch=007-val/:
'mIoU=0.01532.ckpt'

sem_scal and geo_scal loss function don't converge after 32epochs， From a top view perspective, my voxels are shown in the following image 03_000275_1_1

Is this a problem with my data?

anhquancao commented 7 months ago

Hi, it takes 30 epochs to converge (https://github.com/astra-vision/MonoScene/blob/master/monoscene/scripts/train_monoscene.py#L50).

You should try overfitting a single example to see if your network can converge. Usually, this problem arises from an incorrect camera projection of voxels onto the image.

You should also visualize the projected voxels on the image to check if they are correct.