Inference my own dataset

[stanny880913]

Hello , I want Inference my own dataset, image pair pcd file, I think I need to generate the same data_in like you to do predict , may I ask what self.radar_raw_list and self.radar_list mean in pcd file ,and is it necessary to create data_in, or maybr I can use other way to do inference my dataset? Thank you

[stanny880913]

Hello , I found that you resize image to [192,400,3] and feed the model, but how can radar align to resized image, my image is [712,1280,3], and I use your downsample_im function in prepare_flow_im.py, change my image from [712,1280,3] to nuscenes image shape [900,1600,3] and to your last shape [192,400,3], but what can I do next, can you give me some hint to deal radar data and concate together like data_in in data_loader_pda.py, thank you

Or maybe I don't even need to resize my photos, I can also Inference my dataset?

[longyunf]

There is no restriction on image resolutions. To align radar points with downsampled images, you can first project radar points to images of original resolutions and then map them to downsampled images via scaling and offset.

[stanny880913]

There is no restriction on image resolutions. To align radar points with downsampled images, you can first project radar points to images of original resolutions and then map them to downsampled images via scaling and offset.

So I don’t need to resize my image,the model can read 1280x712 image? And I want to create custom radar data to let mode load it,how can I create my own data_in ? what are the parameters in data_in mean .pcd file ? Thank you

[stanny880913]

load

Hello~ Do I really need to create my own mer_2_30_5_0.5.h5 andprepared_data.h5 to inference my own dataset ? thank you

[longyunf]

When you retrain the model, you need GT depths, which can be generated with Lidar measurement. You may also adjust radar features according to your radar data formats.

For inference, images and radar points are required as inputs. mer_2_30_5_0.5.h5 is intermediate data (enhanced radar depths).

[stanny880913]

When you retrain the model, you need GT depths, which can be generated with Lidar measurement. You may also adjust radar features according to your radar data formats.

For inference, images and radar points are required as inputs. mer_2_30_5_0.5.h5 is intermediate data (enhanced radar depths).

Hello , I don't have lidar data so I can't train, I only want to use your pre-trained model to inference, so is mer_2_30_5_0.5.h5 necessary？ can I use normal pcd radar file to inference? Im thinking what is the data type in prepared_data.h5 and I want to generate the same format, but i can't understand what is it!

[longyunf]

gen_h5_file.py saves inputs and labels in the prepared_data.h5.

cal_radar.py generates radar inputs, which include radar depth, RCS and Doppler velocity.

For inference, we need to infer enhanced radar depth (mer_2_30_5_0.5.h5 ) first and then use it to predict final depth.

[stanny880913]

gen_h5_file.py saves inputs and labels in the prepared_data.h5.

cal_radar.py generates radar inputs, which include radar depth, RCS and Doppler velocity.

For inference, we need to infer enhanced radar depth (mer_2_30_5_0.5.h5 ) first and then use it to predict final depth.

so generate prepared_data.h5 is necessary, need to use prepared_data.h5 to infer enhanced radar depth (mer_2_30_5_0.5.h5 ), do I need to run all command before gen_h5_file.py, like python prepare_flow_im.py, python cal_cam_matrix.py .etc, or I can directly generate prepared_data.h5 , maybe copy format like how you generate it!

[longyunf]

Some scripts are used to extract more features from image and radar inputs, such as optical flow and radar point flow, which are inputs for inference. Some scripts are used to create ground truth such as cal_gt.py. gen_h5_file3.py combines all the generated data into one file for training. You do not need to run (6) and (7), as they are used to create ground truth.

[stanny880913]

Some scripts are used to extract more features from image and radar inputs, such as optical flow and radar point flow, which are inputs for inference. Some scripts are used to create ground truth such as cal_gt.py. gen_h5_file3.py combines all the generated data into one file for training. You do not need to run (6) and (7), as they are used to create ground truth.

I understand, but I found that in cal_cam_matrix.py , the code has use 'ego-pose', this info is from LiDAR, but I don't have it? do this step is necessary?and 'calibrated_sensor_token' is necessary too? Thank you

[stanny880913]

Sorry, may I ask why I need ego-pose(translation,rotation) and calibrated_sensor(translation,rotation), if I only have calibration parameters from camera and radar, like

D: [-0.328655, 0.072550, 0.001761, 0.003128, 0.000000]
K: [ 700.63928,    0.     ,  612.81753,
       0.     ,  706.29767,  363.39884,
       0.     ,    0.     ,    1.     ]
R: [ 1.,  0.,  0.,
       0.,  1.,  0.,
       0.,  0.,  1.]

Can this replace calibrated_sensor(translation,rotation) or ego-pose(translation,rotation)? thank you~

[longyunf]

Ego position is useful to project previous radar points to current image or to project current radar to a previous image.

[stanny880913]

Ego position is useful to project previous radar points to current image or to project current radar to a previous image.

So if I don't have Ego position and calibrated_sensor and time stamp, is it possible to inference, or maybe I can hard-coding Ego position and calibrated_sensor , and then set timestamp=0 to countinue running cal_radar.py？ or if i don't have these information, I can't inference!!!

Can you explain the purpose of these three parameters? Thank you~ very appreciate~

[longyunf]

It might be possible to retrain the pda (train_pda.py) for nuScenes after discarding some inputs that are unavailable in your new dataset. For example, ignore duv_radar in https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/lib/data_loader_pda.py#L138C7-L138C7

I do not expect that direct inferencing works if some inputs are missing.

[stanny880913]

It might be possible to retrain the pda (train_pda.py) for nuScenes after discarding some inputs that are unavailable in your new dataset. For example, ignore duv_radar in https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/lib/data_loader_pda.py#L138C7-L138C7

I do not expect that direct inferencing works if some inputs are missing.

I don’t have ego pose calibrated_sensor and timestamp, I hard code some parameters like K ,ego_pose(translation,rotation)…,but if i want to inference,I need data_in, if I ignore duv_radar, the dimensions won’t be different? data_in = np.concatenate((im1, d_radar_norm, uv1_map, duv_im, duv_radar), axis=0) # (10,h,w) what is the most important information in data_in that I can Inference?

[longyunf]

I suggest retraining the model in nuScenes with a simplified data_in, which excludes channels that will not be available in your new dataset and then you can use the new model to do inference.

[stanny880913]

It might be possible to retrain the pda (train_pda.py) for nuScenes after discarding some inputs that are unavailable in your new dataset. For example, ignore duv_radar in https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/lib/data_loader_pda.py#L138C7-L138C7

I do not expect that direct inferencing works if some inputs are missing.

But when I ignore duv_radar , it will showRuntimeError: Given groups=1, weight of size [80, 10, 3, 3], expected input[4, 8, 192, 400] to have 10 channels, but got 8 channels instead , I need to add it back then the error wont't be appear, how to ignore it and won't appear dimention error? thank you

[longyunf]

As you have removed 2 channels from the input, the network structure should be modified accordingly. Specifically, the inChannels should be changed from 10 to 8. https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/scripts/train_pda.py#L279

[stanny880913]

As you have removed 2 channels from the input, the network structure should be modified accordingly. Specifically, the inChannels should be changed from 10 to 8.

https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/scripts/train_pda.py#L279

ok,I understand, I'll try it!! Sorry for another question, in test_depth.py, d_radar have two para called max and min, my max show 0, and it can't show radar image like you, it just show green image, how can I fixed it? thank you

[longyunf]

The argument d_max defines the maximum depth for evaluation. https://github.com/longyunf/rc-pda/blob/de993f9ff21357af64308e42c57197e8c7307d89/scripts/test_depth.py#L193

[stanny880913]

1.

2.

These two photos are a combination, and the image and his pcd file are used as input together. I wonder if the prediction is accurate in terms of this result? Thanks

[longyunf]

It looks less accurate than results from nuScenes.

[stanny880913]

It looks less accurate than results from nuScenes.

The type of my radar data collection is Object, unlike nuscenes which is Cluster type, will this cause inaccurate follow-up detection? Like my Object type, can the depth of the object be estimated normally? Thank you ~