Reproducing issue about ego Vehicle stop permanently for false positive objects

[hz3014]

Dear Author, I am currently reproducing your work and I am seeing following issue. When there is false positive of objects (most likely wrongly recognised as Pedestrian), the ego Vehicle permanently stops. According to the figure below, the the safety control mechanism think the false positive has velocity and will crash with vehicle.

I think the root of cause might be that Pedestrian in dataset always has constant velocity, therefore false positive objects always has velocity when neural network wrongly classify object as Pedestrian. However, I am confused as I also read your paper you have tracking code to further calculate the correct velocity.

I was wondering did you have this problem? Or this is my reproducing issue. Any hints will be helpful. Thank you!

[deepcs233]

Hi! I also found this problem occasionally during my testing, my solutions are:

1. use more data
2. wait continous stop steps > 1200, the ego-car will go on without any restrictions : https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L262
3. or you can modift the code in the controller to get around this problem

[hz3014]

I have another question about HybridEmbed. I saw it under init that you used with torch.no_grad():

https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/interfuser/timm/models/interfuser.py#L37-L47

To my limit understanding, with torch.no_grad() is used to disable gradient back propagation. I understand you might need it for image backbone because you can reuse the pertained model. But why is it also applied to lidar backbone since lidar backbone doesn't use pretrained model? Doesn't that disable the resnet ability to learn for a lidar backbone?

Thank you for your answer in advance.

[deepcs233]

Hi, the code is just used to get the feature_size when initializaing the module if we don't specify the feature_size. It foward the backbone with a zero tensor to get the shape of output.

[deepcs233]

For image backbone, we resue the pertrained model without any update. For lidar backbone, we update their parameters in the training stage. Detailed code is here: https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/interfuser/train.py#L987

[hz3014]

Thank you for your clarification, it is really helpful

[zhenggruk]

Hi! I also found this problem occasionally during my testing, my solutions are:

1. use more data
2. wait continous stop steps > 1200, the ego-car will go on without any restrictions : https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L262
3. or you can modift the code in the controller to get around this problem

Dear Author, what is the meaning of stop steps? And what is the meaning of self.block_stop_sign_distance? I don't understand the explanation. Could you please explain more? Thank you very much! :)
https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L102

[deepcs233]

Hi, we find that in some cases, the agent gets stuck because it detects some non-existent objects or always recognizes there is a red light ahead. To bypass this kind of case in our controller, we use a variable to record the time steps that the ego-car doesn't move. And when this variable self.stop_steps is greater than 1200(in our setting), we will force the ego-car to go on, ignoring any restrictions.

[deepcs233]

self.block_stop_sign_distance: when the ego-car detects a stop sign, it will perfom a emergency stop. But after it's speed drops to 0, it possilby detects the same stop sign again and perfom a emergency stop agin... So we set this distance that if the car hasn't moved self.block_stop_sign_distance, it won't respond to stop sign.

[zhenggruk]

self.block_stop_sign_distance: when the ego-car detects a stop sign, it will perfom a emergency stop. But after it's speed drops to 0, it possilby detects the same stop sign again and perfom a emergency stop agin... So we set this distance that if the car hasn't moved self.block_stop_sign_distance, it won't respond to stop sign.

Thank you for your clarification. One more thing about self.block_stop_sign_distance that confuses me is that self.block_stop_sign_distance is set to 0 but the comment next to it says "# it means in 30m, stop sign will not take effect again". I don't understand that part. If self.block_stop_sign_distance=1, does it mean that within 1 m, stop sign will not take effect again?

[deepcs233]

Sorry, the comment is misleading and i will fix it. When the car detects the stop sign, it will set self.block_stop_sign_distance to 2.0 https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L140

If self.block_stop_sign_distance=1, does it mean that within 1 m, stop sign will not take effect again?

Yes, your unstanding is right. But before the ego-car detects a stop sign, this variable should be zero.

[zhenggruk]

Thank you very much for your reply. It is really helpful! :)

[hz3014]

@deepcs233 Dear Author, I was wondering what is the use of token_global?

https://github.com/opendilab/InterFuser/blob/e0682c350892a243cf40bf448622743f4b26d0f3/interfuser/timm/models/interfuser.py#L867-L870

[deepcs233]

The token_global is the global mean for each view input, and we hope to capture the global feature into one token. But we haven't ablated this design.

[zhenggruk]

Sorry, the comment is misleading and i will fix it. When the car detects the stop sign, it will set self.block_stop_sign_distance to 2.0

https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L140

If self.block_stop_sign_distance=1, does it mean that within 1 m, stop sign will not take effect again?

Yes, your unstanding is right. But before the ego-car detects a stop sign, this variable should be zero.

Dear author, I have more questions.

1. You mention that when car detects the stop sign, it will set it will set self.block_stop_sign_distance to 2.0. From the code, that corresponds to the case when stop_sign < 0.6. So lower stop_sign value represents higher probability of detecting stop sign?

2. Is the meaning of self.red_light_steps similar to self.stop_steps ?https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L133

3. Is the meaning of if self.block_red_light similar to self.block_stop_sign_distance ? https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L136

4. What about the values of traffic_light_state and junction? (lower value represents higher probability or the other way around)

5. Since the safety distance is calculated from d_0/d_05/d_075/d_1/d_15/d_2.What is the meaning of d_0/d_05/d_075/d_1/d_15/d_2? (is the t in the code corresponds to the time step t in the paper? can you explain more?) https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L208-L211

[deepcs233]

Hi！

1. Yes.
2. No. Our model sometimes always detect the non-existent red light and the agent will stop for a long time. At last, it will time out. So we set this variable to avoid the agent to stop for a red light with a long time.
3. It's explained in (2). These two variables are used together.
4. Higher value of traffic_light_state means higher probability of red or yellow lights. Lower value of junction means higher probability that the ego car is in the junction.
5. is the t in the code corresponds to the time step t in the paper? Yes, d_05 means the maximum safe distance in 0.5 seconds. Others and so on. They are computed by this function, the only difference is the parameter t: https://github.com/opendilab/InterFuser/blob/dcc2c20bba2c8656773baab5a552a27fb74e86dc/leaderboard/team_code/interfuser_controller.py#L172

[Naive-Bayes]

Hello, Shao Hao, when I look the controller code(interfused_controller.py), i have a question about the desired_speed.

In paper appendix B, it said the desired speed could get by solving the linear programming optimization

However, in code it is calculated as follow desired_speed = max( 0, min( 4 * d_05 - speed - max(0, speed - 2.5), self.config.max_speed, 2 * d_1 - 0.5 * speed - max(0, speed - 2.5), ), )

I do not understand the relations between the code and the linear programming, and what does mean of the code? Looking for your replay. Thank you!

[deepcs233]

Hi, it's just a easy way to solve the linear programming problem. For safety, we directly compute the max speed that satifies all constrains. For example, in the code you provided, the term 4 * d_05 - speed - max(0, speed - 2.5) means that desired_speed <= 4 * d_05 - speed - max(0, speed - 2.5)

The constrain comes from: [desired_speed (it means the speed after 0.5s) + speed (it means the speed at present) + max(0, speed - 2.5) (for safety)] / 2 (get average speed) * 0.5 (driving time) < d_05 (it means the safety distance in 0.5s)

For clarity, the above inequality can be viewed as: avg speed * driving time <= safety distance