search

microsoft / AirSim

Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research
https://microsoft.github.io/AirSim/
Other
16.25k stars 4.52k forks source link

Improve accuracy of FDetectionInfo.Box2D #4412

Open zimmy87 opened 2 years ago

zimmy87 commented 2 years ago

What feature are you suggesting?

Overview:

Currently, UDetectionComponent::calcBoundingFromViewInfo() calculates the location and size of the 2D box around detected objects using a 2D projection of the output of AActor::GetActorBounds. As can be seen in this video, due to the size and shape of colliders as well as the possibility of an actor containing invisible components, AActor::GetActorBounds may or may not reflect the bounds of the visible meshes attached to an actor. Since AActor::GetActorBounds also returns a rectangular box, it's possible a 2D projection extends beyond the visible meshes depending upon how the target actor is rotated relative to the camera. One possible solution to this would be to iterate through an actor's components and use the output of UKismetSystemLibrary::GetComponentBounds for each visible component as the basis for the 2D box. However, this solution may result in a performance impact due to the additional iteration over each detected actor's components. More research should be done to determine if this is feasible or if there is a better solution.

Smaller Details:

Nature of Request:

Why would this feature be useful?

This would address concerns raised in #4367 that the detection APIs do not accurately represent the bounds of detected objects on a 2D display.

bozcani commented 2 years ago

Is there any progress on this issue? I am not a expert of Unreal programming expert but what do you think about using segmentation information inside the Airsim to refine object bounding boxes?

Currently, we can record segmentation images and refine bounding boxes in an offline manner. Can we make this refinement inside Airsim as a built-in feature?

dellhuyongcai commented 2 years ago

Is there any progress on this issue?I just know the box is axis aligned bounding box,has anyone else tried the original bounding box ? it seems that ue4 does not have original bounding box function.