client.simgetdetections cannot detect object sometimes.

[chongjeelee]

Bug report

• AirSim Version/#commit: AirSim1.6
• UE/Unity version:UE4.26
• autopilot version:
• OS Version: windows10

What's the issue you encountered?

when run the environment ,i found some objects can not be detected always! i change the object position,and the problem also occured ,so i want to know why the detection from camera is not very stable?this problem can directly result in the auto-testing about my own detect algorithom's accuracy. i add the video below, above the image of video ,you can see the drone can not be detected all the time.

Settings

How can the issue be reproduced?

1. 2.

Include full error message in text form

https://user-images.githubusercontent.com/30140969/155256272-582b58e8-73be-4770-9ca1-2554ebd70b3c.mp4

https://user-images.githubusercontent.com/30140969/155274806-32e6056d-7973-413b-84e0-ca36e5e2a5be.mp4

What's better than filing an issue? Filing a pull request :).

[bozcani]

The problem is -most probably- related to the logic behind how detection is computed. See this: https://github.com/microsoft/AirSim/blob/master/Unreal/Plugins/AirSim/Source/DetectionComponent.cpp#L152-L166

Here, the code sends random rays to 10 different points and checks whether any ray hits the actor (or, let's say, object). This is a kind of occlusion check, I think. However, the drawback is that when the object occupies less space in its bounding box, all rays likely miss the object.

I don't know what the most effective solution is. I set the upper limit from 10 to 20 (or any higher number) in this loop: https://github.com/microsoft/AirSim/blob/master/Unreal/Plugins/AirSim/Source/DetectionComponent.cpp#L155 And it worked for me. Again this might not be the optimal solution. I hope someone can come up with a better fix.

[chongjeelee]

thank you,bozcani! it does work for me,when i set the upper limit to 100. maybe this is not a best method,but it can work,thank you again.

[bozcani]

Great! Do you see any performance decrease when you set the upper limit to 100?

[chongjeelee]

Great! Do you see any performance decrease when you set the upper limit to 100?

Hi,i can not find any performace decrease currently . but i find another problem,the 2Dbox is not very close to the object...hhha

[bozcani]

Could you upload a sample image for this problem?

[chongjeelee]

https://user-images.githubusercontent.com/30140969/156274159-524c5a92-7039-43de-aa75-428394b0afcc.mp4

---

you can see that the 2d bounding box on the moving plane is not very close to plane's boundary...

[bozcani]

Ok, I see. I also encounter the same issue. I use segmentation images to refine object bounding boxes.

I couldn't figure out why Detection API can't give tight boxes for some cases. I think we can open another issue for this

[chongjeelee]

ok,you said right!

[zimmy87]

Hi @bozcani and @chongjeelee, I have filed #4412 to track the issue raised about the detection APIs not giving tight boxes around some objects. Part of the reason for this is that the detection APIs rely on AActor::GetActorBounds for calculating the 2D box for each detection. As you can see here, AActor::GetActorBounds may return a rectangle larger or smaller than the visible meshes attached to an actor depending upon the size and shape of the colliders attached to the actor as well as the presence of any invisible components.

As an immediate work-around for this issue, I recommend adjusting the size of the colliders of any actor who you feel doesn't receive a tight box currently to match the size of the actor's visible meshes as closely as possible. If this doesn't produce desirable results, we may need to spend some time researching alternatives to AActor::GetActorBounds inside of UDetectionComponent::calcBoundingFromViewInfo. I have one potential solution mentioned inside my issue, but it may not be performant and other better solutions may be preferable here. Feel free to brainstorm alternatives with a comment inside #4412.