How to implement real time object detection with bounding boxes inside AirSim environment

[thejeshk]

Hi LiaoSteve,

I have created my own custom environment and also my own custom vehicle in Unreal and built airsim. Now Airsim is plugged in my custom environment

Project Pipe-line: Create environment and Vehicle --> Object detection (for custom class e.g. Traffic cone) --> Path planning (RRT*) --> PID controller --> Implement in Real vehicle

Now I am stuck at object detection part. I have manually collected the traffic cone images from my environment and trained using YOLO model.

So, now I have trained weights and I need to know if there is any possible way to run inference on pre-trained YOLO model.

What I need is --> When my vehicle is running in Airsim environment, when it sees the traffic cone it has to detect and put bounding box over it.

Can you please help me with this??

[LiaoSteve]

Hi thejeshk,

You should get car camera image, and input this image in your yolo model, and you can get bbox and scores, then use cv2.putext() to put result to image, and use cv2.imshow(). I dont' understand what you want, but i think this code will help you:

1. My drone

client = airsim.MultirotorClient()

2. Get Depth and RGB data

responses = client.simGetImages([airsim.ImageRequest("0", airsim.ImageType.DepthPlanner, pixels_as_float=True, compress=False), airsim.ImageRequest("1", airsim.ImageType.Scene, False, False)])

3. Get RGB image

color = responses[1] imgcolor = np.fromstring(color.image_data_uint8, dtype=np.uint8) imgcolor = imgcolor.reshape(responses[1].height, responses[1].width, -1)
if imgcolor.shape[2] == 4:
imgcolor = cv2.cvtColor(imgcolor,cv2.COLOR_RGBA2BGR)

4. Create my yolo calss

my_yolo = YOLO()

5. Use yolo to detect objects

image = Image.fromarray(imgcolor) image = my_yolo.detect_image(image)
result = np.asarray(image)
cv2.imshow("result", result)

6. Referrence :

yolov3 : https://github.com/LiaoSteve/Drone-GCS-and-AI/blob/django_app/copter/yolo.py

airsim : https://github.com/LiaoSteve/Drone-GCS-and-AI/blob/django_app/airsim/ForAirSim/Simulator.py

[thejeshk]

Hi @LiaoSteve

Thank you so much for the reply. Sorry for the late response....I did not get any notification.

I have skimmed through your code. You are using way points aswell. an you please tell me if I can use waypoints to drive my vehicle from one point to another ?

My project goal is: Able to make a vehicle to move from position "A" to "B" with obstacles places in between "A" and "B" (for example traffic cone). So now when my vehicle sees the cone it has to detect as traffic cone and put bounding box on it and somehow If i get to know the distance of the cone I can make vehicle stop and maneuver to avoid obstacle

[LiaoSteve]

Hi @thejeshk

In our project, we move multirotor from point A to point B, first my multirotor is take off at point A and turn yaw to point B, and go straight to point B.

So, if you want to move car from point A to point B, you need to steer your car until the car is heading to point B. In this case, I think you can give your car a little throttle, and try steering until the car is heading to point B.

You can see this example: https://github.com/microsoft/AirSim/blob/master/PythonClient/car/drive_straight.py