search

uzh-rpg / flightmare

An Open Flexible Quadrotor Simulator
https://uzh-rpg.github.io/flightmare/
Other
965 stars 340 forks source link

vision as input for RL #70

Open HosseinSheikhi opened 3 years ago

HosseinSheikhi commented 3 years ago

I wonder if someone helps me how I can run an RL algorithm by feeding vision as input in flightmare.

yun-long commented 3 years ago

You would have to implement your own pybind wrapper in order to get the image using Python and train vision-based policy.

Specifically, you have to 1) create a new environment or modify the quadrotor_env.hpp file. attach a RGB camera on the quadrotor. 2) retrieve the image using the camera 3) modify the step function of the environment and such that it returns the image to python, instead of low level states. https://github.com/uzh-rpg/flightmare/blob/master/flightlib/src/envs/quadrotor_env/quadrotor_env.cpp#L93