Closed yrik closed 1 year ago
Hi! Thanks for the kind words and interest!
The entire runtime pipeline would look something like:
And during training, we have two separate training regimes:
For more details on how everything is tied together on a real UAV, I'm currently doing a project using Dronekit (it's a bit old but less cumbersome than ROS/PX4 for small systems) with the code here. A good place to start would probably be in this file.
Alternatively, you could also go the full stack route, where you train the RL agent directly in the simulation using domain randomization. I attempted this in the past, but found it too hard to get working in real life since my real world environment is much more complex than what can be feasibly designed in Bullet under realistic timelines. The code is here should you be interested, but note that the repo is horrifically old and predates PyFlyt by many moons.
set_mode(-1)
which are direct commands to motors, or other modes if you wish. Unfortunately, PyFlyt doesn't publish raw gyro and accelerometer data (though this would be an add-able feature) so I'm not sure what kind of sensor data you will be passing to SITL.For ideas about passing data between separate pieces of software, you could use ROS (and relevant rospy packages) if you're passing around lots of data, or simply use zmq (pyzmq package in Python) if it's just a few numpy arrays.
Let me know if you have any other questions, I'd be happy to assist wherever.
Thank you for the work on this awesome package! Super elegant and clean.
I have a question about the best usage of this package.
My goal would be to train a model to fly towards the targate using only video stream and drone sensor values as an input. Without relying on coordinates. So I would have a video with an intermediate model that does object detection/tracking and the drone would need to fly towards that detected object.
Is it possible to take a video frame, apply object detection, and use it as an input for the model? What is the best to implement it using this repo?
And a side question about integration with betaflight, to train a model that uses the realistic commands from betaflight's SITL. What do you think the best approach is to do it?