Example Using PPO (Or any Controller) With Real Crazyfile Drone

[zcase]

I have been trying to use the Crazyflie 2.x python api to control a drone using the trained PPO controller. I think I might be running into reference frame issue. Do yall have any examples of taking a trained controller like the PPO Stab model and integrating it into a real drone?

[Federico-PizarroBejarano]

Hi @zcase, yes I have used PPO models trained in simulation on real crazyflies before. However, it is never easy unfortunately. How are you controlling the drone? In our gym, the controllers operate by sending direct thrust commands to the four rotors. However, that is not the way most drones have their control interfaces set up. When I do real experiments, I actually switch to using an interface native to the crazyflie, either position, attitude, or rate commands, and train a controller that sends those commands. To do that I use the crazyflie interface created for this paper: https://arxiv.org/pdf/2308.16743. The main branch that does this I believe is https://github.com/utiasDSL/safe-control-gym/tree/beta-iros-competition, but it is unfortunately super out of date. In my own fork i have been doing real crazyflie experiments using https://github.com/Federico-PizarroBejarano/safe-control-gym/tree/new_crazyflie, which works but also has a lot of experiment specific stuff on it.

I am happy to help you get the real crazyflies working, but unfortunately I have found it is a relatively difficult process that is super specific to the drone, task, and controller you are using.

[zcase]

@Federico-PizarroBejarano This is great information. I have been trying to control the drone via the motor commands as outputed from training with this repo and the gym-pybullet-drone repo. I moved to this one because it seemed to have more of the stuff to aid in actual drone flight such as the random ranges in weight and other stuff so that the model doesn't suffer from model mismatch errors when it goes onto the real drone.

So are you saying rather than controlling the drone to go to a point, stabilize, or follow a trajectory using the 4 motor outputs, you train it to use a different interface?

Ill have to checkout out your repo and see what you are doing. I would really like to overcome the sim2real problem with this crazyflie and any input would be helpful! Thank you!

Are there any specific interfaces you use to control the real crazyflie? I have been working with the Crazyflie python API in a script that I built and just been using that but I believe I might be dealing with rotation matrix issues between what this repo trains the ppo algorithm with and what the actual telemetry the drone outputs.

[Federico-PizarroBejarano]

Yeah essentially controlling the real drone using direct motor commands seems to suffer from two main issues:

1. It is very hard and unstable. Controlling via a higher-level interface is far more stable because it uses the in-built stabilization and tracking controllers of the drone (PID or Mellinger). This should be obvious: controlling the drone using position or velocity setpoints is obviously way easier than controlling it via direct motor commands. Additionally, direct motor commands need to use extremely high rate control, usually 100Hz-500Hz, which is more than I can do for a few reasons. And it needs very accurate state measurements, which also I cannot provide at a sufficiently high accuracy or rate.
2. The crazyflie has a few control interfaces: https://crazyswarm.readthedocs.io/en/latest/api.html#pycrazyswarm.crazyflie.Crazyflie.cmdFullState. I do not think direct motor control is one of them. Although you can build your own interface, I chose to use their well-supported interfaces. However, this comes with its own issues because then, during training, you need to simulate their whole stack, including their interface, lower-level controllers, etc. That is exactly what the branches I sent do

[zcase]

@Federico-PizarroBejarano I agree that it is a hard problem, but has any one from your team controlled the crazyflie via direct motor control? I was under the impression that was what people were doing in some of the research. Or are they at one step hire like controlling via body axis and thrust command?

I guess what level are yall usually doing your research at?

[Federico-PizarroBejarano]

Hi @zcase, no one in my group has even attempted direct motor control of crazyflies. Part of this is because we work a lot with MPC, and its simply too slow to control a drone via direct motor commands. To my knowledge some of my lab mates in Munich have been able to control the drone in the x-z plane successfully with attitude rate commands, ie roll, pitch, yaw rates + overall thrust (RPYT). I don't think their work has been published yet or anything, but if this is of interest to you I can speak to them. I have gotten it to work in 3D with attitude commands, but not attitude rates.

[zcase]

@Federico-PizarroBejarano yea that would be of interest to me.

[Federico-PizarroBejarano]

Hi @zcase this is still ongoing work but we are trying to streamline and publish our pipeline. Feel free to email me at federico.pizarrobejarano@mail.utoronto.ca to chat more about your specific pipeline. I will try to update this thread when our work is published, but for now I will consider it closed.

[zcase]

@Federico-PizarroBejarano I have started looking into training a model to be controlled by angular velocities and thrust, you mentioned earlier that you do something similar rather than using direct motor commands. In your repo you linked above, can you point me to a file or set of files that directly control a physical drone with a different approach from the direct motor control?

[Federico-PizarroBejarano]

To my knowledge, there are two ways of accomplishing this:

1. Using the sim2real pipeline from https://arxiv.org/abs/2308.16743, found here (https://github.com/utiasDSL/safe-control-gym/tree/beta-iros-competition) and updated in my own fork (with some other code unfortunately) here (https://github.com/Federico-PizarroBejarano/safe-control-gym/tree/crazyflie). What this does is that it simulates the entire software stack of the crazyflie, converting between the API commands of the real crazyflie to the thrust they would output. This is the approach I have used in my own papers.
2. Identifying a model of the quadrotor that takes in the desired commands and inserting this model directly into the safe control gym as a new environment altogether. This would require extensive experiments but would allow you to bypass simulating the software stack of the crazyflie. This is somewhat being pursued by https://github.com/utiasDSL/safe-control-gym/pull/166 but it is not complete and currently only functions for a quadrotor moving in the x-z plane.

Overall, this is a major current limitation of our gym as it was made for thrust commands, which are simply not practical for real deployment. We are working on fixing this but it is not complete and so I cannot give you a fully working pipeline yet. Your best bet may be to work with option 1 for now, and hopefully we will figure out option 2 in the next few months.