Random Crazyflie Instabilities

[jusevitch]

Our lab is seeing multiple Crazyflies randomly drop to the ground or going unstable when trying to execute fairly mild waypoint tracking with multiple CF's. This YouTube video shows a representative example.

The CF's usually drop without any warning. The ones which drop often seem stable for most of the flight. The ones which do not drop also seem pretty stable as well. The VICON data seems to be clean, without any noise or object axes flipping.

We're trying to narrow down the most likely cause of this happening. Would this be due to improperly tuned controller gains? This post seems to suggest that additional hardware requires the controller gains to be retuned. We're using 3D printed frames to mount the markers on--this is because we're using the motionCapture option and need more room for unique marker patterns. The frame increases the mass to ~39g, but we've updated the value in the crazyflieTypes.yaml file. I've attached an up-close picture of the CF's below for reference.

We haven't been able to execute any maneuver with 8 CF's without at least 1 CF dropping. Usually 2-3 will drop before the end of the flight. Downwash doesn't seem to be the issue--there is 1m of space between the top and bottom CF's, and the random dropping occurs even when all CF's perform a maneuver at the same altitude.

Additional Notes:

• We're using an old commit (20cee6b) because last semester the then-current software simply would not run on our computer. @whoenig was kind enough to help us debug (which we are extremely grateful for!), and the only fix we could find was to revert to the old commit mentioned above.
• We're using the motionCapture option because, despite trying the suggestions in this issue, we can't get rid of ICP convergence errors. motionCapture works great for us though, so we're using that.
• Picture of the CF's is below.

[whoenig]

A few comments/questions:

1. Are you able to fly consistently with a subset? E.g., can you fly well with 1,2, or 4 CFs at a time? Background: with 39g you are actually very close to the payload limit. I have seen such issues with weak batteries that are unable to deliver the current. In general, the hobbyking batteries we reference on our webpage seem to be able to deliver more current than the stock batteries.
2. If you want to reduce the payload, you can also put a single (big) marker on the CF and only track position (not pose). This is not implemented in the official Crazyswarm yet, but I have seen researchers at the University of Toronto using this approach successfully with a slightly older VICON system. The on-board EKF can recover the pose, as long as there is some movement in x/y.
3. If you have a uSD card deck, it would be good to collect a log of the internal state estimate. Things like axis flips or many packet drops are hard to observe otherwise.
4. I don't think this issue can be solved by tuning the controller. I agree that this is likely not downwash related.
5. I suggest trying the latest version. I know we didn't close the other issue yet, but I've rolled back some changes in the master branch that I believe might have caused the issues you were having. Even if you can't go to the latest master branch version, you might want to go to the latest firmware (official or Crazyswarm) to pick up various firmware bug fixes.

[jusevitch]

1. No, we've had problems flying one, two, and four CFs. Occasionally we've had stability issues running the niceHover.py script with just one CF. There isn't any one CF that we can consistently trust to fly well 100% of the time, although each one will fly without error about 60-70% of the time. Stability seems to degrade somewhat over time--the more time that the script runs, the more CF's seem to have problems. When leaving niceHover.py running for a while (>15 seconds), most of the quads tend to wobble around more as time goes on.

2. Thanks for the information about the payload limit--that is really good to know. Is there a hard upper bound? Also, do you happen to know how the other university associated each individual point with the correct CF? (since points can't be unique objects in VICON.) This one-marker solution seems like a great option for us.

3. We do have some uSD decks--we'll try logging some data and post back if they reveal anything instructive. I'm assuming this page is what we should refer to in order to get that data?

4. Glad we can rule those out! That'll save a lot of time.

5. We'll try this out too. Will let you know if any issues come up.

Thanks for your help!

[hanzheteng]

Hi @jusevitch ,

We use the same kind of prop guard as you mentioned and observed this strange behavior as well. During the experiments, we only use takeoff, goTo and land commands. Here are our observations and some inferences.

1. Though it seems like Turnigy battery has a better discharge ability (35C-70C as shown on package), during our experiments, the original battery has a better performance/stability. Also, as you mentioned, we observed similar degradation (in terms of stability) over time. To this end, we suspect that it's partially due to the insufficient current supply. In other words, battery in a low voltage level cannot afford high weight payload. The corresponding solutions could be 1) using batteries in good condition and 2) reducing the payload.

2. The second reason could be the biased tracking center created by improper marker arrangement. As @whoenig shows in the documentation here, they put markers on side and therefore the object center is in between battery and center board. If we use prop guard and put markers on top of it, by default in mocap the object center is a bit above the battery in z axis and may be biased from the center as well. This offset should work fine with translational movement or small-angle tilt, but may have issue with fast movement or large-angle rotation. For example, the robot is about to tilt and move towards some direction. Theoretically, a pure rotational movement would not change the center of the position, but due to center offset the position feedback in mocap would show that it moves some distance. This may be a bad thing for controller. From our observations, the center works better in 0.025-0.027 height in z axis, which is in between the center board and the battery.

3. The next thing we suspect is the weight distribution. @whoenig mentioned that the controller is pre-turned for 33g weight with a LED ring. Note that the mass of robot is quite centered (LED ring also helps center the mass). The markers together with prop guard actually change the system dynamics and hence affect the robustness of the controller. I guess adding a LED ring to center the mass would be better, but this is approaching the maximum payload, which is bad in another aspect. The corresponding solutions are 1) centering the mass of robot and 2) moving slower. We indeed observed something like this. (e.g., taking off to 1m height in 4s works fine but in 3s often fails.)

4. We also observed some behaviors related to airflow and wind disturbance. During one experiment, we observed an obvious downwash effect. When two robots came to the place below the other two robots, these two robots immediately fell down (0.5m and 1.5m height respectively). Additionally, we observed somewhat ground effect as well. The robot is more stable in 1m height than in 0.5m height. The last thing I suspect is that the prop guard frame under propeller would affect the aerodynamics property of the airflow. This part is hard to test and verify.

To sum up, we think bad things are most likely credited to the prop guard. It changes robot dynamics with which the controller cannot handle, and therefore affects the stability.

We need to get some uSD decks and investigate a bit more about this. The above is the progress we made for now. I hope these analyses would help improve the stability of our systems in the future.

[jusevitch]

@hanzheteng,

Thanks so much for the response. As an update, we changed our code to the most recent commit. Although this showed a little improvement, we're still seeing the same essential problems.

1. The best option for us seems to be reducing the payload. However, this will be difficult since we need the unique marker patterns for the motionCapture option (the libobjecttracker option never worked for us; we had lots of problems with the ICP algorithm not converging). @whoenig mentioned an experimental option for 1 marker tracking in this issue--is there any documentation on how to try it out?

2. We'll make sure to arrange the object center more carefully in the VICON tracker software. We tried to make sure it was fairly centered along the X and Y axes, but we'll check the Z axis coordinates more carefully next time we fly.

3. We suspected the weight distribution as well. Seems that removing the LED ring would overall be helpful since that'll keep the CF from being near the max payload. We can try tuning the controller for the different weight distribution.

4. We've had a little more success as well when we've increased the distance between CF's to prevent wind disturbance effects. However, I wonder if these wind effects are exacerbated by the weight distribution problems. In other words, we can try tuning the controller to match the new weight distribution and seeing if that makes them more robust against the wind disturbances.

Another option we may try is simply using a larger frame size. Just as a heads up though, our lab has some other deadlines we're working on and so we may not be able to conduct more tests until June. I'll post again if we get any new data relating to this issue.

[whoenig]

Thanks for all the insightful comments. There is no documentation on 1-marker tracking, but here are the steps:

• In the marker configuration only have a single point, e.g., by switching https://github.com/USC-ACTLab/crazyswarm/blob/master/ros_ws/src/crazyswarm/launch/crazyflieTypes.yaml#L6 to 3
• Use the official firmware (our EKF currently cannot update positions only)

I tested this with two CFs so far. I plan to do more tests soon and also make it work with kalmanUSC.

[hanzheteng]

Updates:

After taking off the prop guard, we are able to consistently fly five robots for more than 30s. We repeated this experiment for three times and no one dropped. You can find the video here (https://youtu.be/ClCdcSn8Uk0). The following are our robots with new marker configuration.