Crazyflies start crashing after flying more than 3-4 at the same time

[vchavezfs]

Hello,

I just recently came across the crazy-swarm video in the following link: https://crazyswarm.readthedocs.io/en/latest/index.html and was amazed on how this was achieved. I am particularly interested in performing indoor drone shows and was hoping to use this approach / system.

So I bought a set of 10 crazyflies 2.0's and with 4 crazyradio PA and decided to give it a try. I use 4 17W Prime Optitrack Cameras (5 x 3.5 x 3 volume) as my tracking system/area and I managed to fly 3-4 crazyflies following the steps provided by the documentation.

However, when I start to add more crazyflies, they become extremely unstable and start crashing.

My crazyflies.yaml is as follows: I alternated every crazyflie in 3 different channels across 3 different radios on the same PC. cf1 -> channel 110 -> address 01 cf2 -> channel 100 -> address 02 cf3 -> channel 90 -> address 03 etc... **Originally, I tried flying everything on the same radio, but I had similar instability issues.

My launch file uses motioncapture rather than libobjecttracker.

I'm using 3 reflective markers per drone (unique configuration set to false in motive software). However I do try to make them as distinct as possible and the system is tracking all 10 different rigid bodies.

I noticed that the RGB lights flicked from green to red from time to time, could this possibly be the issue? Otherwise what could be causing the crazyflies to become unstable when I try to fly more than 3-4 of them?

Another note, when I fly each crazyflie individually, they work fine, so I'm ruling out the fact that each drone have any defects.

Thanks, hoping for some help regarding this issue.

[whoenig]

See also https://forum.bitcraze.io/viewtopic.php?f=5&t=3439.

[whoenig]

As Arnaud already pointed out (and you fixed), in your setup I recommend using a single radio (i.e., all CFs using the same channel). Make sure the channel has good transfer rates.

For your actual problem, I suspect that tracking is an issue. I am not familiar with the Motive software: If you set "unique configuration to false", does it actually do frame-by-frame tracking and outlier rejection?) I have seen the issue you describe in two cases: a) if there is an axis flip, i.e., the position is estimated correctly but the orientation suddenly flips; and b) if two tracked objects are swapped. There are two ways around this issue:

1. Use libobjecttracker instead of Motive for tracking. Our library does a lot of outlier rejection and avoids sudden jumps.
2. (experimental) I recently implemented a tracking mode where you only need a single marker (preferably a bigger one placed on top). The on-board state estimate than only fuses external position data (similar to GPS). Thus the issue a) cannot occur. This is not well documented yet (because I only tested with 2 CFs so far), but I can write-up some instructions if switching to libobjecttracker doesn't help.

Flickering from green to red is fine, as long as it is red for a short amount of time. If you see a lot of red, try switching to a different channel (or turn of 2.4 Ghz wifi if possible).

For the turbulence: 0.5m should be totally fine. There is some air-flow effect, but as you can see in the Crazyswarm video (where drones were spaced 0.5m apart), the controller is able to handle that case (with a slightly larger position error compared to flying a single drone alone.)

[vchavezfs]

Thanks for the response Whoenig!

Yes you were correct, orientation was an issue. At certain frames, we would sometimes see the rigid bodies roll/pitch jump to a negative value. So I have rearranged the markers in a non symmetrical way and managed to fly 5 drones now on a single crazy radio pa using motionCapture. However, I did come across an issue when I added the 6th drone. Each of cfs couldn't properly hover after that and crashed the second they took off. When i took out the 6th cf, they were flying properly again.

What fixed this issue was to use another crazy radio pa with the 6th cf on its channel. After doing so, I managed to fly 6 cfs! :) (Currently adding cf's to the new radio/channel and will be testing it soon)

Additionally, I have another pc configured to use libobjecttracker method. It works really well for a single cf. However, when i try to add the second cf with the same configuration (4 markers), RVIZ hover_swarm doesn't detect them. It detects each drone individually but when both are present, they don't seem to be showing on RVIZ? Maybe I've missed a step, but I have a doubt. In the crazyflieTypes.yaml, should the marker configuration set be of only one cf or all the cfs that are being tracked?

[whoenig]

I am glad you got to fly 6. Although it is strange that you ran into this limit - it should support more CFs per radio. Do you have logging enabled? 15 CFs/radio are with logging disabled (which I believe is the default setting.)

You should use one marker configuration for all your CFs. You can share your crazyflieTypes.yaml, allCrazyflies.yaml, and hover_swarm.launch here if you are in doubt about the configuration.

[vchavezfs]

Sorry for the late reply @whoenig.

Interesting, I had my logging set to enabled earlier when I first started so I could log some errors. I set it to false now and managed to fly 7. But the drones seem inconsistent with their stability. The following link shows the current test run with 8 CF's.

https://youtu.be/PkOocSCpiWo

As for the other method, these are my configuration files (had to change them to .txt to upload). It doesn't seem to be tracking both of them at the same time. (Note: Our drones are around 37g without the RGB light)

crazyflieTypes.txt hover_swarm.txt crazyflies.txt

[jpreiss]

It looks like your markers configuration is too symmetrical.

[vchavezfs]

Update Good news! We found out the issue why our cf's were not detecting using libobjecttracker. Our axis was flipped, so when we defined the initial positions in the crazyflies.yaml, they couldn't locate the objects in those defined positions.

We ran several tests using the 3 marker configuration & our newly ordered MOCAP deck provided by Bitcraze. We still do get similar instabilities when flying 7-8 cf's. We noticed the frames generated in RVIZ where erratic on some of the cfs. It was random, whenever we restarted the hover_swarm server it would sometimes detect some configurations or some of the frames would be flipping rapidly.

Furthermore, we tried the single marker configuration as you suggested (setting configuration type to 3 as mentioned in the crazyfliesType.yaml). We managed to fly 3 cf's properly, however when we added the 4th cf, one of them randomly started spinning in circles. The spinning eventually got bigger and bigger until it crashed.

@jpreiss You could be right, our current 3 marker configuration is just a small triangle where all the markers are on the same level. Maybe we need to offset some markers on different heights so it could identify its orientation?

Note: We are using 12.7mm markers rather than the 7.9mm

[whoenig]

Different heights can be achieved by using small plastic standoffs; I suggest at least 1cm of height difference (for one of the markers). Furthermore, it is better to have 4 instead of only 3 markers to handle cases where one of the markers gets occluded properly. It should be fine to use bigger markers, as long as you keep the weight as low as possible.

Thanks for the feedback for the single-marker configuration. I only tested with up to two CFs so far, but I plan to test a larger configuration soon. One thing to keep in mind is that with just position feedback, the yaw state estimate can only be recovered when the CFs are moving in the x-y plane. Thus, with the single marker configuration, you cannot hover for an extended period of time (up to ~30sec should be fine).

[vchavezfs]

Thanks for the support @whoenig @jpreiss After multiple tests, I could not manage to fly more than 6-7 cf's (just hovering) without them losing control and crashing. The instability still persists when adding multiple cf's :( I have followed your suggestions and advice but still happen to come across the same issues from time to time. When using libobjecttracker, the cf's are clearly visible on RVIZ, and start off hovering well. But when more and more cf's start taking off, the instabilities of the some of the drones increase and eventually crash (rgb ring turns red instantly).

I'm starting to think that maybe the issues stem from the limitations of my tracking system. I am using only 4 cameras, and trying to fly 10 cf's. Do I need to scale up and get more cameras?

[whoenig]

Does your rgb ring turn red before or after the crash? It is hard to tell if you need more cameras - you would need to figure out if the crashes happen after you lost tracking, or if you lost tracking because a drone crashed... One way to debug this case is by using the uSD card deck and enabling data logging (although, we currently do not have ready analysis tools that show you exactly what happened.)

I personally doubt it is the tracking system, because you track all your drones all the time (even if some of them are initially at the ground.) One thing I noticed from your video is that your frame is fairly big - this will reduce flight time and makes the controller less stable (which is fairly aggressively tuned). Could you put the markers on without your frame?

[vchavezfs]

We noticed that during flight, the markers start flickering. I removed the custom frame and placed all the markers on the mo cap deck. However, due to the size of my markers, some of the cameras seem some markers as a single point merged (depending on the angle). It is difficult to tell whether the red on the rgb ring turns red before / after the crash. But I did notice that the RVIZ frames will start shaking erratically and I get ICP did not converge errors on the cf's losing stability.

This is a picture of my current setup.

[whoenig]

I assume you use the big markers because the resolution of your cameras is not high enough for smaller markers?

If that's the case, I think your best bet is the mode with a single marker on top for you. I know you did try it before and had some problems, but did you ensure an accurate initial placement of the CFs in that case?

[vchavezfs]

Sorry for the late reply @whoenig

We have tried using the single marker configuration and it works well up to a point when we start adding 3-4 cfs. Some of them randomly start going in circles and eventually crash. We have paused using the camera system and got ourselves the loco positioning system. The accuracy is far from that of the camera system but we are able to fly 10 drones using the crazyswarm firmware. Will update you once we resume with the camera setup.

Thanks again for the help!

[hanzheteng]

One thing to keep in mind is that with just position feedback, the yaw state estimate can only be recovered when the CFs are moving in the x-y plane.

Hi @whoenig

Do you mean that with full pose feedback the estimator can recover yaw attitude? Because I notice that crazyflie server only accepts 3d position update (no yaw or other attitude) from the external_position topic. The bridge node is implemented in publish_external_position_vicon.py.

I have tried an experiment with crazyflie_ros repo that makes crazyflie follow my wand, and I found that it cannot follow my yaw movement. Does crazyswarm use something else that can support yaw update?

[jpreiss]

@vchavezfs were you able to resolve this?

[jpreiss]

closing due to inactivity.