Launch crazyflie that is turned off

[araujokth]

Hi Wolfgang,

I noticed from the beginning that if I launched the application with multiple CFs, and one of the CFs would be turned off, then the complete ROS application hangs for all CFs. Today I was trying to investigate how to avoid this, since it could be the case that I would be starting the application and one of the CFs had ran out of battery and died, or if I would just like to remove one CF from the experiment, but not remove it from the launch. I would like all the others to still run their applications in case that would happen. Do you think that it would be possible to avoid this in a nice way?

I dig a bit into the code and saw that things get stuck at crazyflie.cpp, line 81 to 85, after the constructor is initiated at the crazyflie_server. I could not think of a good way yet to do this and thought of asking you first before starting to modify the code.

Thanks for the help again!

[whoenig]

That is certainly a bug and should be handled more gracefully! The main issue is that the code is executed in the Crazyflie constructor and that the Crazyflie object is created outside of the thread (in https://github.com/whoenig/crazyflie_ros/blob/master/crazyflie_driver/src/crazyflie_server.cpp#L74). So there are two possible solutions: a) move the code to a function which needs to be called; call this function within the run() function in Crazyflie_server. b) Leave crazyflie.cpp as is, but create the crazyflie object dynamically in the run function, rather than in the constructor. That also requires to move the ROS specific functions for subscription to the run() function to avoid corner cases where a ROS message is received, but the m_cf is not yet created.

I can work on a fix, but will need to wait until sometime next week for testing in the lab before I can push it.

[araujokth]

Hi Wolfgang,

thanks for the tips and very sorry for the late reply! we went on to try your option a) and now the application does not hang but no message arrives to the CF (so the alive CFs do not work normally, even though ROS keeps working), but I have not understood yet why this happens. In case you would have any time to look into this issue it would be awesome. Otherwise we will keep trying :)

Thanks so much again!

[araujokth]

Just one additional thing. When I launch with many CFs (5 now, with 3 radios), sometimes I only get communication with 2, 3 or 4 of them, so I have to run the launch many times until I get all 5. But once I get all the 5, then rarely less than 5 appear. Is this something you have experienced before as well? When they work, the red led in M4 is solid I think, and when they work the M4 is mostly green and red sometimes.

The funny thing is that the ones that get to communicate, are always changing and it does not seem to be dependent on which radio I pick for which CF.

[whoenig]

That is certainly not normal - my best guess is that there are communication issues.

• Make sure each radio operates on a different channel (i.e. 3 different channels in your case)
• Make sure the channels you use are not used by other 2.4GHz equipment. WiFi does channel hoping. If you can, turn off any 2.4GHz WiFi network - at least for close-by Access points. In general it might be helpful to switch to 5Ghz WiFi instead.
• Check the LEDs on the Crazyradios. Green means the package received ACK, red means it didn't. Make sure that you largely see greens. Otherwise it indicates that you use a bad channel or that there is some other interference happening.
• Make sure each CF has its unique address.
• Space the Crazyradios out a little bit, i.e. don't place them directly next to each other on a hub.

Hope some of that helps! The connection actually works quite stable for me using 6 CFs and 3 radios. /// I didn't try implementing a fix as of yet, but it's still on my list for this week!

[araujokth]

I did everything you said besides:

• disconnect wifi around the area where I fly, since that is not possible to do
• placed all antenas in a row on a hub :) I will go and buy some more cables to spread them apart.

For the CFs, I defined them all with a 10 channel number apart, so like channel 10, 20, 30, etc. What is happening is that I get mostly RED, but things never change. I only have one flying at the moment and all the rest and in the landing pads, but I guess the initialization blasts are done at around the same time and it creates massive interference. Will keep you updated!

Thanks again for all the help!

[whoenig]

There shouldn't be interference between channels as long as they are spread at least 2 apart (depending on the connection speed.) However, there might be interference with other 2.4Ghz devices. Did you try the channels independently (i.e. use a radio alone with channel 10 and see if it works)? I saw a lot of difference between channels. You can also use the RSSI capability to get a better idea on signal strength. Also, channels > 100 are not allowed to use (at least in the US), so those are usually not occupied...

[araujokth]

Thats great to know! so you mean that I should maybe set the channels to 101, 103, 105, etc. I will have 12 CFs so I need to have 12 different channels.

I tried all CFs individually and they all work fine. The thing is that the problem I am gettng(RED led mostly ON) always varies between the CFs. So sometimes its like only 2 green, some other times its 4, and the set that works is always changing. Pretty strange.

[whoenig]

Please try the issue27 branch to see if that fixes the issue. It is basically variant a), however you also need to add a sleep in the polling loop to make sure that a single thread doesn't stall the whole process. I only tested it briefly and it worked fine, but I would like to hear your feedback before I merge it to the master-branch.

How many Crazyradios are you planning to use? You only need to have different channels per radio, i.e. if you use 12 CFs with 6 radios you should only use 6 different channels (otherwise performance decreases).

The changing LEDs are a strong indicator for interference issues - I assume you are using the Crazyradio PA? Good channels to use depend on your location. You can implement a channel scan (i.e. switch channel, measure RSSI, repeat for all 128 channels) to find "good" channels. Unfortunately, that hasn't been done in the official code yet (it does require both firmware and client changes.) If you have WiFi or BlueTooth closeby this might not help due to the channel-hoping they implement. Bitcraze had a great idea about implementing channel-hoping for CF and Crazyradio as well. This would certainly reduce the problem you are facing, but it hasn't been finished yet. If you are familiar with the different firmwares you could try implementing it yourself (I could give you some pointers on how to do it.)

[araujokth]

Thanks for updating this! I tested it and it appears to be working so far! thanks a lot!

I am planning to use 6 radios for the 12 CFs. Thanks for the suggestion! Why would the performance decrease if 12 different channels would be used? I never thought of that.

Yes, I am having the crazyradio PA. Unfortunately I wont have time to look into the channel hopping. Since I only fly one or two at a time, and the other 11 or 10 are only turned on and not receiving inputs, I guess that I should in principle be OK with the interference right?

I have now placed the radios spread out instead of all in a single USB hub and it looks like the problem may be gone!

Thanks again for the help!

[whoenig]

The current firmware of the Crazyradio is not very efficient when it comes to changing the channel, i.e. you would pay an additional USB call for every channel switch. Since it operates in time-slicing mode, this will affect performance. Again, there are plans to improve that, but it is not finished yet.

If some of them are only turned on, you can lower their update frequency (default is 50Hz, but you can change it, see https://github.com/whoenig/crazyflie_ros/blob/master/crazyflie_controller/src/controller.cpp#L248). This way you reduce congestion even further.

Let the two you want to fly use different radios and lower the frequencies for the "static" ones to get better tracker performance.

[araujokth]

Oh thats great to know about the channels. Will change that today to try out.

On the frequency reduction, there is no problem of changing that dynamically right? Because the ones flying are picked up dynamically and the main application will not stop for a long time.

Thanks again for these great tips Wolfgang!

One problem I am having now is that I have 2 or 3 that are a bit crazy. I fly them, and after some time they just cannot track the point and fall, very similar to what happens when the battery becomes too low, but I have checked and the battery values are still high when this happens. I tried changing motors and propellers, but the behavior only gets slightly improved. Also after improvement, I see that they cannot track the waypoint well, and mainly stay hovering around 5cm of the waypoint for a very very long time before actually moving to the exact waypoint. Have you ever experienced that?

[whoenig]

The current ros driver uses a ROS parameter and hence it is static. It would be possible to switch that to either use dynamic reconfigure or a topic instead if you need it dynamically.

I had cases where they couldn't track anymore and it were mainly HW issues (propellers not balanced, battery not balanced). I actually never exchanged a motor to date. That they suddenly fall can happen if there is a communication outage or if the payload is too high.

For the 5cm hovering off target - you could try reducing the I-gains for X and Y (see https://github.com/whoenig/crazyflie_ros/blob/master/crazyflie_controller/config/crazyflie2.yaml#L5 and line 13). Sounds to me like it somehow picks up wrong corrections and then sticks with them. Even a PD controller should work reasonable for X/Y.

[araujokth]

Right, I was planning to have it as a topic. I was meaning if changing it on the fly could not possibly create a problem? But I will just try it and see how it goes. I was planning to leave the baseline at 10Hz, and then push up and down the ones that are flying. Actually I have been running all the tests at 100 Hz now, so that may have caused some issues :)

Hmmm in terms of payload, its very low since now I have only the QI charger, plus the markers in carbon fiber sticks. So it could be the comm outage. Do you recommend any good way to try to debug the comm?

I will try the controller change. Thanks for the tip! Do you think that this could be something that roll/pitch trimming could improve? I tried but with no apparent improvements.

[rs1990]

Hey iv been trying to do the same thing as well, to change the waypoints while the crazyflies are running by publishing them as a topic but i havent been able to. @araujokth were you able to get this done ? Id appreciate any help on this. Im sorry iv got limited programming skills and any help would be really appreciated.

Thanks,

Raghavendra

[whoenig]

I merged the fix in the master branch. Closing.