DataLogging, takes too long

[GregoireH]

As it was discussed in pull request #89 Sometimes, logging takes way too much computational time. Which leads to some hudge delay in the stabilization loop !! (140ms delay compare to a period of this task of 4ms => we miss 35 stabilization loops...)

We should spend sometime spoting the issue and improving this. Otherwise we will have to go back to something like writing bytes on the SD card, which is not that user friendly compare to having a Fat fs system file manager.

Best

[jlecoeur]

A way to reduce the write time is to use binary format. For example, logging 3 floating point variables takes:

• csv text: 27 bytes, with a precision of 7 digits : 3 x ( 7 bytes for the data, +1 bytes for dot, +1 bytes for commas and end of line characters).
• binary: 12 bytes

[mdouglas90]

I'm going to bring this issue back as I've found some issues that seem to be caused by data logging.

There appears to be some delay that is caused by data logging. When this delay happens in the drone dome, there is frequently a spike in the position estimation. Sometimes upwards of 1 km (so there may be something wrong somewhere else as well) but usually on the order of a couple meters. The weird this is, is that it seems to be consistently occurring every ~140 s.

This is a graph where the drone was just sitting in the drone dome for 1 hour. The y axis is the x position estimation during that time. As the drone is still, it is expected that the position estimation should be close to 0 for the entire time; however, you can see the consistent spikes every 140 s.

Zooming into these spike,...

...we can see that there are several groups of data logging points that are missing. I assume that if the data logging points are missing, then we are also missing stabilization calls. Some gaps on other spikes reached 1200 ms.

The gaps also happen every 140 s when not using the drone dome as well, but the spikes do not appear to be there, see figure below:

I also tested the code by reducing the data logging frequency from 100 Hz to 10 Hz. With this, the gaps/spikes seem to occur 10 times less frequently, i.e. the first gap occurred at 1352 s. As the test only was 1450 seconds, there was only 1 gap. But it does seem that the data logging is what is causing these gaps to occur.

[jlecoeur]

I think these delays are inherent to FatFs. Apparently writing 512 bytes at a time helps in terms of write speed, but there are always periodical delays due to housekeeping inside FatFs. I see two solutions:

• ditch FatFs, directly dump binary data to the SDcard. I think we could write mavlink messages to the SDcard as it would be more fault tolerant, and relatively easy to read.
• use an RTOS, with data_logging in one task, and the rest in another task.

[jlecoeur]

In the non-RTOS case, write speed could be improved with a DMA version of the SD SPI driver. We would write 512 bytes in a first DMA buffer, then trigger the DMA transfer while we keep writing in a second DMA buffer.