search

ArduPilot / ardupilot

ArduPlane, ArduCopter, ArduRover, ArduSub source
http://ardupilot.org/
GNU General Public License v3.0
10.98k stars 17.52k forks source link

Size limitation of dataflash log #15747

Open bigboy061293 opened 4 years ago

bigboy061293 commented 4 years ago

Feature request

The large log file (dataflash log) is difficult to copy/load/examine. Usually, I just log the IMU and GPS data, it took more than 1GB for 1 hour of logging. Could we have some feature to choose the maximum size it could be. It is better for loading and even much better for downloading via Mavlink.

Splitting the large file into smaller ones should be considered. Thanks.

Platform [ x ] All [ ] AntennaTracker [ ] Copter [ ] Plane [ ] Rover [ ] Submarine

WickedShell commented 4 years ago

I'm not sure what the maximum size benefit would be? At some point you will either have to just stop logging, or rotate to a new file. That leaves you with the same overall amount of stuff to download so I'm not sure it's worth it.

Log download over MAVLink is very slow, that's a definite issue we are stuck with for now.

You can change your logging bitmask to reduce the amount of data you log. I tend to push large file sizes relative to most people and I'm only at around 300-400MB an hour. Even with that I find I have data drop outs in the log. What setup are you using/how are you coping with dropping data? (And if it doesn't matter to have all the raw data, could I just suggest some slower logging)

bigboy061293 commented 4 years ago

I'm not sure what the maximum size benefit would be? At some point you will either have to just stop logging, or rotate to a new file. That leaves you with the same overall amount of stuff to download so I'm not sure it's worth it.

Log download over MAVLink is very slow, that's a definite issue we are stuck with for now.

You can change your logging bitmask to reduce the amount of data you log. I tend to push large file sizes relative to most people and I'm only at around 300-400MB an hour. Even with that I find I have data drop outs in the log. What setup are you using/how are you coping with dropping data? (And if it doesn't matter to have all the raw data, could I just suggest some slower logging)

The problem does not just come from slow MAVLink download but analyzing. For example, using FFT for 1-hour log (nearly 1GB) usually cause crashing; transferring the log into Matlab also took forever. If we have the vehicles working for 2 hours or more, is it impossible to use the log? Just try yourself to use MP for plotting GPS and IMU only, ticking a new parameter will be a century.

I am also curious how could we stop and restart the log file? (any methods: manually, using commands or scripts?) It is also be fine if we manually split the large file into smaller ones in the post processing stage. Thanks.

WickedShell commented 4 years ago

You can filter and extract sub parts of a log in advance which helps a lot. MAVExplorer has generally gotten much more responsive recently with adding more and more graphs, although the panning of a large graph while zoomed out is still painful. (And I don't find MAVExplorer to be all that stable on my system).

Anyways if you just want to split the log into parts that can easily be done afterwards using mavextract.py or similar.