search

ArduPilot / ardupilot

ArduPlane, ArduCopter, ArduRover, ArduSub source
http://ardupilot.org/
GNU General Public License v3.0
10.96k stars 17.49k forks source link

All: Dataflash log file size rotation #5356

Open gmorph opened 7 years ago

gmorph commented 7 years ago

Some users run their vehicles for a VERY long time. In the case of boat users they often run 24 hours a day over several days. At the moment his generates one monolithic huge log file. First question is what happens when the SDCard becomes full and its only got 1 file on it and its the file currently being written to? Ideally at a certain point we would split the log file into separate files of say 1Gig each for ease of handling. There are a few hurdles to overcome.
All the self describing information, parameters etc are gathered at the beginning of the log file so we either have to copy that stuff to the new log file as its created and treat each log file as self contained OR assume that subsequent log files are part of the first one and we need to stitch them back together after download or have the log file processing tools be able to handle multiple files. If we do split into multiple files we still need to decide what happens when the SDCard becomes full.

lvale commented 7 years ago

Easy, A Parameter to enable rollover logs, ie, when the card is full erase the oldest log and start a new log file. If the Parameter is to not enable rollover logs, than logging should stop. User choice. Shouldn't also the log files be less than 4GB due to file system constraints ?

magicrub commented 7 years ago

Luis,

That is already in place on boot. Problem is if there are zero logs and then you create a single huge log then it will sort of just fail right now. Grant is talking about logging for multiple days, we need a way to stop logging, then start a new file with new headers and such and continue logging and then if you run out of room then we do what you were saying.

On Dec 9, 2016 7:52 PM, "Luis Vale Gonçalves" notifications@github.com wrote:

Easy, A Parameter to enable rollover logs, ie, when the card is full erase the oldest log and start a new log file. If the Parameter is to not enable rollover logs, than logging should stop. User choice. Shouldn't also the log files be less than 4GB due to file system constraints ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ArduPilot/ardupilot/issues/5356#issuecomment-266176877, or mute the thread https://github.com/notifications/unsubscribe-auth/AEj7G-r21WF9D9fnjEPtdzsO3K3e35u_ks5rGiHxgaJpZM4LIjlB .

lvale commented 7 years ago

Yes @magicrub I have a 4GB full card of logs. This situation touchs everybody. My 2 suggestions are to enable the rollover and limit the file size to 4GB for a single log file.

magicrub commented 7 years ago

File size should be smaller. I agree with the 1GB limit that someone mentioned earlier

lvale commented 7 years ago

@magicrub the 4GB is absolute max due to file system restrictions, and it's not a good size (too large) to deal. The 1GB is a good proposal

peterbarker commented 7 years ago

On Thu, 8 Dec 2016, Grant Morphett wrote:

Some users run their vehicles for a VERY long time. In the case of boat users they often run 24 hours a day over several days. At the moment his generates one monolithic huge log file. First question is what happens when the SDCard becomes full and its only got 1 file on it and its the file currently being written to? Ideally at a certain point we would split the

Logging will stop within a few percent of filling the SD card to avoid corruption which NuttX does not handle well.

log file into separate files of say 1Gig each for ease of handling. There are a few hurdles to overcome.

Rolling the log file over at 1G seems reasonable to me. We can parameterize it the first time someone complains their Linux-based-autopilot has 1 1TB SSD on it and that this is dumb :-)

All the self describing information, parameters etc are gathered at the beginning of the log file so we either have to copy that stuff to the new log file as its created and treat each log file as self contained OR assume that subsequent log files are part of the first one and we need to stitch them back together after download or have the log file processing tools be able to handle multiple files.

We could add a "critical" message to mark a log as a "continuation log", possible referencing the original log somehow. Maybe we could have a META message which has a UUID in it, and continuation-logs reference that?

If we do split into multiple files we still need to decide what happens when the SDCard becomes full.

Yes, this is the thorny part. Does the user want to keep the most recent logs or the oldest logs? Sadly, a parameter is probably the only option here.

magicrub commented 7 years ago

I reluctantly agree to all of that, including the params.

On Dec 11, 2016 3:24 PM, "Peter Barker" notifications@github.com wrote:

On Thu, 8 Dec 2016, Grant Morphett wrote:

Some users run their vehicles for a VERY long time. In the case of boat users they often run 24 hours a day over several days. At the moment his generates one monolithic huge log file. First question is what happens when the SDCard becomes full and its only got 1 file on it and its the file currently being written to? Ideally at a certain point we would split the

Logging will stop within a few percent of filling the SD card to avoid corruption which NuttX does not handle well.

log file into separate files of say 1Gig each for ease of handling. There are a few hurdles to overcome.

Rolling the log file over at 1G seems reasonable to me. We can parameterize it the first time someone complains their Linux-based-autopilot has 1 1TB SSD on it and that this is dumb :-)

All the self describing information, parameters etc are gathered at the beginning of the log file so we either have to copy that stuff to the new log file as its created and treat each log file as self contained OR assume that subsequent log files are part of the first one and we need to stitch them back together after download or have the log file processing tools be able to handle multiple files.

We could add a "critical" message to mark a log as a "continuation log", possible referencing the original log somehow. Maybe we could have a META message which has a UUID in it, and continuation-logs reference that?

If we do split into multiple files we still need to decide what happens when the SDCard becomes full.

Yes, this is the thorny part. Does the user want to keep the most recent logs or the oldest logs? Sadly, a parameter is probably the only option here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArduPilot/ardupilot/issues/5356#issuecomment-266316545, or mute the thread https://github.com/notifications/unsubscribe-auth/AEj7G72TLw7D9gGPirwqZrxhcUGIdpH6ks5rHIZEgaJpZM4LIjlB .