Out of memory error when loading large dataflash logs (>20MB)

[mikerob]

Loading a large (50MB) dataflash logs results in an "out of memory" error. 20MB seems to work fine. Don't have anything in between to test!

While this may sound like excessively large logs, EKF really blows out the log size, AND requires detailed log analysis.

https://drive.google.com/file/d/0B9AHb7NB3l6gRVBHUEZTcG1fbWM/edit?usp=sharing

[meee1]

What no version

[mikerob]

Sorry, 1.2.99 build 1.1.5197.12871

[meee1]

can I get you to test one thing.

convert the .bin file to a .log file then try opening the .log file.

[mikerob]

Converting the 20MB .bin file to a .log works fine - it ends up as a 50MB .log file which will open (but the 20MB .bin opens as well).

A 27.5MB .bin file fails to open. This one gave me a difference message: https://drive.google.com/file/d/0B9AHb7NB3l6gd2xuU2NFNktadE0/edit?usp=sharing

It converts to a 74.8 MB .log which opens.

A 99.3 MB .bin file fails to open. It converts to a 251 MB .log which just plain breaks MP!

A 51 MB .bin file fails to open. It also said "Log Browser will not work without FMT messages in your log. It appears your log does not have FMT messages". It converts to a 136 MB .log which kind of opens - it doesn't show the table of data, and MP has real problems trying to graph it, then goes "Not Responding".

Looks like there are a few places things can break - either flat out running out when it tries to read it, or if it reads successfully it could be maxxing out the actual table display. When it's the table error, you can still graph the log.

If the table display issue is a built-in limit of the GUI, maybe you could allow selection of which fields from a log to read?

[mikerob]

In 1.31 build 1.1.5229.30673, larger logs seem to load - a 36MB .bin just opened fine, and I've had similar experience with logs around 20MB that didn't show the table in the previous version.

The 99MB .bin still fails :(

[meee1]

im still working on this one. its not going to be easy to fix.

[mikerob]

A couple of suggested options if it's proving intractable: 1) A function to split the .bin logs into smaller chunks 2) The ability to load only a portion (%) of the log for analysis at a time 3) The ability to only load some of the logged parameters

These would probably enable most of the functionality, but I guess the question is whether the patches would take longer than the proper fix.

I've got perl scripts that can do most of this, and I think there are python processing ones as well so it may not be that high a priority.