Tool To Convert Ardupilot Telemetry Log To CSVs and Excel

[jeffstjean]

Overview

This mavlogdump.py script let's us take a telemetry log (.tlog) file from QGroundControl and turn parse it into some more useful files.

It's fairly easy to convert files to JSON format. If you had a file called telemetry.log, you can convert it to JSON by running the command:

python3 mavlogdump.py telemetry.log --format json > telemetry.json

and it'll spit out a telemetry.json file with all of the messages. However, we want to be able to output this telemetry into a CSV format as well. Luckily the tool can output CSVs with the --format csv option, but you'll run into an error:

$ python3 mavlogdump.py telemetry.tlog --format csv             
You must specify a list of message types if outputting CSV format via the --types argument.

You can see the list of available types like so:

$ python3 mavlogdump.py telemetry.tlog --show-types
GLOBAL_POSITION_INT
SERVO_OUTPUT_RAW
RC_CHANNELS
SCALED_IMU2
AIRSPEED_AUTOCAL
TERRAIN_REPORT
REQUEST_DATA_STREAM
...

And output a single CSV file (eg. for GLOBAL_POSITION_INT) with:

$ python3 mavlogdump.py telemetry.tlog --format csv --types GLOBAL_POSITION_INT > telemetry.csv

You can even specify multiple types to output to the same CSV file:

$ python3 mavlogdump.py telemetry.tlog --format csv --types GLOBAL_POSITION_INT,SERVO_OUTPUT_RAW > telemetry.csv

However, the resulting CSV is quite ugly.

Since these messages all arrive at different times and at different intervals, you can't just put GLOBAL_POSITION_INT on the same row as SERVO_OUTPUT_RAW. It would be much nicer to have these all in separate CSV files to make it easier to inspect. Better yet, it would be great if each message was a separate tab (worksheet) in a single excel workbook for easy analysis.

Task

The task is to create a Python script called mavcsvexport.py that uses the output of mavlogdump.py to generate a CSV files for each telemetry message or generate an excel workbook where each telemetry message has its own worksheet. It should choose the output format based off of command line arguments.

To output a bunch of CSVs, the user could run the command:

$ python3 mavcsvexport.py telemetry.log --csv-dir ./csv-files --csv-prefix telemetry-

This script should then create a directory called csv-files/. It will get the list of messages from mavlogdump.py and export each message to a file in that directory with the specified prefix. For example, all the GLOBAL_POSITION_INT messages would be inserted into a CSV called ./csv-files/telemetry-global_position_int.csv.

To output a an excel workbook, the user could run the command:

$ python3 mavcsvexport.py telemetry.log --excel telemetry.xlsx

This script should then get the list of messages from mavlogdump.py and put each message into a different worksheet of a newly created excel workbook called telemetry.xlsx. For example, all the GLOBAL_POSITION_INT messages would get inserted into a tab on the spreadsheet called global_position_int.

The user should be able to output both CSVs and excel workbooks by running the command once. This means that the following command will create a csv-files/ directory with a list of CSV files and a telemetry.xslx file.

$ python3 mavcsvexport.py telemetry.log  --csv-dir ./csv-files --csv-prefix telemetry- --excel telemetry.xlsx

Parsing Arguments

You can parse arguments from the user by using a module called argparse. You can read about how to use this module here and view the full documentation here. There's also a few examples of using argparse in the tools/ directory of this repo here. Try to follow a similar style of using a parse_args() function.

Running mavlogdump.py

You can run the mavlogdump.py script using Python's subprocess module. You can read about how to use this module here and the full documentation here. Basically, you'll want to run the script and capture the output using the capture_output option. You can read the stdout property of the resulting object to get the text that otherwise would've been printed to your terminal's screen. You can then manipulate it however you want (eg. if you used the --format json option, you can use Python's json.loads() function to convert it to a Python dictionary to manipulate.

CSV Writing

You can write CSV files using Python's csv module. Read about how to use it here and the full documentation here. You can iterate through the dictionary to find all the messages that have the same type and use the csv module to write those messages to a file. Repeat for each message!

Excel Workbook Writing

This one is a little advanced but there's an awesome third-party Python library called xlsxwriter. Read about how to use it here and the full documentation here. Similar to the CSV writing, once you have lists for each message type, create a workbook with a tab for each message type and write the list of messages!

[jeffstjean]

You can use this file under PADA > PADA 2024 > Software > Dataset 3 as an example input file.

https://uwoca.sharepoint.com/:u:/s/AeroTest/ESFKGEh4H1JDiFB6ofCvSlwB3o3TzNOLmWNPA6K08_jECQ?e=UDvlAg