Payload mission log file / metadata

[LorenzMeier]

Describe problem solved by the proposed feature Right now Geotagging requires a full log file, which is overkill for just camera trigger locations and battery status, etc.

Describe your preferred solution We should have a way to download the mission logfile via telemetry. This should either work via MAVLink microservices (e.g. trigger events) or as one bulk MAVLink FTP file with only relevant events. Assuming we have 1000 events per flight and one event contains 8 double-precision floating point numbers, that would be 8 6 1000 = 48K on a 1 KByte/s link (out of which we have maybe 50% for logfile download, so 1.5 minutes for a fairly large flight.

That suggests it is not perfect from a UX perspective and sending the metadata during the flight might be preferable, as a flight with 1000 images would have been a fairly long mission.

Describe possible alternatives Logfile download via microSD, which is painful

Additional context The event interface plays also into this as this would be also things that are relevant. @bkueng @bresch.

[Antiheavy]

Some thoughts:

• The event log file (camera triggers, etc) should be in a universally accessible file format (.txt or .csv) and human readable. Ideally the file saved on the SDcard would be in this format, or at a minimum QGC should do the translation and save the file in one of those formats.

• time should be saved in a way to make it possible to sync one of the trigger events to an image (e.g. the first image and trigger) and allow programs (e.g. QGC) to match/sync the image save times (different reference clock) to the closest trigger time.

[FringerRinger]

This is somewhat externally based but what about how some of the companion computers do it. I have a Event38 companion computer that geotags the images in real-time, I connect it (it's a rpi) to telem2 (and my sony cam) and it pulls over all the geo info and adds it to the actual images as I take them (no post processing). I know that means there is an external board in the plane but maybe if we break it down a bit and see if there is something that can be done similarly since I think it's just a python script. Maybe no geotagging in real-time but at least the txt file for post processing. I might be off base here since this isn'y my forte but this definitely takes the headache out of it.

[Antiheavy]

Only certain models of Sony cameras allow you to write metadata to the image files from an external source. Most off-the-shelf cameras don’t let you do this. Of course, there are mapping cameras on the market that are specifically designed for drones and automatically geotag the images in real time, but not everybody uses those cameras.

[tubeme]

@FringerRinger event38 CC is not available anymore... I like the idea to have PC on board to take care for some tasks like this. But I'm not sure if there are any benefits in draining 2A for something that is not needed in real time. I agree that Pi could do tremendous work but rather for more complex tasks like visual analysis etc.

In every survey workflow I've seen so far, there is always, a transfer of files from Cam solid state storage to PC and then to Stitching software.

The idea is to make it as simple as possible and at the same time fast and convenient to make the parsing procedure.

1. If we make TXT CSV separate log we reduce volume and reduce the time needed to have the TXT at our disposal with telemetry or USB link.

2. If we push the TXT automatically to QGC after the survey and store it in library we reduce time as well and give convenience.

3. After the mission is done, on the field, while QGC is still open, we attach the Cam storage to a reader, we open the geo-tag page of QGC. We select the Cam storage as source, we select the survey log from the library and proceed.

4. This creates TAGGED folder on the PC and copys the images images from the Solid state (SD card) adding all the info - time, lat, lon, alt, pitch, roll, yaw in the metadata. At the same time copy's the TXT file from the QGC library to TAGGED folder, make the parsing by time and inserts the names of the image files in the TXT. This way we will have all the images and the indexed TXT file ready for post production in one folder, after just a couple of minutes after the mission is over.

In mapping we always have the copy process and transfer of files from a solid state medium to PC. We just make QGC do the copy process and at the same add its value to the transfer procedure.

[Antiheavy]

Is the concept here to create two log files for each flight? 1) a full data log (current .ulg) 2) a much smaller "event log" (new .csv)

I suppose both types of files would be viewable and downloadable in the QGC log download interface?

[tubeme]

Yes that is the idea...

And yes both types of files should be downloadable from QGC. the ulg file for flight analysis purposes, which should happen relatively seldom given the PX4 is well set and the craft performs its regular duties.

And the CSV file for everyday use for Mapping and other purposes if there are other events in the future.

[LorenzMeier]

Update on this: @bkueng is starting to look into it.

[Antiheavy]

Here are some more thoughts on requirements/desirements from my perspective.

I put the need for this into 2 categories:

1. Image geotagging after flight
2. Quick and simple event log for extracting: flight statistics, system status changes (sensors, estimator, flight modes, etc), user comands

If the log contains both categories of information, the Geotagging tool (e.g. the one in QGC) will need to know how to parse the image triggers and metadata.

These logs should be a simple readable format, like .csv

These logs will be used after the flight so real-time downloading (or background downloading) during flight isn't critical. However, it might be nice for a number of reasons, for example it might be nice to just have the logs available right there on the laptop/tablet in an immediately accessible folder location. If downloading during flight is the approach, lossy datalinks must be considered, also what happens to the logs in a loss of commnuications RTL event, etc.

Logs should always be stored on the SDcard and be available to download over the datalink upon request (e.g. via QGC). Download time of a 1000 event log should be < 1 minute with 57kbaud serial modems (few seconds with megabit class datalinks).

The image triggers and metadata should contain (at a minimum):

• GPS date and time
• Latitude (wgs84 decimal degrees)
• Longitude (wgs84 decimal degrees)
• Altitude (amsl meters, e.g. egm96)
• Accuracy horizontal (meters)
• Accuracy vertical (meters)
• Roll (degrees)
• Pitch (degrees)
• Yaw (degrees)

[tubeme]

@Antiheavy Very accurate description thanks. I could give some ideas on top.

Information Formatting for Pix4D

Here is what we need for the Pix4D.

Omega,Phi,Kappa are the oriantation angle of the camera lens plane.

IE in perfectly aligned camera to IMU the Omega = 0, Phi = 0 and Kappa = 90 (the camera is landscape and regarding the Z plane it is rotated at 90 degrees.)

The idea is to store more events in the Quick and Simple event log than just the events needed for geotagging. So in this quick log we can store Roll, Pitch and Yaw for the cam trigger events, but then the geo tagger (parser) could calculate the omega, phi and Kappa and export it to the CSV file that is needed by the Pix4D.

Here is the format of the file needed by Pix4D.

https://support.pix4d.com/hc/en-us/articles/202558539-Input-files#label2

I don't know how our current parser is implemented. But I think using RTKLib http://www.rtklib.com/ is what everybody else is doing. Emlid is using it in their reach https://emlid.com/reach/. So if we use RTKLib we could do a lot more regarding photogrammetri.

Workflow:

Having this Quick log download in real time is not high priority. Even I don't know if it is needed at all. You cannot geo tag without images, and you cannot have images downloaded in real time unless you use some 4G / LTE which is pointless for mapping work.

But despite this it is good idea when the craft detects landing to push the Quick log to the QGC with a warning window pop up on the QGC screen.

There is file structure already in the Documents:

we could add Events Log folder and store the quick logs there.

When we open the LOG view in QGC we could see a list sorry for the example.

The list could be like Mission Name, Time, Date, Flight Duration, Flight Distance, Average Speed, Relative altitude.

Then you select one of the logs you point to where the images are and push GEO-TAG button.

Geo tagging is the same as the current geo tagging with small addition:

you choose source folder that holds the images and destination folder to copy the images with POS meta data. + generate the CSV file with omega, phi, kappa angles formatted for Pix4D in the tagged folder.

From then on there could be made a viewer that can view all the event types as you mentioned "flight statistics, system status changes (sensors, estimator, flight modes, etc), user comands" We can make a quick viewer for the mission on the map etc.

That would be more of a simplistic log file for the user to interact with. The average user does not want to know what a .ulg is and how it works. ULG will allays stay in the professional hands in one way or the other and I hope (pray to god) used very very seldom.

[tubeme]

Sorry something happen with one of the images:

[tubeme]

@Antiheavy GPS time and Date are not needed for the matched images in the generated csv file from the parser. because the Pix4D anly wants to see the coordinates of the image. The time and date is used during the parsing process to match real precision trigger timing to image camera captured timing.

[tubeme]

Thanks @LorenzMeier

[bkueng]

Thanks for all the input here. Let me collect what I have so far.

Use-cases

These are the use-cases that I see for a mission log:

• geotagging/survey (more generally: extract data collected by the pixhawk during a mission and needed for post-processing)
• cloud upload & statistics: keeping track of where and how long a vehicle has flown, and if there were problems.
• regulation: keeping a small log to show where a drone was flown, what failures happened etc.

So it should contain:

• position + velocity + time
• events (failures, system errors, ...)
• flight mode
• battery levels
• camera triggers (with time, attitude, position, sequence number, ...)
• general firmware versioning & hardware info (similar as we already have in the ULog header)

Hardware setups

The following setups should be supported:

• Pixhawk w/o companion (SD card is fixed inside, no USB port accessible), with telemetry
• Pixhawk w/ companion (optionally with LTE)

Implementation

I plan to implement it as following:

• I know you ask for CSV, but there are various reasons to go for a binary file, or ULog in particular:

  • you lose accuracy when storing floats in a CSV (or you need to store lots of digits)
  • CSV is less efficient in terms of size
  • The mission file contains more data than required for Pix4D. So a conversion step is necessary anyway. From the ULog the CSV for Pix4D can easily be extracted.
  • ULog is easily extendible: we can add/remove data as needed.
  • We already have good tooling available for ULog, in particular for the cloud and log streaming
  • It will allow us to scale up and down in terms of what the full log vs. mission log contains.
  • The full log can just be a superset of the mission log.

  Therefore I plan to use ULog.

• events: since it's ULog, they can easily be added
• retrieve logs in real-time:
  • telemetry: since the connection can be lost during flight for a longer time, we would need arbitrary buffering, so I don't plan to support that. Also it sounds that it's not a required feature.
  • However I still want to enable having the mission log available in real-time for LTE. We can enable this by using the existing ulog log streaming. First only either the full log or the mission log, and if needed both in parallel later on.
• log download: it can happen through FTP or the existing log download in QGC.
• estimated log size:

data	message size (B)	update rate (Hz)
events	27	1 (generally there should be less)
camera trigger	64	0.3 (assuming triggering every 3 sec)
position	67	1
flight mode+battery	109	0.2

Which gives almost 1MB for a 2 hour flight (and 2160 images). This is still quite large for telemetry download, but we can scale it down, in the extreme case to only the camera trigger message, which gives 138kB, and around 3min download time over a 1kb/s link.

Does that sound ok?

[CarlOlsson]

Great summary, this feature is really needed. Two comments:

• 0.3 Hz trigger speed is very low, 1 Hz is probably more realistic. Even that is slow for some consumer cameras.
• Not 100% related but the failure detection, do you plan to add all detection to the failure detector in commander or distribute them over all modules? E.g. say you want to detect and report a malfunctioning airspeed sensor, either you do that in the ekf since that module already has all the data or you subscribe and do the detection in another module.

[tubeme]

And some more notes:

We need to be able to transfer the file from the telemetry. That is because it is better that the file is pushed to the QGC after landed state. It is fine to be up to 2-3mb I think. It transfer reasonably fast. We can make a warning window in QGC to not stop the data link until upload finished. This will make the tagging more integrated and without the need for human intervention for the transfer.

I understand that there will be a second smaller ulg to hold all this data right?

There is important development regarding the Mapping Precision:

https://github.com/PX4/Firmware/pull/10295#issuecomment-425817504

that should be taken in to account for the camera triggers logging.

We need the CSV file after the parsing due to legasy support reasons. The same data will be written in the images itself I suppose without loosing on accuracy correct directly from the ulg?

Then when we open GeoTag page we can have something like this:

Of course this is my wishful thinking.

In this page in the list you see all the ULGs that are downloaded to the QGC automatically.

You select one from the list, select images folder and target folder and hit GeoTag.

In the destination folder we need the geotagged images with extedned info precision+Attitude. and the CSV file generated from the ULG.

That is just what we need for the mapping.

The other part with the flight statistics problems errors etc, may be there should be some viewer made in the Log section.

[tubeme]

@LorenzMeier @DonLakeFlyer @dagar @dogmaphobic

I think "Mission names" could be a good human layer to the mission interface. Something to name and identify easily later when you have to parse or look in the archives,