Program no longer reads the log files with the new Beta Firmware.

[KBFlyer]

Tried to read my log files from the beta firmware today and it would not read them. Just thought you would like to know something has changed. Also when download the log files there is no more peeping which is nice.

[tomSny]

Thanks, will look into once I upgrade.

[KBFlyer]

I also emailed DroneLogBook about the problem. Not sure what they did to resolve the problem but some of the info seems to be back when you upload a log file on their site but if you convert it to the GPX format from there and try to use it in Dashware it is useless information.

[tomSny]

I finally got around to looking into this issue, yes they did make significant changes to the log, the structure size has been increased and most of the log types have been replaced with a new set of log types. I will release a new version after the beta that will support the new and old formats. If I get enough buzz, I could release a interim beta release.

[meclark]

An update to handle the new log format would be great! Thanks.

[yrless]

Thank you tomSny for looking into this.....just noticed the problem myself this evening when trying to view some logs after the move to V2 firmware.

[KBFlyer]

Thanks Tom for looking into it. Getting it to work again would be great in the interim.

[meclark]

I've modified the Python script, sdlog2_dump.py so that it works with the new format. My changes are probably applicable here. Let me know if you're interested and I'll send you a list of my modifications.

[yrless]

The sdlog2_dump.py that downloaded with the log file out of my XSP (with V2.0.x FW) worked well without mods. Hopefully I'm not missing anything :)

[meclark]

Thanks yrless. Your experience with sdlog2_dump.py and post-beta log001.bin files hasn't been my experience. Could you do me a favor and run "sdlog2_dump.py log001.bin -m VER_Arch" and note the version number that appears? On my system (beta firmware) I see "V2.00.27" for new files and "V1.08.40" on a pre-beta file. Another check is to look at line 40 in sdlog2_dump.py: MSG_FORMAT_PACKET_LEN. Pre-beta requires a value of 89, post-beta value is 333. (The only version of sdlog2_dump.py I've been able to find is Version 1.2. If you've got a later one, I'd love to see it.) Thanks

[yrless]

Hi....here's the script that I ran on V2 FW......if I get a chance tomorrow I'll give yours a run with my data but give this one a shot.

sdlog2_dump.zip

[yrless]

Here's what I see when I run version....its V1.08.42.....slight difference

[yrless]

ok...silly me.....I think I just noticed that the log file I was operating on came from pre-FW2.0! So in fact perhaps my python script is NOT up to date.....my bad! I just purchased a second XSP that has FW2.0 installed so I made the assumption that all the log outputs were from 2.0....it appears that the first 2 flights were done on FW 1.0 which is where I got my data.....the rest of the flights are FW2.0...dang!

[meclark]

yrless, Yep, that script won't work with the new format. I'll clean up the script that I'm using for the new format and drop it here tomorrow. By the way, "VER_Arch" appears to be the flight controller firmware version -- I had mistakenly thought it was the file format version.

[yrless]

Thanks meclark! Appreciate your efforts....I'm looking inside the script now and can verify the value of "89" on line 40. I guess the folks at OrangeLand are keeping the new script to themselves for the time being...perhaps until the FW is GA'd.

[meclark]

Here's a copy of sdlog2_dump.py that will parse the log00x.bin files generated by Autel X-Stars using the updated firmware. sdlog2_dump_rev.zip

Note: there appears to be a bug in the file format as it's currently being written by the X-Star. I've kludged the script to deal with it (comments are in the script.) This suggests to me that the folks Autel are no longer using this format in house. This is pure speculation on my part, but I'm assuming that with the recent update to StarLink, which now supports log files in the app, the older format is not being used. (I examined the new format but lack the skills to make any sense of it. I'd be interested to hear if anyone has any info about it.

[yrless]

meclark, thank you so much for the file.....it appears to produce the csv perfectly! I've attached a screen shot of my command window just as reference so you can see if it matches your results...thank yoy again!

[yrless]

I do see some issues with the new format.....it could be that some column names have changed but it seems like some critical columns I use for heading data are gone..IMU AccX, IMU AccY, IMU AccZ, IMU MagX, etc, etc.

[meclark]

yrless, Your screenshot indicates a problem, The struct.error message suggests that the script has encountered a problem with a format description string similar to the one I patched for the "MISSION" message. My guess is that you've got a field in your files that I don't have in mine. It probably also means that your CSV output file is incomplete.

[yrless]

Yikes......ok, if I get some time I'll dig into it but I'm a radio engineer not a coding pro so I may have to live with it at least in the short term :( Thanks for your assistance!

[meclark]

yrless, If you're interested in looking a little deeper, try this updated version. sdlog2_dump_rev.zip I've added an argument. Run it with args of "-v -c" and it should dump the information about the message format that's causing the error. Use: python sdlog2_dump.py log001.bin -v -c I'd be curious to see what you get. You should see something similar to this:

Malformed format string: <BBBBiihhiihBBBBhIiiiiBBhiiIIBBBBHBBBBH in message MISSION Error: ('unpack requires a string argument of length 84',)

[KBFlyer]

This is what I get.

C:\log\conv>python sdlog2_dump.py log001.bin -v -c Traceback (most recent call last): File "sdlog2_dump.py", line 390, in _main() File "sdlog2_dump.py", line 387, in _main parser.process(fn) File "sdlog2_dump.py", line 190, in process self.parseMsg(msg_descr) File "sdlog2_dump.py", line 276, in parseMsg data = list(struct.unpack(msg_struct, self.buffer[self.ptr+self.MSG_HEADER_LEN:self.__ptr+msg_length])) struct.error: unpack requires a buffer of 84 bytes

[yrless]

Same results as KB.

[yrless]

Here's another shot with just the -v -c arguments.....no request for export.

[meclark]

yrless, D'oh!, You're (probably) running Python 3 and I'm lagging behind at Python 2.7.x. I'll update my script and get a new version out this afternoon. Sorry.

[KBFlyer]

OK so I am trying to follow along but I am just learning some of this stuff off of you guys. I can upload my log file from firmware V2.0.8 to dronelogbook.com and it will show my route in doarama.com and google maps. Dronlogbook allows you to download either a IGC file or a KLM file. I choose IGC. Now that has to be converted to something dashware can read. I did find this post on GitHub. But as a beginner I am kind of stuck. https://github.com/OverloadUT/IGC2CSV

[yrless]

meclark...you're correct...I'm running 3.7 :) Thanks again for the follow-up!

[meclark]

yrless, try this, it should work with Python 3.x: Rev: 2.0a01 - sdlog2_dump_rev_20a01.zip Run with "-c" argument and you should get something like this:

python sdlog2_dump_rev.py log001.bin -c

Malformed format string: <BBBBiihhiihBBBBhIiiiiBBhiiIIBBBBHBBBBH in message MISSION. Error: unpack requires a string argument of length 84. Skipping all subsequent occurences of MISSION.

Also, I've added a couple of other args: -V (uppercase) will output the current version; -k will turn OFF my kludge for fixing malformed "MISSION" messages when writing to .csv files, useful for testing perhaps. Good luck.

[yrless]

KB...looks like that export option is a Paid service? I don't see export in the freebie version so I can't confirm being able to convert to IGC format but if you can use that converter to put the data into a CSV file then Dashware is just a small step away...one note....the site says the IGC2CSV utility has been tested on Python 2.7.5.....you may have loaded 3.7 so keep that in mind when attempting to run.

[yrless]

it ran meclark.....but the resulting output looks nothing like the stuff we get from 1.1.3 FW....totally different column names...in fact I think the only column names I recognize are Lat/Lon.....when you run the script on a bin file from 2.0 FW do you still see any column names like IMU Magx, IMU Magy, IMU Magz, IMU AccX?

[meclark]

yrels,

When you ran the script with a "-c:" arg, what was the output? I'm really curious to see why it was failing before.

Regarding the missing column names -- they're gone. In the terminology of sdlog2_dump.py: data is stored in Messages, i.e. GPS, ATT, IMU, etc. Messages have Fields, i.e. AccX, AccY, AccZ, GyroX. . . The combination of Message_Field make up the columns you see in the cvs file. In the log00x.bin file I looked at, only 4 or 5 Messages remained from the old format. They were GPS, TIME, VER, and PARM. All the others have changed. Some obviously carry info similar to the old format but under a different name. Most, though, are too obscure for me to intuit. The new .bin file format allows more detailed information in the Messages/Fields, which is probably why they were changed.

I think I'm pretty much done. I posted here because I modified a sdlog2_dump.py to process the new log00x.bin files and thought the info contained might be applicable to others who were writing log software for the Autel X-Star. I don't have of a source for documentation for the format, so I'm of no help there. I'll be interested to hear if anyone makes progress. Thanks for all the useful testing of the script. mc

[yrless]

Hi meclarck.....here's a screenshot of what I get when I run with -c argument......I think I'm kinda done as well until I can get clarity on what the new column headers mean....when I look at a few of them it almost seems like they might be directly related to the old data but with scaling factors of 1000, etc.....no way to prove this at the moment. I was using the magnetometer outputs to calculate heading and display on screen as this is of interest to certain parties I deal with...now I'm dead in the water wrt this capability...hopefully this will get sorted in the coming weeks.....thanks again for your help....much appreciated.

[meclark]

yrless, thanks for the screenshot. That clears up a mystery for me: Your file has the same malformed format string as mine but it wasn't being fixed because I wasn't handling python 3 properly.

[KBFlyer]

Taking one of my current 2.0.8 bin files and uploading it to Dronelogbook and then using their download of a IGC file from that flight I then used the conversion option on gpsvisualizer to give me a GPX file. With this data GPX file I am able to load it in Dashware. It is missing a lot of data I use on my dashware faceplate but some of it is still there. One thing from dronelogbook they are able to show you battery voltage from start to finish. This information is not given in the downloaded IGC file. Would be nice if we could file in the blanks.

[yrless]

Interesting....I'll have to give that a shot next week perhaps with a simple up and down flight with known headings......I see that the the altitude field is blank but that can be calculated in dashware. Thanks for the info.

[KBFlyer]

I have emailed Dronelogbook about the missing information and they seem to be actually looking into it. They have the resources and knowledge to perhaps fill in the blanks. They passed it on to their coding team. Not saying anything will come of it but if you don't ask it will stay the same.

[yrless]

Great...thanks for looking into that....I was able to do a couple of controlled flights with known headings, etc but haven't had the time to pull the logs....hopefully I'll get to it this week. Seems like you're the first guy to inform dronelogbook of the changes :1st_place_medal: Would you like me to email them as well to reinforce the request?

[KBFlyer]

Could not hurt I guess. They put out a monthly update about the improvements they have made during the month and ask you to respond with any improvements that you might like to see so I sent in my request for more info in the IGC file we can download. Also asked if they could make a Dashware specific file to import right into dashware.

[yrless]

I just left Dronelogbook a note about the issue.

[yrless]

The Dronelogbook folks responded back to me and requested a new log file which I provided this morning...perhaps this means the gears are turning 😁

[KBFlyer]

Great. I have heard nothing from them since my first contact. If they can get the Alt to show and battery voltage I would be happy with just that added. Having our batteries doing that critical voltage drop I figure that would be handle to have. Keep us posted if you hear anything more. Was wondering what you asked them to provide?

[yrless]

Here's the text of my message...I left it kind of vanilla thinking they might attack as much as they could at once. "Hi....just so you guys know, Autel Robotics has issued beta firmware 2.0.x and as a result the log files now contain completely different column headings as well as different data types from the previous fw release....just a heads up that there will be import/export issues ahead"

[KBFlyer]

Any response from them other then thanks for the file? LOL

[yrless]

Not yet but I'm sure that's because they're busy like elves working the issue, maybe even as I type :)

The person I spoke with over there is Yves Jusot just in case you're looking for a contact. yjusot@dronelogbook.com

[yrless]

Haven't heard back from anyone at dronelogbook yet and I tried reading a V2 bin file yesterday...same old issues so they haven't updated their stuff yet.....we wait.

[tomSny]

Now that the beta has been promoted as a release I started to get my head back into this. As you are aware the the structure and data of the log is much different from V1.x firmware.

My goal would be to make it easy for you to import into other visualization tools like dashware. I will need your help in determine the scope of data to add in the export, the export format(s) and transformation needed.

The release below will read the v2 logs but the exports that actually require some computation/transformations are no longer working (gpx,kmz) the raw dumps however (log, csv) are working and should help determine the needed transformations.

XStarLogViewer

[yrless]

Hi tomSny....thanks a bunch for putting this together.....I filtered on all the offered column headings and performed a csv export as well. From my quick look through the data I can identify a number of obvious items but there's a lot of cryptic stuff in there that they've abbreviated and/or obscured...also some output appears to be scaled by x1000, etc. Not sure where we go from here.....I submitted a ticket to AR CS to get clarity on the log file headings but I didn't even receive a ticket acknowledgement email which tells me they're holding this stuff close to the vest. Some notes: PVT: contains Lon, Lat, ht, gSpd (ground speed?), hMSL (position and altitude above sea level). STATES: contains hAGL (altitude above ground). GPS: contains Lat, Lon, Alt (altitude?), Time HMC5883: contains Mx, My, Mz....magnetometer outputs to compute heading??? MPU: contains Gx, Gy, Gz, ax, ay, az....gyro outputs to compute roll, pitch, yaw???

Now I know that HMC5883 refers to the magnetometer.....now the question is how to compute heading? https://moderndevice.com/product/hmc5883-3-axis-compass/

..and MS5611 refers to the barometer http://www.amsys.info/products/ms5611.htm

[tomSny]

Thank you yrless for sharing your findings.

I put out a new release today that attempts to calculate the roll, pitch, yaw. This took a little research and still not sure I did it properly on the first go. I went with a Madgwick's AHRS algorithm to compute the quaternion and then used the follow to compute the angles:

`public static Vector3 QuaternionEulerAngles(Quaternion q) { // Store the Euler angles in radians Vector3 pitchYawRoll = new Vector3(); float pi = (float)Math.PI;

        double sqw = q.W * q.W;
        double sqx = q.X * q.X;
        double sqy = q.Y * q.Y;
        double sqz = q.Z * q.Z;

        // If quaternion is normalised the unit is one, otherwise it is the correction factor
        double unit = sqx + sqy + sqz + sqw;
        double test = q.X * q.Y + q.Z * q.W;

        if (test > 0.4999f * unit)                              // 0.4999f OR 0.5f - EPSILON
        {
            // Singularity at north pole
            pitchYawRoll.Y = 2f * (float)Math.Atan2(q.X, q.W);  // Yaw
            pitchYawRoll.X = pi * 0.5f;                         // Pitch
            pitchYawRoll.Z = 0f;                                // Roll
            return pitchYawRoll;
        }
        else if (test < -0.4999f * unit)                        // -0.4999f OR -0.5f + EPSILON
        {
            // Singularity at south pole
            pitchYawRoll.Y = -2f * (float)Math.Atan2(q.X, q.W); // Yaw
            pitchYawRoll.X = -pi * 0.5f;                        // Pitch
            pitchYawRoll.Z = 0f;                                // Roll
            return pitchYawRoll;
        }
        else
        {
            pitchYawRoll.Y = (float)Math.Atan2(2f * q.Y * q.W - 2f * q.X * q.Z, sqx - sqy - sqz + sqw);       // Yaw
            pitchYawRoll.X = (float)Math.Asin(2f * test / unit);                                             // Pitch
            pitchYawRoll.Z = (float)Math.Atan2(2f * q.X * q.W - 2f * q.Y * q.Z, -sqx + sqy - sqz + sqw);      // Roll
        }

        return pitchYawRoll;
    }`

` I updated the kmz export to show as a FPV path (use google earth, and turn on 3D buildings for effect) to debug if it's calculating properly. Its seems to be heading in the right direction but had a lot of vibration so when viewing be sure to take you dramamine.

The beta version released detects if update is available and prompts to automatically update on launch, let me know otherwise. This will be needed since there will be a lot of changes to come.

I looked into the dashware import options, the good news it will work with any simplified csv file with the needed data and it will align. The not so bad news is the logs records have doubled since the V1 and will prob need a way to change the resolution to like (10 row/sec) so we don't have so many records.

[yrless]

Thanks tomSny.. great stuff 😁 I'm away from my desktop for a couple days but I'm itching to test your updated version early next week....just out of curiosity what log file output did you use for roll, pitch, yaw computation?

[tomSny]

I used the the MPU and the HMC5883 reading Acc, Gyro and Mag, and use this class. to get the the quaternion.

BTW, after further testing I don't think it's calculating heading properly, I need to pull some new logs I didn't have too many good samples in the new format.

[KBFlyer]

Thanks Tom for all you do. All you guys rock. I have learnt so much from all of you and hope we can get a working program out of all your efforts. I don't have much to offer in the figuring out part but if I can help just let me know. yrless knows where I hang out so if I can help just ask. Highly unlikely but ask anyways LOL