Are my dates and times borked?

[VancouverUmbrella]

Thanks for posting this script. Very neat.

It looks like I'm having an issue with the date and time columns. See attached. Is there something weird in my Otx log?

Best, Andrew (landru on RCG)

Phoenix_S-2022-05-01.mod.csv

[agibson2]

Your Time column data is in a format I haven't seen FrSky use before. It is using the windows format mm:SS.f where mm=Minutes 00-59, SS=Seconds 00-59, and f=tenths of a second. It is very odd data because it is missing hours, and isn't using thousandths of a second.

Tenths of a second is strange but not having hours makes no sense at all.

Your time column data... 02:25.3 mm:SS.f

Normal time column data... 17:47:45.480 HH:mm:SS.fff

HH is hours 00-59 fff is Thousandths of a second 000-999

What radio OS are you using? (your comment mentioned OpenTX) What version of OS? Did you use default GPS sensor settings? Are you using the FrSky GPS sensor? Are you using FrSky firmware for the GPS sensor?

[VancouverUmbrella]

Tx: QX7 ACCESS OS: Otx 2.3.15 GPS sensor: openXsensor

I haven't modified any GPS settings from the Otx defaults that I recall. Plus, my log files have always displayed properly in Google Earth via the Companion integration. However, I have continually run into problems with the date and time formats in my Otx logs when trying to load the data in other programs for plotting data.

In the past, I've solved that problem by opening the .csv in Excel and applying a custom format of hh:mm:ss.000 to the time column. I tried that with your script just now. The script reported a successful conversion afterwards as shown below. Sorry for not trying that earlier. I really should have.

Still, for some unknown reason, both ayvri.com and Google Earth cannot handle the resulting .gpx. Ayvri perpetually displays an error page that processing has not been completed. Google Earth shows the track in the middle of the Pacific Ocean. I don't rule out user error on my part in all this. As soon as possible, I'll try the process again with another log file to see if I can figure out what's going wrong. Sorry for the bother and thanks again.

[VancouverUmbrella]

Replied on RCG: https://www.rcgroups.com/forums/showthread.php?4140519-Script-to-convert-FrSky-logs-to-gpx-GPS-file-format#post49249249

[agibson2]

It looks like I messed something up in one of the lastest fixes. I will look into it.

[agibson2]

I was printing an exception when it should not have been. I fixed it in 1.0.6.

For your location showing up not where you expected, When GPS first starts to get location data, it can be off by quite a bit until it gets enough satellites to get a better location fix. If you look at the raw data for the time you are expecting and plug the longitude and latitude coordinates into google, it should show the same location.

To deal with that does take some manual manipulation of the gpx (or source csv file). I usually look at the raw GPS data and just remove any lines until the GPS coordinates look like they are closer to the same values.

One thing I have thought about is looking at all GPS data and then removing any data that is not within several miles of the majority of the data. I haven't put any effort into that though and still not sure if that is how I would want to tackle it or not. It currently processes one like at a time and doesn't check any history or future data when outputing a line to the gpx file.

[VancouverUmbrella]

Thanks for your persistence. 1.0.6 ran for me without any errors.

However, I'm still having trouble with the resulting file in both ayvri and Google Earth. Perhaps that's caused by stray data points as you suggested. I note that GE loads the log properly when called from Companion. Perhaps some cleanup is performed along the way? Also, I discovered that a kml file saved from GE and converted to gpx with Kml2gpx.com loads properly in ayvri. That's a cumbersome process compared to the elegance of your script.

For your amusement, here's the result: https://ayvri.com/scene/0jgrggmeko/cl3tl8sxb00022e6crzdjq0ij

I've also attached a 'clean' version of my original logfile from which the above was generated by the process described. By clean, I mean the file wasn't modified in Excel. The only change from the source was my editing a column header from "Alt2(m)" to "Alt(m)". I did that in the Mac text editor.

I'd be curious to know if you can succeed with my source file in ayvri or GE after conversion. However, that's well beyond the call of duty. Don't even think about it unless it you find the idea useful or entertaining. Meanwhile, I'll keep puttering to see if I can figure out what's going wrong. I'm happy to close this issue if it's outlived its usefulness.

Phoenix_S-2022-05-01.copy.csv

[agibson2]

Can you include the working gpx file converted by kml2gpx.com? Ill look and see if I can find the problem that way.

[VancouverUmbrella]

Definitely. Here it is. And thanks! 0315_l54bfq89l_Phoenix_S-2022-05-01.copy.gpx.zip

[agibson2]

Even the processes you went through manually didn't get the time data so those tools don't like your data either (maybe you knew that already).

All the time data is empty on that gpx file <time></time>

[VancouverUmbrella]

No, I didn't realize that.

It's beginning to seem that there's something very odd with the time data in my logs. What could it be, I wonder. I know Excel can add hidden characters to data. Can the same happen in a csv file?

[agibson2]

From my simple research, Latitude is supposed to be from -90 to 90. Your latitude seems to be -123.130621. Are you sure your GPS values are valid?

[agibson2]

Maybe when I do conversion for OpenTX that uses a single column for lat and long combined, I am assigning them wrong. I am looking at that.

[agibson2]

That was it. For the OpenTX files where latitude and longitude are combined into a single GPS column, I was assigning it wrong. It worked for Ethos formatted files with separate latitude and longitude.

Try version 1.0.7

[VancouverUmbrella]

Wonderful!

I'll have a chance to try 1.0.7 later today or perhaps tomorrow morning. Will let you know.

[VancouverUmbrella]

Success! Nice work.

Check it out: https://ayvri.com/scene/0jgrggmeko/cl3uyf1ra00012e6cxq76ma6o

There's still some hitch with the time of day, but that's small potatoes as far as I'm concerned. My main interest is in being able to upload log files to ayvri easily and watch the video and the stats. Your script certainly accomplishes that.

Many thanks, Andrew

[agibson2]

The time is converted to UTC time based on your current timezone on the computer that you run the script. All of the examples of .gpx files seemed to show that they were UTC (ends with Z).

[VancouverUmbrella]

Ok, I think things are becoming clearer for me. I'm on PDT (UTC-7). The first time recorded in my log was 13:03. Ayvri shows that as 20:03, which is obviously my local time plus 7 hours.

Is it odd that ayvri labels that as "LOCAL TIME"? Seems more like UTC time. Or have I got my head twisted around? GE seems to get the local time right for the same gpx file. Again, this is mostly curiosity for me.

[agibson2]

Yea. It seems to be that they just show whatever time is in the file regardless of the Z (UTC) specification on the time data. I sent a question to them about it because I would like it displayed in local time also if it says local time when others view it.

If they don't plan to fix it, I might include another option to just put local time (like what is recorded in the FrSky logfile data).

[VancouverUmbrella]

Thanks for confirming. Here's hoping you get a positive response.

I was a paying Strava subscriber for a number of years. In that time, I reported a number of problems and suggestions for site improvements. I found the support staff pleasant and responsive but unable to effect any significant change in the platform. Hopefully, Ayvri is different.

[VancouverUmbrella]

v1.0.8 works perfectly. It even handles the peculiarities of ayvri.com. Thank you!