search

openambitproject / openambit

openambit
277 stars 82 forks source link

Export to GPX #181

Open BryceStevenWilley opened 6 years ago

BryceStevenWilley commented 6 years ago

Given the recent downtime of Movescount, it might be a good idea to implement an offline exporter to the GPX format, or something that operates better with 3rd-party apps. Adding an option right below "Write movescount file" would be a good place I feel.

This is something I might look into, but not for a month or two.

onedayfishsale commented 6 years ago

The tools/openambit2gpx.py script works pretty well for me. Not integrated with the GUI but easy enough as a stop-gap.

BryceStevenWilley commented 6 years ago

Oh! I didn't see that script. Definitely fills my use-case. There's still value in GUI integration I think, so I'll keep this issue open.

marguslt commented 6 years ago

One issue with tools/openambit2gpx.py is relying only on HR samples in a log file.

Thanks to Movescount being down I happened to use a fresh build of Openambit for a first time after moving from Ambit to Ambit3. Was bit surprised to find out that Ambit3 logs are actually without HR samples, only including IBI data chunks . So for Ambit3 the current tools/openambit2gpx.py can only output a GPX without heart rate data.

onedayfishsale commented 6 years ago

@marguslt I'm not seeing that with my Ambit3 Sport. I have HR samples in both the Openambit log and in the GPX from openambit2gpx.py.

            <Sample>
                <Type id="512">periodic</Type>
                <UTC>2018-02-04T19:58:04.684Z</UTC>
                <Time>914684</Time>
                <VerticalSpeed>1</VerticalSpeed>
                <Cadence>88</Cadence>
                <HR>156</HR>
                <Altitude>228</Altitude>
                <Distance>2850</Distance>
                <Speed>298</Speed>
                <Time>914684</Time>
            </Sample>
   <trkpt lat="47.1560608" lon="-88.6910717"><ele>228</ele><time>2018-02-04T19:58:04.284Z</time><extensions><gpxdata:hr>156</gpxdata:hr><gpxdata:cadence>96</gpxdata:cadence><gpxdata:speed>297</gpxdata:speed></extensions></trkpt>

I do notice that the gps-small and periodic samples in the Openambit log seem to get combined to create a GPX trkpt.

marguslt commented 6 years ago

@onedayfishsale mh.. interesting. Can you check your watch SW version, either from logs or by holding down Back/Lap and Start/Stop buttons? Mine is Ambit3 Peak with 2.4.17, it was released in June 2017, but basically the same issue in ambit2gpx was brought up allready in 2015 - montant/ambit2gpx#10 (2 last comments through google translate)

onedayfishsale commented 6 years ago

@marguslt Ambit3 Sport 2.4.17. Here's the full Openambit log and the GPX created from openambit2gpx.py.

log GPX

I'm not sure precisely what version of Openambit, it was from back when I was helping find HR data in the Ambit3 packets. I can try a fresh pull when I'm back home with the watch and cable.

onedayfishsale commented 6 years ago

Just pulled and built a fresh copy of Openambit from today, and can confirm HR data in both the Openambit log and the converted GPX.

pygoubet commented 6 years ago

Ambit3 Run 2.4.17. For the same reasons as @BryceStevenWilley, I would have liked to export my moves as GPX and so that, used tools/openambit2gpx.py script. The conversion works fine but there are no HR data. The problem seems to come from log file which contains no _<HR>value</HR>_ tags.

onedayfishsale commented 6 years ago

@pygoubet, @marguslt are you seeing HR data on Movescount when you upload via Moveslink2?

pygoubet commented 6 years ago

Yes, on Movescount website, HR data/graph are present. My log file has no _<HR>value</HR>_ in _<Sample>_ tag but has in the _<Log>_ section:

    <Log>
        <Header>
...
            <HR>
                <Avg>127</Avg>
                <Max>145</Max>
                <Min>94</Min>
                <MaxTime>3050117</MaxTime>
                <MinTime>1617</MinTime>
            </HR>
onedayfishsale commented 6 years ago

Hmm, I'm not sure what might be going on. I wouldn't think the Peak or Run versions would be so different that HR data is in a different location but I suppose it's possible. Anyone with a Ambit3 Sport 2.4.17 having HR issues?

marguslt commented 6 years ago

Yup, no issues with HR in Movescount with my Peak, though I'm mostly using Andoid app for syncing. And <HR> samples are not missing in just one Openambit log, I did a full sync, 20+ logs and different activity types (running, xc skiing, indoor sessions without gps & with HR), all are without <HR> samples. But as I said, IBI (interbeat interval) data does exist in Openambit logs. And at least Andoid app does not send sepparate HR samples to Movescount, just IBI and Movescount builds HR from that.

I need to plan ahead a bit to see what's going on with some recent version Moveslink2 and what will be stored in Moveslink SML logs. At least I now know it's not just my watch and my build :)

@onedayfishsale , by any chance, do you use some otther HR sensor insetad of Suunto SmartSensor - http://www.suunto.com/Products/Heart-Rate-Belts/SuuntoSmartSensor/ ?

onedayfishsale commented 6 years ago

@marguslt Mio Link, but I'll try the Suunto sensor tonight if I can find it.

marguslt commented 6 years ago

Somewhat related - just noticed that latest QMapShack release, 1.10.0, supports Openambit log and Moveslink2 SML files, can export to GPX & TCX. https://bitbucket.org/maproom/qmapshack/wiki/Home

vbelloir commented 6 years ago

Hi all, I'm quite new to openambit. I'm trying to use openambit2gpx to export gpx files, but I can't find where openambit generates log files.

Where are theses files?

I'm using openambit 0.3 from ubuntu 18.04 repo.

Vincent

paddy-hack commented 6 years ago

They are in ~/.openambit/, as documented at the top of tools/openambit2gpx.py.

vbelloir commented 6 years ago

Yes, this is what I saw yesterday. Seing to the directory didn't exist, I create it, but with a typo :(

Now that's ok, thanks for help.

pkel commented 5 years ago

I just ran tools/openambit2gpx.py on my log files. It failed on two of them. I provide them for debugging.

one.log

Traceback (most recent call last):
  File "tools/openambit2gpx.py", line 217, in <module>
    lat1=float(lapArray[i][5])

TypeError: float() argument must be a string or a number

two.log

Traceback (most recent call last):
  File "tools/openambit2gpx.py", line 219, in <module>
    latInterPolEP=str( ((lat2-lat1)/timeDiff(t1,t2))*timeDiff(t1,t) + lat1 )
  File "tools/openambit2gpx.py", line 74, in timeDiff
    secs1=utcSplitConvSeconds(utcTime1)
  File "tools/openambit2gpx.py", line 64, in utcSplitConvSeconds
    tmpTime=utcTime.split("T")[1].split("Z")[0].split(":")
IndexError: list index out of range

These come from an Ambit 3 Vertical with FW version 1.1.22.

markusdecker commented 5 years ago

hi going back to the HR conversion with ambit 3 peak. the data are stored after a (n) sample interval in the log file, in the Interbeat interval (IBI) format and they look like this:


<Sample>
                <Type id="774">ibi</Type>
                <UTC>2018-10-07T10:16:56.676Z</UTC>
                <Time>413676</Time>
                <IBI>568</IBI>
                <IBI>574</IBI>
                <IBI>577</IBI>
                <IBI>565</IBI>
                <IBI>569</IBI>
                <IBI>566</IBI>
                <IBI>569</IBI>
                <IBI>564</IBI>
                <IBI>570</IBI>
                <IBI>564</IBI>
                <IBI>556</IBI>
                <IBI>567</IBI>
                <IBI>565</IBI>
                <IBI>564</IBI>
                <IBI>566</IBI>
                <IBI>575</IBI>
                <IBI>563</IBI>
                <IBI>571</IBI>
                <IBI>564</IBI>
                <IBI>568</IBI>
                <IBI>570</IBI>
                <IBI>556</IBI>
                <IBI>563</IBI>
                <IBI>560</IBI>
                <IBI>563</IBI>
                <IBI>559</IBI>
                <IBI>560</IBI>
                <IBI>558</IBI>
                <IBI>559</IBI>
                <IBI>571</IBI>
                <IBI>573</IBI>
                <IBI>579</IBI>
            </Sample>
markusdecker commented 5 years ago

hi i modified the script to read the ibi records and convert it to hr, if someone is interessted i ll c&p,

m

vascotenner commented 5 years ago

@markusdecker: I am interested in you script. Can you C&P it here or add it to a gist or even create a pull request?

markusdecker commented 5 years ago

@vascotenner ,

i wrote this function:


def ibiToHr():

    global element, ibitime, ibitimeLast, hrlist, hr
    SampType=element.findtext("Type")
    if sampType=='ibi':
        ibitime=element.findtext("Time")
        if ibitimeLast!=None and ibitimeLast!=ibitime:
         del hrlist
         hrlist=[element.text for element in element.findall('IBI')]

# filter 1: average hr data, flatten data
         hrlist=map(int, hrlist)
         tmpav=sum(hrlist)/len(hrlist)
         for i in hrlist:
             if abs(i-tmpav)>20:
                 hrlist = [tmpav if x==i else x for x in hrlist]
        ibitimeLast=ibitime

#filter 2: sensor errors
    if len(hrlist)>0:

        hr=str(int(60/(float(hrlist[0])/1000)))
        if int(hr)>190 or int(hr)<50: hr=hrLast
        del hrlist[0]

    if hr==None: hr=hrLast
    return hr

and replaced the hr=element.findtext("HR")... searchline with this:

hr=ibiToHr()

however, the heartrate data are good shaped and correct. i compared it to suuntos gpx exports, but theres a tiny shift on the timeaxis, caused by the first gps data samples in the log_file until the first ibi data are recorded. right now, i dont have a solution for this.

ciao m

vascotenner commented 5 years ago

I have implemented a slighty difference version in PR #211 (https://github.com/openambitproject/openambit/pull/211/commits/2ce29d8117c530d7c93553350a205d9b0afa9cc0)

@markusdecker : I had some problems getting your code working, and I have questions about it. What is the function of these lines: https://github.com/openambitproject/openambit/pull/211/commits/2ce29d8117c530d7c93553350a205d9b0afa9cc0#diff-ff33b6b061b8ffacad92f1d0b857efb8R101

Can someone explain/document what this list of IBI values mean? Why is there a whole list?

markusdecker commented 5 years ago

hi vasco this lines are a filter: descibed by: filter 1: average hr data, flatten data filter out heartrate peak to smoothen the graphical curves when viewed in a gpx program. you can simply ignore that lines.

i ll test all you code soon. super.

ciao m

vascotenner commented 5 years ago

In this post I try to understand from which moment the IBI data is originating, and how to convert it to a gpx file.

Sample data:

    <Sample>
                <Type id="774">ibi</Type>
                <UTC>2018-12-25T08:54:37.570Z</UTC>
                <Time>36570</Time>
                <IBI>540</IBI>
               .....
    </Sample>

Observations:

For me it is not clear from which moment the IBI data originates. There is no timestamp associated to each IBI innode. However, I could imagine that this is all historic data, hence it is captured before the UTC date reported in the sample section. In that case a UTC time could be associated by to any IBI: IBIs_time[i] = UTCsample - sum(IBIs[i:])

In the current implementation, the reported HR always lags behind the GPS points, by a maximum of 32 * 1s = 32s (assume hr of 60 bpm).

When converting to GPX, only datapoints are saved when a GPS datapoint is available. This means that much sample data will be lost, e.g. when only every 60s a gps point is measured while the hr/cadance/etc senors reports every 2 s, 14 datapoints of the sensor (28s) are discarted, only the sensordata just before that moment is saved to GPX.

Now two questions remain:

  1. Is discarting this data the best idea? Or better take an average since the last GPS point? This will greatly reduce measurement noise.
  2. How to cope with the IBI data lag?

I propose to solve both 1 and 2 in one go: first create time-stamped lists of all sensor data, and then report the average before the time of GPS point. I could make some time to implement these ideas.

marguslt commented 5 years ago

I currently don't have access to any openambit instance nor log files, but maybe this helps -

IBI values (and corresponding HR) can be quite noisy , especially with bluetooth HR belts (Ambit3 + Suunto Smart Sensor), Smart Sensor seems to pick up quite a lot static noise from clothing and struggles a lot when belt contacts are dry and / or dirty. From what I've seen, Movescount seems to have no sanity check nor filtering built in, but from users perspective some filtering might actually be reasonable. Few filtering examples: https://github.com/Aura-healthcare/hrvanalysis/blob/master/hrvanalysis/preprocessing.py https://github.com/milegroup/ghrv/blob/master/DataModel.py (FilterNIHR)

Anyway, when I last looked into it, I couldn't come up with a solid idea how to interpolate IBI series over time samples that I could implement with my crude copy-paste python skills :P Right now I don't think small offset actually matters too much, HR is lagging anyway and we would still get valid HR zone distribution and reasonable recovery indication between intervals, even if there's something like a 1s .. 5s offset between HR and actual effort.