JaapvanEkris / openrowingmonitor

A free and open source performance monitor for rowing machines
GNU General Public License v3.0
26 stars 7 forks source link

Additional data at the end of TCX files #59

Open chrzr opened 14 hours ago

chrzr commented 14 hours ago

Hi!

First of all thanks to everyone involved in creating this great piece of open source software 👏 I got my first waterrower, a Joroto MR380, a few weeks ago and was able to retrofit it with ORM in just a few hours.

I also wanted to automate the upload of my acitivities to Garmin Connect, so I created a service which uses inotifywait to trigger the conversion of the .tcx to a .fit file and also the upload to Garmin Connect.

The problem I am currently facing is that sometimes there are additional entries at the end of the tcx file created by ORM.

         <Trackpoint>
            <Time>2024-10-27T10:19:33.967Z</Time>
            <DistanceMeters>8442.17</DistanceMeters>
            <Cadence>25</Cadence>
            <Extensions>
              <ns2:TPX>
                <ns2:Speed>4.72</ns2:Speed>
                <ns2:Watts>294</ns2:Watts>
              </ns2:TPX>
            </Extensions>
            <HeartRateBpm>
              <Value>115</Value>
            </HeartRateBpm>
          </Trackpoint>
        </Track>
        <Extensions>
          <ns2:LX>
            <ns2:Steps>743</ns2:Steps>
            <ns2:AvgSpeed>4.69</ns2:AvgSpeed>
            <ns2:AvgWatts>291</ns2:AvgWatts>
            <ns2:MaxWatts>424</ns2:MaxWatts>
          </ns2:LX>
        </Extensions>
      </Lap>
      <Notes>Indoor Rowing, Drag factor: 32000.0 10-6 N*m*s2, Estimated VO2Max: UNDEFINED</Notes>
    </Activity>
  </Activities>
  <Author xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Application_t">
    <Name>Open Rowing Monitor</Name>
    <Build>
      <Version>
        <VersionMajor>0</VersionMajor>
        <VersionMinor>9</VersionMinor>
        <BuildMajor>0</BuildMajor>
        <BuildMinor>0</BuildMinor>
      </Version>
      <LangID>en</LangID>
      <PartNumber>OPE-NROWI-NG</PartNumber>
    </Build>
  </Author>
</TrainingCenterDatabase>/Activities>
  <Author xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Application_t">
    <Name>Open Rowing Monitor</Name>
    <Build>
      <Version>
        <VersionMajor>0</VersionMajor>
        <VersionMinor>9</VersionMinor>
        <BuildMajor>0</BuildMajor>
        <BuildMinor>0</BuildMinor>
      </Version>
      <LangID>en</LangID>
      <PartNumber>OPE-NROWI-NG</PartNumber>
    </Build>
  </Author>
</TrainingCenterDatabase>

Oct 27 10:27:43 orm ORMWatchdog.sh[2156]: /opt/openrowingmonitor/data/recordings/2024/10/2024-10-27_09-49-31_rowing1.tcx Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: Traceback (most recent call last): Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "/home/pi/Coxswain2Fit/CoxswainToFit.py", line 61, in Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: main(args.input) Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "/home/pi/Coxswain2Fit/CoxswainToFit.py", line 12, in main Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: rootxml, Amountlaps = TCXextractor.lap_amount(file) Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "/home/pi/Coxswain2Fit/TCXextractor.py", line 156, in lap_amount Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: tree = etree.parse(tcx, parser) Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/etree.pyx", line 3589, in lxml.etree.parse Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 1958, in lxml.etree._parseDocument Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 1984, in lxml.etree._parseDocumentFromURL Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 1887, in lxml.etree._parseDocFromFile Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 1200, in lxml.etree._BaseParser._parseDocFromFile Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: File "/opt/openrowingmonitor/data/recordings/2024/10/2024-10-27_09-49-31_rowing1.tcx", line 10485 Oct 27 10:27:44 orm ORMWatchdog.sh[5950]: lxml.etree.XMLSyntaxError: Extra content at the end of the document, line 10485, column 26 Oct 27 10:27:55 orm ORMWatchdog.sh[2156]: rowing-session.fit Oct 27 10:28:01 orm ORMWatchdog.sh[5991]: Trying to login to Garmin Connect using token data from directory '~/.garminconnect'... Oct 27 10:28:01 orm ORMWatchdog.sh[5991]: File to upload not found: rowing-session.fit

If I manually remove the additional entries the conversion to .fit and also the upload works without any problems.

Oct 27 10:28:11 orm kernel: [51730.023566] hwmon hwmon1: Undervoltage detected! Oct 27 10:28:17 orm kernel: [51736.069837] hwmon hwmon1: Voltage normalised Oct 27 10:28:39 orm ORMWatchdog.sh[2156]: /opt/openrowingmonitor/data/recordings/2024/10/2024-10-27_09-49-31_rowing2.tcx Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> main header created Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> file_id message + data created Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> write data for session Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> write data for activity Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: <<< file is : 30894 bytes Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> file size placed into the header with removed 14 bytes of header from the size file Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: <<< checksum calculated: fb4f Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> checksum placed at the end of the file Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: >>> file exported to rowing-session.fit Oct 27 10:28:42 orm ORMWatchdog.sh[6050]: finished Oct 27 10:28:52 orm ORMWatchdog.sh[2156]: rowing-session.fit Oct 27 10:28:55 orm ORMWatchdog.sh[6094]: Trying to login to Garmin Connect using token data from directory '~/.garminconnect'... Oct 27 10:28:55 orm ORMWatchdog.sh[6094]: File uploaded successfully!

I already tried looking for and adapting the corresponding code, which I found in WorkoutRecorder.js, but I just could not fix the issue. In the logging of openrowingmonitor service I see that there are multiple lines regarding the creation of tcx files which I suspect might be the problem here?

Oct 27 10:20:27 orm npm[5637]: saving session as RowingData file data/recordings/2024/10/2024-10-27_09-49-31_rowingData.csv... Oct 27 10:20:27 orm npm[5637]: saving session as tcx file data/recordings/2024/10/2024-10-27_09-49-31_rowing.tcx... Oct 27 10:20:27 orm npm[5637]: saving session as tcx file data/recordings/2024/10/2024-10-27_09-49-31_rowing.tcx... Oct 27 10:20:27 orm npm[5637]: saving session as tcx file data/recordings/2024/10/2024-10-27_09-49-31_rowing.tcx... Oct 27 10:20:37 orm ORMWatchdog.sh[2156]: /opt/openrowingmonitor/data/recordings/2024/10/2024-10-27_09-49-31_rowing.tcx

Could it be that the corresponding function is called multiple times in parallel causing the additional entries in the tcx?

Any ideas would be appreciated!

Kind Regards, Christoph

JaapvanEkris commented 12 hours ago

It indeed generates additional data: the recovery heart rate is measured every minute (max 3 times) and added to the notes field for sessions with a HR above 70% MaxHR. I use a simple bash polling for new files in a folder that are older than 3 minutes.

Why convert to a fit file? Garmin accepts our tcx files direcly (used that for years).

chrzr commented 10 hours ago

Thanks for the quick reply! Ok so this is intended behaviour - but what still confuses me a bit and what also seems to cause the problems with conversion is the following duplicate data at the end of the tcx file (what i meant by additional, probably phrased wrong)

  </Activities>
  <Author xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Application_t">
    <Name>Open Rowing Monitor</Name>
    <Build>
      <Version>
        <VersionMajor>0</VersionMajor>
        <VersionMinor>9</VersionMinor>
        <BuildMajor>0</BuildMajor>
        <BuildMinor>0</BuildMinor>
      </Version>
      <LangID>en</LangID>
      <PartNumber>OPE-NROWI-NG</PartNumber>
    </Build>
  </Author>
</TrainingCenterDatabase>/Activities>
  <Author xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Application_t">
    <Name>Open Rowing Monitor</Name>
    <Build>
      <Version>
        <VersionMajor>0</VersionMajor>
        <VersionMinor>9</VersionMinor>
        <BuildMajor>0</BuildMajor>
        <BuildMinor>0</BuildMinor>
      </Version>
      <LangID>en</LangID>
      <PartNumber>OPE-NROWI-NG</PartNumber>
    </Build>
  </Author>
</TrainingCenterDatabase>

I tried commenting out the code where the Author data is generated, but then I just get the data above as duplicate, which is why I thought that it must have something to do with parallel execution of the function.

Using the tcx directly works - but I also sometimes had to manually remove the last part of the tcx for Garmin Connect to import the file 🤔 I also like that it converts the data to be more rowing specific (strokes, strokes rate, pace etc.)

JaapvanEkris commented 8 hours ago

Thanks for the quick reply! Ok so this is intended behaviour - but what still confuses me a bit and what also seems to cause the problems with conversion is the following duplicate data at the end of the tcx file (what i meant by additional, probably phrased wrong)

This is just plain weird behaviour. In my current setup (which uses Architecture_Revision, but the core tcx writer is the same) I don't see this behaviour.

I tried commenting out the code where the Author data is generated, but then I just get the data above as duplicate, which is why I thought that it must have something to do with parallel execution of the function.

There is a minute in between. That should be plenty of time to recalculate and write any data. Might be that you keep grabbing the file as soon as it is written. Perhaps give it a bit more time to write?

Using the tcx directly works - but I also sometimes had to manually remove the last part of the tcx for Garmin Connect to import the file 🤔 I also like that it converts the data to be more rowing specific (strokes, strokes rate, pace etc.)

Changing the type of session to "Indoor Rowing" always did that trick for me. Another (easier) approach is to see if your Garmin watch supports the indoor rowing ANT+ profile and add a simple ANT+ dongle to ORM. That actually adds data as not only strokerate gets picked up, but on higher end watches also the distance per stroke and breathing.

chrzr commented 7 hours ago

Ok so the tcx file is only generated after pressing the reset/end session button? My script indeed only waits 10 seconds after the file is created before triggering the conversion, so I'll up that to 120 seconds just to be safe.

What is weird is that I have older tcx files where I didn't have any watchdog script running and there I see the same duplicate data - maybe I'll try and setup a second SD card with a fresh install to see if it shows the same behaviour 🤔

I actually have a Garmin Instinct 2X and I already have an ANT+ dongle connected for HR monitoring - I'll have to check if my watch supports it. That would indeed make things a bit easier :)

JaapvanEkris commented 7 hours ago

Ok so the tcx file is only generated after pressing the reset/end session button?

It is also generated on a stop or pause.

My script indeed only waits 10 seconds after the file is created before triggering the conversion, so I'll up that to 120 seconds just to be safe.

Yeah, although your next paragraph suggests it isn't due to a too quick pickup.

What is weird is that I have older tcx files where I didn't have any watchdog script running and there I see the same duplicate data - maybe I'll try and setup a second SD card with a fresh install to see if it shows the same behaviour 🤔

In the new architecture it is nicely separated. I am considering a complete rewrite of the tcx recorder as its structure isn't easy to read/understand. I am still thinking about adding a native fit-file writer as it can contain more data than tcx, but it isn't the easiest format to create.

I actually have a Garmin Instinct 2X and I already have an ANT+ dongle connected for HR monitoring - I'll have to check if my watch supports it. That would indeed make things a bit easier :)

That is way cooler if it works. Switched to that approach when @Abasz made it, never looked back.