search

ncssar / sartopo_python

Python calls for the caltopo / sartopo API
GNU General Public License v3.0
14 stars 2 forks source link

geom ops: excessive 'spur removed' reports #44

Closed caver456 closed 2 years ago

caver456 commented 2 years ago

Seems strange and not optimal:

02:05:48 [sartopo_python:1926:INFO] c2
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9126041355144, 39.310118329944245]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9126041355144, 39.310118329944245]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91263752065922, 39.31010004898821]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9125344970488, 39.31004803526187]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91259932056552, 39.310048895948334]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91258761258968, 39.3100522232373]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91258761258968, 39.3100522232373]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91257530187691, 39.31008052472255]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91259754835431, 39.31004019006705]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91259754835431, 39.31004019006705]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9125995843535, 39.310036188551095]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91268413640815, 39.30989426087681]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91268413640815, 39.30989426087681]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91269079872413, 39.30996047096191]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91269079872413, 39.30996047096191]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91269079872413, 39.30996047096191]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91254825992463, 39.309896225146176]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275401351751, 39.30966451738259]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275401351751, 39.30966451738259]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275401351751, 39.30966451738259]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275401351751, 39.30966451738259]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91263961088234, 39.30964028329966]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91263961088234, 39.30964028329966]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9126281792258, 39.30961602705325]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91257313533602, 39.309510912437425]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91257313533602, 39.309510912437425]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91265858381428, 39.309422132684944]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91252077573373, 39.30923270096881]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91252077573373, 39.30923270096881]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275581247423, 39.30903640494049]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275581247423, 39.30903640494049]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275581247423, 39.30903640494049]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91275581247423, 39.30903640494049]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9129765225106, 39.308919263050484]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91664434523977, 39.309040843829905]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91664434523977, 39.309040843829905]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91706137770058, 39.30896241395171]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91727546974599, 39.30880115614261]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91767887321512, 39.308742971492364]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91771641270711, 39.30874476047834]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91760815708591, 39.30876946388727]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9176309632685, 39.30882456684909]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91773482268562, 39.30877624442109]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91773482268562, 39.30877624442109]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91773730667346, 39.3087597996725]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91789626481506, 39.30924211742154]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91789626481506, 39.30924211742154]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91789626481506, 39.30924211742154]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183843652976, 39.3097844276888]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91844810936844, 39.30982953819222]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183707747579, 39.30983566418877]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91828814458634, 39.310080893996194]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91828814458634, 39.310080893996194]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9186138888842, 39.31055512849298]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9186138888842, 39.31055512849298]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91832196666999, 39.3112750908487]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183612353993, 39.31124668524785]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183612353993, 39.31124668524785]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.9183612353993, 39.31124668524785]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91808458057946, 39.31115454773156]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91515866699034, 39.31225845004681]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91515866699034, 39.31225845004681]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91409908830832, 39.31169644395109]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91434956379747, 39.31138323254108]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91434956379747, 39.31138323254108]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91445660493173, 39.311431677464064]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91438966700335, 39.31134311866739]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91433757690525, 39.3113346794685]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91418345094378, 39.31140139051778]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.918421924866, 39.311148638482855]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.918421924866, 39.311148638482855]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91947172012924, 39.310650154488]
02:05:48 [sartopo_python:1469:INFO] spur removed at [-120.91947172012924, 39.310650154488]
02:05:48 [sartopo_python:1945:INFO] c3
        logging.info('c2')
        tg=targetShape['geometry']
        tgc_orig=None
        targetType=tg['type']
        if targetType=='Polygon':
            tgc=tg['coordinates'][0]
            tgc=self.removeSpurs(tgc)
            targetGeom=Polygon(tgc) # Shapely object
        elif targetType=='LineString':
            tgc_orig=tg['coordinates']
            tgc=self.twoify(tgc_orig)
            # logging.info('tgc before ('+str(len(tgc))+' points):'+str(tgc))
            tgc=self.removeSpurs(tgc)
            # logging.info('tgc after ('+str(len(tgc))+' points):'+str(tgc))
            targetGeom=LineString(tgc)
        else:
            logging.warning('crop: target feature '+targetStr+' is not a polygon or line: '+targetType)
            return False

        logging.info('c3')

need to make a minimal test case and pair it down.

caver456 commented 2 years ago

This implies that removeSpurs actually removed everything:

02:26:02 [sartopo_python:1926:INFO] c2
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:26:02 [sartopo_python:679:ERROR] Exception during sync of map AKC; stopping sync: IndexError:('pop from empty list',)
caver456 commented 2 years ago

That was recoverable during sync (though it's not clear if sync actually restarted after pressing the Play button a few times) but during initial read on restart it was not recoverable - it showed the uncaught exception dialog then the program exited:

02:29:33 [sartopo_bg:1510:INFO]    t1: cropping cfbf3f4b-7af3-4163-a694-7478e2af6fee
02:29:33 [sartopo_bg:1516:INFO]    t2
02:29:33 [sartopo_python:1910:INFO] c1
02:29:33 [sartopo_python:621:INFO] refresh requested for map GKH: 1392ms since last completed sync; shorter than syncInterval; forceImmediate not specified: not syncing now
02:29:33 [sartopo_python:1926:INFO] c2
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91045418017751, 39.309811765302285]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [sartopo_python:1469:INFO] spur removed at [-120.91048799674455, 39.30977389835933]
02:29:33 [plans_console:280:CRITICAL] Uncaught exception
Traceback (most recent call last):
  File "C:\Users\caver\Documents\GitHub\plans_console\plans_console.py", line 1286, in <module>
    main()
  File "C:\Users\caver\Documents\GitHub\plans_console\plans_console.py", line 1281, in main
    w = PlansConsole(app)
  File "C:\Users\caver\Documents\GitHub\plans_console\plans_console.py", line 510, in __init__
    self.debriefButtonClicked()
  File "C:\Users\caver\Documents\GitHub\plans_console\plans_console.py", line 792, in debriefButtonClicked
    self.dmg=DebriefMapGenerator(self,self.sts,self.debriefURL)
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_bg.py", line 468, in __init__
    self.newFeatureCallback(f)
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_bg.py", line 1646, in newFeatureCallback
    self.addOuting(f)
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_bg.py", line 1330, in addOuting
    self.cropUncroppedTracks()
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_bg.py", line 1517, in cropUncroppedTracks
    croppedTrackLines=self.sts2.crop(utid,bid,beyond=cropDegrees)
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_python.py", line 1938, in crop
    tgc=self.removeSpurs(tgc)
  File "C:\Users\caver\Documents\GitHub\plans_console\sartopo_python.py", line 1470, in removeSpurs
    out.pop() # delete last vertex
IndexError: pop from empty list
caver456 commented 2 years ago

There actually are runs of several points with the exact same coordinates in the source data:

                    [
                        -120.91046407708849,
                        39.30979678292522,
                        1219,
                        1636773455156
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773458157
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773461165
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773464154
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773467164
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773470162
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773473165
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285,
                        1219,
                        1636773476153
                    ],
                    [
                        -120.91048419316307,
                        39.30978941464972,
                        1219,
                        1636773479165
                    ],

Notice these reports are three seconds apart; this was about 18 seconds of absolute same coordinates. That's an issue with the upstream locator code (not sure if this was apptrack or Garmin) but it looks like removeSpurs is reporting them correctly.

But, it's not clear if or how these strings of repeated coordinates lead to the 'pop from empty list' error. The pop was intended for a real 'spur' as on a-b-c-b-d - so the out list modification should go like:

But that sequence doesn't hold up when you have a long string of repeated coordinates. So, some logic should be added before the spur check, that simply doesn't add a vertex if it's the same as the previous one. (Note that four-element vertices will not be entirely identical due to the timestamp differences - we only want to compare lat and lon.)

The numbers above are for line AI103f from 'training.json' (localhost AKC).

caver456 commented 2 years ago

Ok it worked - here's the json exported from the debrief map for that same line - notice the vertices before and after are there, but the duplicates are removed:

                    [
                        -120.91046407708849,
                        39.30979678292522
                    ],
                    [
                        -120.91045418017751,
                        39.309811765302285
                    ],
                    [
                        -120.91048419316307,
                        39.30978941464972
                    ],

So the duplicates are removed before spur-checking, so the offending pop is not called for duplicates. The crash was probably because the long chain of duplicates was encountered close to the start of the list.

Note, this does not use the removeDuplicatePoints function which isn't called by anything internally right now; it might be good to use that if the goal is to reduce output data size, but for strictly identical points, it's easier to skip those points during removeSpurs.

Also, note that preserving the four-element vertices is a different issue.