Pandas deprecation of .ix

[ng9891]

Hello!

I am having an error when I run with _--write_geom --writescore for PG in the command line. The algorithm runs fine otherwise (Haven't ran it with the test data. Sorry). I am running it on a WSL console.

Error after printing "Matched edges for 5: "[47646, 50417, 706645,...]"

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "pgMapMatch/main.py", line 103, in mm.writeMatchToPostgres(writeEdgeIds=args['--write_edges'], writeGeom=args['--write_geom'], writeMatchScore=args['--write_score']) File "pgMapMatch/mapmatcher.py", line 429, in writeMatchToPostgres if writeGeom: self.writeGeomToPostgres() # needed for writeMatchScore, so do this first File "pgMapMatch/mapmatcher.py", line 457, in writeGeomToPostgres self.getMatchedLineString() File "pgMapMatch/mapmatcher.py", line 494, in getMatchedLineString prevNode = self.edgesDf.source[route[0]] if self.edgesDf.target[route[0]] in self.edgesDf.ix[route[1]][['source', 'target']].tolist() else self.edgesDf.target[route[0]] File "/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py", line 5274, in getattr return object.getattribute(self, name) AttributeError: 'DataFrame' object has no attribute 'ix'

Command used to run: python3 pgMapMatch pg veh107_line id geom_m --streets=at_2po_4pgr --ids=5 --write_edges --write_geom --write_score

[amillb]

This looks like a pandas issue. Are you running v 1.0+? It seems that the .ix function has been removed.

I will try and fix this issue soon. In the meantime, my suggested workaround is to downgrade to pandas v0.25.

[ng9891]

Yes. I am running 1.0.1.

Thanks for the help!

[amillb]

Reopening until I fix this issue.

[ng9891]

Sorry, not sure if I should open another issue. I downgraded pandas to 0.19.2. After running the command line again I get this error now.

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "pgMapMatch/main.py", line 103, in mm.writeMatchToPostgres(writeEdgeIds=args['--write_edges'], writeGeom=args['--write_geom'], writeMatchScore=args['--write_score']) File "pgMapMatch/mapmatcher.py", line 429, in writeMatchToPostgres if writeGeom: self.writeGeomToPostgres() # needed for writeMatchScore, so do this first File "pgMapMatch/mapmatcher.py", line 462, in writeGeomToPostgres self.db.execute(cmd) File "pgMapMatch/tools.py", line 122, in execute return self.cursor.execute(cmd) psycopg2.errors.AmbiguousColumn: column reference "geom" is ambiguous LINE 4: ... SELECT 1 AS lorder, ST_Reverse(ST_LineSubString(geom, 0, S...

[amillb]

This was indeed a separate problem regarding name conflicts.

Both the pandas issue and the name column should now be fixed in 4d073a6. If you can let me know if it works, I'd appreciate it.

[ng9891]

Thanks for the reply. Sorry for the delay. I ran the program again but this time the same issue with another column:

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "pgMapMatch/main.py", line 103, in mm.writeMatchToPostgres(writeEdgeIds=args['--write_edges'], writeGeom=args['--write_geom'], writeMatchScore=args['--write_score']) File "pgMapMatch/mapmatcher.py", line 429, in writeMatchToPostgres if writeGeom: self.writeGeomToPostgres() # needed for writeMatchScore, so do this first File "pgMapMatch/mapmatcher.py", line 462, in writeGeomToPostgres self.db.execute(cmd) File "pgMapMatch/tools.py", line 122, in execute return self.cursor.execute(cmd) psycopg2.errors.AmbiguousColumn: column reference "id" is ambiguous LINE 4: ...1) AS st_geom FROM at_po_4pgr, veh_line WHERE id=1 AND i...

As of pandas, I haven't tested it yet. I will try it next time!

Thank you for your help.

[amillb]

That's a strange error. There should not be any ambiguous columns in this part of the code.

Can you do me a favor and (i) send me your config file, and (ii) the command string that is causing the problem. If you are using ipython, simply type %debug and then print(cmd) after the error occurs. Alternatively, add a line before line 462 as follows:

print(cmd)

Let me know if you need help figuring out how to do this.

[ng9891]

Here is the output for the first 7 lines (where the problem is):

UPDATE veh_line SET matched_line = q.geom FROM (SELECT ST_RemoveRepeatedPoints(merged_line) AS geom FROM (SELECT ST_LineFromMultiPoint(ST_Collect((nodes).geom)) AS merged_line FROM (SELECT lorder, st_dumppoints(st_geom) AS nodes FROM ( SELECT 1 AS lorder, ST_Reverse(ST_LineSubString(geom, 0, ST_LineLocatePoint(geom, ST_StartPoint(geom_m)))) AS st_geom FROM at2_nyc, veh_line WHERE id=7 AND id = 102032 UNION ALL SELECT 2 AS lorder, geom AS st_geom FROM at2_nyc WHERE id = 74288 UNION ALL SELECT 3 AS lorder, geom AS st_geom FROM at2_nyc WHERE id = 74700 ...

The first "id=7" refers to the vehicle id table and the second id must be the edge id from osm2po table.

The cmd string:

python3 pgMapMatch pg veh_line id geom_m --streets=at2_nyc --ids=7 --write_edges --write_score --write_geom

As for the changes I made from the config.py, here is maybe the relevant section:

# column identifiers for the PostGIS table of streets # the default values here are compatible with osm2po streetIdCol = 'id' # unique id for street edge (i.e. segment or block) streetGeomCol = 'geom' # geometry column (LineString) for street edge startNodeCol = 'source' # id of node at which the street edge starts endNodeCol = 'target' # id of node at which the street edge ends travelCostCol = 'cost' # generalized cost to go from startNode to endNode travelCostReverseCol = 'reverse_cost' # generalized cost to go from endNode to startNode. Can be same as travelCostCol if you have no one-way streets streetLengthCol = 'km' # length of street, in km speedLimitCol = 'kmh' # speed limit on street, in km per hour

Please let me know if you need more information.

[amillb]

Aha. Thanks for finding this. The code wasn't robust to situations where the GPS trace table has column names that are the same as the streets table.

It should be fixed now, but I'd be grateful if you could confirm.

[ng9891]

I ran the recent commit code again. It gave me the following error:

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "pgMapMatch/main.py", line 103, in mm.writeMatchToPostgres(writeEdgeIds=args['--write_edges'], writeGeom=args['--write_geom'], writeMatchScore=args['--write_score']) File "pgMapMatch/mapmatcher.py", line 436, in writeMatchToPostgres if writeMatchScore: match_score = self.getMatchScore() File "pgMapMatch/mapmatcher.py", line 577, in getMatchScore if self.qualityModelCoeffs is None: self.loadQualityModel() File "pgMapMatch/mapmatcher.py", line 607, in loadQualityModel raise Exception('Quality model coefficients file %s not found' % self.qualityModelFn) Exception: Quality model coefficients file mapMatching_coefficients.txt not found

Problem seems to be on the camel case in the string "mapMatching_coefficients.txt". So I changed the string to all lower case "mapmatching_coefficients.txt" in line 151 to match the name of the file.

It did not give me more problems when I ran the changes. The column _matchedline seems to work perfectly. However, the _matchscore column gave me a score of 0. Does it mean that it is confident with the match?

[amillb]

Thanks for fixing the filename. That's now fixed in the code.

No, the zero score implies 0% confidence in the match. How does the match look to you?

I added a verbose mode to getMatchScore(), so if it doesn't look right, please do run: mm.getMatchScore(verbose=True) after matching a trace, and let me know what the output looks like. (If you need help getting that to run, let me know.)

Thanks for identifying these issues. We only tested it on a small number of system configurations, so this is very helpful.

[ng9891]

It is my pleasure to be able to help. I checked the resulting graph and compared it to my GPS traces, visually it seems to be working correctly. However, for some reason the match_score is 0.

I tried running pgMapMatch by importing it into a python file but gave me this error:

File "/home/user/.local/lib/python3.6/site-packages/pgMapMatch/__init__.py", line 2, in <module>
    from mapmatcher import *
ModuleNotFoundError: No module named 'mapmatcher'

So I looked online for the solution for this problem and I changed __init__.py:

from mapmatcher import * to from .mapmatcher import *

and it did not give me that error again. However, I get the following error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import pgMapMatch
  File "/home/user/.local/lib/python3.6/site-packages/pgMapMatch/__init__.py", line 2, in <module>
    from .mapmatcher import *
  File "/home/user/.local/lib/python3.6/site-packages/pgMapMatch/mapmatcher.py", line 18, in <module>
    import tools as mmt
ModuleNotFoundError: No module named 'tools'

It seems to be the same type of error from the previous file. So, I changed mapmatcher.py line 18 to: from . import tools as mmt and it worked fine.

Ran the following code:

import pgMapMatch
mm = pgMapMatch.mapMatcher('at2_nyc', traceTable='veh_line', idName= 'id', geomName='geom_m')
mm.matchPostgresTrace(10)
mm.bestRoute
mm.getMatchScore(verbose=True)

Gave me the following error:

Coefficients: {'intercept': 4.5289042565, 'frechet_dist': -0.0446756552965, 'll_topol_min': 0.00200883204845}
Frechet: nan
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    mm.getMatchScore(verbose=True)
  File "/home/user/.local/lib/python3.6/site-packages/pgMapMatch/mapmatcher.py", line 587, in getMatchScore
    if verbose: print(llName+': '+self.LL[ii])
TypeError: must be str, not float

Jumping to line 587, I changed it to: print(llName+': '+str(self.LL[ii])) and this is the output:

Coefficients: {'intercept': 4.5289042565, 'frechet_dist': -0.0446756552965, 'll_topol_min': 0.00200883204845}
Frechet: nan
ll_dist_mean: -0.676858044432683
ll_dist_min: -3.3673442771955595
ll_topol_mean: 0.8633089866650284
ll_topol_min: 0.2232703948377163
ll_distratio_mean: -0.8655811623533194
ll_distratio_min: -9.099864629357617

Not sure if Frechet being nan causes a problem.

PS: A brief description on how to use pgMapMatch.mapMatcher() together with matchPostgresTrace() in the readme would be appreciated as some of the input variables are mandatory for it to work.

[amillb]

Thank you for your detailed explanation. It looks like the Frechet distance is not being calculated, and thus the match score is not correct.

I have updated (not yet commmited) to return a match score of Null when this error occurs. However, I'd like to figure out why your Frechet distance is not computing.

Would you mind replacing line 1015 (except:) with the following 3 lines:

        except Exception as e:  # 'AttributeError if matched_line is None
            print(e.message)
            print(cmd)

And let me know the output.

I've moved the other questions about imports, etc. to a separate issue. It's probably a question of python paths, but I will need to test on a wider variety of systems.

[ng9891]

Here is the output: Two of the same error.

print(e)-----------------------------------------------------------
'NoneType' object has no attribute 'strip'
print(cmd)*********************************************************
WITH l AS (SELECT ST_Force2D(matched_line) as l1,
                                       ST_Force2D(geom_m) as l2
                                 FROM veh107_line where id=18)
SELECT  ST_AsText(ST_LineInterpolatePoint(l1,0.0)), ST_AsText(ST_LineInterpolatePoint(l2,0.0)), ST_AsText(ST_LineInterpolatePoint(l1,0.0025)), ST_AsText(ST_LineInterpolatePoint(l2,0.0025)),...
...ST_AsText(ST_LineInterpolatePoint(l2,0.9875)), ST_AsText(ST_LineInterpolatePoint(l1,0.99)), ST_AsText(ST_LineInterpolatePoint(l2,0.99)), ST_AsText(ST_LineInterpolatePoint(l1,0.9925)), ST_AsText(ST_LineInterpolatePoint(l2,0.9925)), ST_AsText(ST_LineInterpolatePoint(l1,0.995)), ST_AsText(ST_LineInterpolatePoint(l2,0.995)), ST_AsText(ST_LineInterpolatePoint(l1,0.9975)), ST_AsText(ST_LineInterpolatePoint(l2,0.9975)) FROM l

print(e)-----------------------------------------------------------
'NoneType' object has no attribute 'strip'
print(cmd)*********************************************************
WITH l AS (SELECT ST_Force2D(matched_line) as l1,
                                       ST_Force2D(geom_m) as l2
                                 FROM veh107_line where id=18)
SELECT  ST_AsText(ST_LineInterpolatePoint(l1,0.0)), ST_AsText(ST_LineInterpolatePoint(l2,0.0)), ST_AsText(ST_LineInterpolatePoint(l1,0.0025)), ST_AsText(ST_LineInterpolatePoint(l2,0.0025)),...
...ST_AsText(ST_LineInterpolatePoint(l2,0.9875)), ST_AsText(ST_LineInterpolatePoint(l1,0.99)), ST_AsText(ST_LineInterpolatePoint(l2,0.99)), ST_AsText(ST_LineInterpolatePoint(l1,0.9925)), ST_AsText(ST_LineInterpolatePoint(l2,0.9925)), ST_AsText(ST_LineInterpolatePoint(l1,0.995)), ST_AsText(ST_LineInterpolatePoint(l2,0.995)), ST_AsText(ST_LineInterpolatePoint(l1,0.9975)), ST_AsText(ST_LineInterpolatePoint(l2,0.9975)) FROM l

[amillb]

Interesting. Thanks for checking. It seems that either matched_line or geom_m are Null.

What happens if in postgres you run this? Perhaps the matched_line is not being written correctly to the database.

SELECT ST_Force2D(matched_line) as l1,
                                       ST_Force2D(geom_m) as l2
                                 FROM veh107_line where id=18;

[ng9891]

It seems that when running in code, the python code doesn't write to the Database to fill the matched_line column so it is going to be Null.

So, I tried with the console again. This time it didn't give me any output for the Exception. However, match_score is still 0.

[amillb]

Thanks. I just pushed many changes to 7ca4efa, which add more documentation, and should fix the import issues you mentioned earlier as well.

Let me know if this works. Note that with the console version, you need to explicitly ask to have match_score written to the database using the --write_score argument. For example:

python pgMapMatch pg gps_trace_table trace_id trace_geom --streets=sf_streets --write_geom --write_score

I'm puzzled that when running from the Python interpreter, the matched line is not being written to postgres.

If there's still a problem, please report the exact commands that you are using, and any output that results.

Thanks again for being a guinea pig. It's very helpful to see where the code is breaking. I've tried to make it more generally applicable, but there are still places where it's very specific to our installations/usage.

[ng9891]

Sorry for the delay. Here is the code I used to trace id=9. This trace id already has a matched_line written in the PostgresSQL table from the command line but has a matched score of 0.

import pgMapMatch
mm = pgMapMatch.mapMatcher('at2_nyc', traceTable='veh107_line', idName= 'id', geomName='geom_m')
mm.matchPostgresTrace(9)
print(mm.bestRoute)        # returns the sequence of edge ids (based on the id column in the streets table)
print(mm.getMatchScore(verbose=True))  # returns the match score (probability that the match is good)

Output:

Coefficients: {'intercept': 4.5289042565, 'frechet_dist': -0.0446756552965, 'll_topol_min': 0.00200883204845}
Frechet: 3572.280866477734
ll_dist_mean: -0.7352206439522809
ll_dist_min: -3.6290331809093597
ll_topol_mean: 0.8257424718278962
ll_topol_min: -0.21604906078591038
ll_distratio_mean: -0.9853087973324897
ll_distratio_min: -12.643200730127068

However, the score on the PG table still remains 0. It does not seem to be writing on the table.

I tried again with trace id=12. This trace, however, does not have anything on the matched_line column. Here is the error:

Coefficients: {'intercept': 4.5289042565, 'frechet_dist': -0.0446756552965, 'll_topol_min': 0.00200883204845}
'NoneType' object has no attribute 'strip'
WITH l AS (SELECT ST_Force2D(matched_line) as l1,
                                       ST_Force2D(geom_m) as l2
                                 FROM veh107_line where id=12)
SELECT  ST_AsText(ST_LineInterpolatePoint(l1,0.0)), ST_AsText(ST_LineInterpolatePoint(l2,0.0)),...
...Point(l2,0.995)), ST_AsText(ST_LineInterpolatePoint(l1,0.9975)), ST_AsText(ST_LineInterpolatePoint(l2,0.9975)) FROM l
Frechet: nan
ll_dist_mean: -0.9323813199701173
ll_dist_min: -6.379891361460712
ll_topol_mean: 0.8348653133770559
ll_topol_min: 0.5859094077704937
ll_distratio_mean: -0.8910367729581739
ll_distratio_min: -2.32031771520984
nan

As expected, it did not write into the database either when I checked the row. Maybe that is why it is giving a score of nan.

If it is not too much to ask, I would like to know if the score is correct for the first result where it has a Frechet of 3572.280866477734.

[amillb]

Thank you for the detailed reports.

For trace id 9, the score of zero is correct if the Frechet distance is 3572. How does the matched line compare to the original trace when you view it in (say) QGIS? Does 3.6km look correct?

I just pushed an update that will calculate Frechet distances correctly when the projection units are not in meters.

For trace id 12, it looks like the match was not possible. I fixed the exception handling, and it should print the error and the command this time, before skipping to the next trace. Let me know if this works.

[amillb]

Closing this issue; please reopen if you still have problems.