jgc234 / optiburb

Cycling route generator for optimum "burbing" route.
MIT License
32 stars 7 forks source link

Western Australia Suburbs #6

Closed oog123456 closed 4 years ago

oog123456 commented 4 years ago

Hi Jim,

Awesome! Thanks a lot for releasing. This is my first foray into Python/coding and i have managed to figure out enough to get things setup. I am playing around with Perth suburbs. But having some issues. It seems there are no suburb boundaries saved and just nodes for the centre of the suburb. Example: https://www.openstreetmap.org/node/31071659#map=14/-31.9799/115.8057

If i search for "City of Nedlands Western Australia Australia" then i get the council boundaries. https://www.openstreetmap.org/relation/11343563 GPX route attached; City of Nedlands Burbing.zip Only 275km to try and do in a day. ;)

But a lot of the councils don't have their "City of __" boundaries listed on OpenStreetMaps. So i guess i need to learn how to use shape files now?

Also, any idea how to get it to work for USA? Or would the code work if there is a way to make a custom shape file for an area?

Look forward to your responses. And keen to learn more from here.

Thanks

sugarlata commented 4 years ago

I've bumped into the same issue with Doncaster in Victoria. After doing some research into the cause this is what I'm finding - this is more for @jgc234 as repo owner.

One possibility could be to provide instructions on identifying the boundary ID, then use that as an input in the script. Reuse the code from osmnx.gdf_from_place() to ingest a overpass api json response instead. I'd be interested in helping out if that would be helpful and a worthwhile direction to go @jgc234?

jgc234 commented 4 years ago

That's interesting.. OSM doesn't seem to have the suburb boundaries for WA, unless I'm not looking in the right place.

For the shapefile, try this.. Download the shapefile from:

https://data.gov.au/data/dataset/6a0ec945-c880-4882-8a81-4dbcb85e74e5/resource/141fc7bd-c75f-49b5-a116-35250eea68cd/download/wa_locality_polygon_shp.zip

Unzip it, and point optiburb to it with --shapefile=WA_LOCALITY_POLYGON_shp,WA_LOCAL_2 NEDLANDS

If that doesn't work gimmie a yell and I'll test try it myself.

oog123456 commented 4 years ago

Awesome, thanks very much. Will give it a shot in a couple of hours.

oog123456 commented 4 years ago

@jgc234 Is it possible to use that shape file for multiple suburbs sharing a boundary?

jgc234 commented 4 years ago

Yup - just put both names at the end of the command line and the polygons will be merged..

oog123456 commented 4 years ago

It has worked for a few metro suburbs i have done so far, but not Margaret RIver a down South of Perth.

Traceback (most recent call last): File "optiburb.py", line 654, in <module> polygon = burbing.get_shapefile_polygon(shapefile, key, name) File "optiburb.py", line 112, in get_shapefile_polygon polygon = suburb['geometry'].values[0] File "/Users/JB/opt/anaconda3/lib/python3.8/site-packages/geopandas/array.py", line 324, in __getitem__ return _geom_to_shapely(self.data[idx]) IndexError: index 0 is out of bounds for axis 0 with size 0

ANOTHER EXAMPLE: Traceback (most recent call last): File "optiburb.py", line 682, in <module> graph_aug, euler_circuit = burbing.determine() File "optiburb.py", line 241, in determine odd_pair_paths = self.get_shortest_path_pairs(self.g, odd_node_pairs) File "optiburb.py", line 152, in get_shortest_path_pairs shortest_paths[pair] = nx.dijkstra_path_length(g, i, j, weight='length') File "/Users/JB/opt/anaconda3/lib/python3.8/site-packages/networkx/algorithms/shortest_paths/weighted.py", line 244, in dijkstra_path_length raise nx.NetworkXNoPath( networkx.exception.NetworkXNoPath: Node 5519839242 not reachable from 5404385285

jgc234 commented 4 years ago

Margaret River seems to be working OK for me so far (for your first example).. Make sure you've quoted any names with spaces.

./optiburb.py --shapefile=/Users/me/Downloads/WA_LOCALITY_POLYGON_shp/WA_LOCALITY_POLYGON_shp.shp,WA_LOCAL_2 "MARGARET RIVER"

I'm not sure about the second case.. That's further along in the code (the last part of the calculations), and it looks like part of your suburb got isolated. What args did you use for that one?

oog123456 commented 4 years ago

I removed --prune from some suburbs and then it started to work, however here is another in Perth with arguments attached.

Downloads JB$ python3 optiburb.py --debug=debug --simplify-gpx --save-boundary --shapefile ~/Downloads/WA_LOCALITY_POLYGON_shp,WA_LOCAL_2 MOUNT HAWTHORN

[1737 rows x 13 columns], key=WA_LOCAL_2, name=MOUNT 2020-08-02 09:39:46 optiburb.py:get_shapefile_polygon:110 [INFO] suburb=Empty GeoDataFrame Columns: [LC_PLY_PID, DT_CREATE, DT_RETIRE, LOC_PID, WA_LOCALIT, WA_LOCAL_1, WA_LOCAL_2, WA_LOCAL_3, WA_LOCAL_4, WA_LOCAL_5, WA_LOCAL_6, WA_LOCAL_7, geometry] Index: [] Traceback (most recent call last): File "optiburb.py", line 654, in <module> polygon = burbing.get_shapefile_polygon(shapefile, key, name) File "optiburb.py", line 112, in get_shapefile_polygon polygon = suburb['geometry'].values[0] File "/Users/JB/opt/anaconda3/lib/python3.8/site-packages/geopandas/array.py", line 324, in __getitem__ return _geom_to_shapely(self.data[idx]) IndexError: index 0 is out of bounds for axis 0 with size 0

I havent been using "" around the suburbs with the shapefile. This may be the cause of all my errors?

jgc234 commented 4 years ago

Yes, you'll need quotes around suburb name if it has spaces in it.

oog123456 commented 4 years ago

@jgc234 Now to start riding all these routes!

felt2129 commented 4 years ago

I was able to successfully create a gpx file for Bellfield as per the example, but I'm getting an error for WA suburbs. I followed the instructions above but am getting the error below.

Thanks in advance for any suggestions.

runfile('C:/Users/felt2129/Documents/Personal/optiburb-master/optiburb.py', args='--shapefile=/Users/felt2129/Documents/Personal/optiburb-master/WA_LOCALITY_POLYGON_shp.shp,WA_LOCAL_2 "WEMBLEY DOWNS"', wdir='C:/Users/felt2129/Documents/Personal/optiburb-master') 2020-08-29 13:21:21 optiburb.py:init:86 [WARNING] WARNING - this program does not consider the direction of one-way roads or other roads that may be not suitable for your mode of transport. You must confirm the path safe for yourself 2020-08-29 13:21:21 optiburb.py::724 [INFO] shapefile=/Users/felt2129/Documents/Personal/optiburb-master/WA_LOCALITY_POLYGON_shp.shp, key=WA_LOCAL_2 2020-08-29 13:21:22 optiburb.py:load_shapefile:538 [INFO] df= LC_PLY_PID ... geometry 0 20345 ... POLYGON ((116.60021 -31.50806, 116.60019 -31.5... 1 21003 ... POLYGON ((118.14971 -30.23658, 118.15478 -30.2... 2 20306 ... POLYGON ((116.81134 -31.60829, 116.81142 -31.6... 3 20296 ... POLYGON ((115.74417 -33.50593, 115.74477 -33.5... 4 20621 ... POLYGON ((115.77720 -33.54038, 115.77721 -33.5... ... ... ... 1733 21311 ... POLYGON ((115.82114 -31.91525, 115.82114 -31.9... 1734 21370 ... POLYGON ((115.65816 -33.51382, 115.65824 -33.5... 1735 21273 ... POLYGON ((115.97219 -32.19487, 115.97240 -32.1... 1736 21274 ... POLYGON ((115.99235 -32.18022, 115.99294 -32.1... 1737 21606 ... POLYGON ((115.90081 -31.96990, 115.90052 -31.9...

[1738 rows x 5 columns] 2020-08-29 13:21:22 optiburb.py:load_shapefile:539 [INFO] df.crs=epsg:4283 2020-08-29 13:21:22 optiburb.py:get_shapefile_polygon:126 [INFO] shapefile= LC_PLY_PID ... geometry 0 20345 ... POLYGON ((116.60021 -31.50806, 116.60019 -31.5... 1 21003 ... POLYGON ((118.14971 -30.23658, 118.15478 -30.2... 2 20306 ... POLYGON ((116.81134 -31.60829, 116.81142 -31.6... 3 20296 ... POLYGON ((115.74417 -33.50593, 115.74477 -33.5... 4 20621 ... POLYGON ((115.77720 -33.54038, 115.77721 -33.5... ... ... ... 1733 21311 ... POLYGON ((115.82114 -31.91525, 115.82114 -31.9... 1734 21370 ... POLYGON ((115.65816 -33.51382, 115.65824 -33.5... 1735 21273 ... POLYGON ((115.97219 -32.19487, 115.97240 -32.1... 1736 21274 ... POLYGON ((115.99235 -32.18022, 115.99294 -32.1... 1737 21606 ... POLYGON ((115.90081 -31.96990, 115.90052 -31.9...

[1738 rows x 5 columns], key=WA_LOCAL_2, name=WEMBLEY DOWNS Traceback (most recent call last):

File "C:\Users\felt2129\Documents\Personal\WPy64-3830\python-3.8.3.amd64\lib\site-packages\pandas\core\indexes\base.py", line 2646, in get_loc return self._engine.get_loc(key)

File "pandas_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc

File "pandas_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc

File "pandas_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item

File "pandas_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item

KeyError: 'WA_LOCAL_2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "C:\Users\felt2129\Documents\Personal\optiburb-master\optiburb.py", line 729, in polygon = burbing.get_shapefile_polygon(shapefile, key, name)

File "C:\Users\felt2129\Documents\Personal\optiburb-master\optiburb.py", line 130, in get_shapefile_polygon suburb = df[df[key] == name]

File "C:\Users\felt2129\Documents\Personal\WPy64-3830\python-3.8.3.amd64\lib\site-packages\geopandas\geodataframe.py", line 576, in getitem result = super(GeoDataFrame, self).getitem(key)

File "C:\Users\felt2129\Documents\Personal\WPy64-3830\python-3.8.3.amd64\lib\site-packages\pandas\core\frame.py", line 2800, in getitem indexer = self.columns.get_loc(key)

File "C:\Users\felt2129\Documents\Personal\WPy64-3830\python-3.8.3.amd64\lib\site-packages\pandas\core\indexes\base.py", line 2648, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key))

File "pandas_libs\index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc

File "pandas_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc

File "pandas_libs\hashtable_class_helper.pxi", line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item

File "pandas_libs\hashtable_class_helper.pxi", line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item

KeyError: 'WA_LOCAL_2'

jgc234 commented 4 years ago

@felt2129 - that looks like it should work to me..

Some thoughts:

Here's my output:

(slave-5)[/Users/jgc/Projects/optiburb] jgc% ./optiburb.py --shapefile ~/Projects/gis/WA_LOCALITY_POLYGON_shp/WA_LOCALITY_POLYGON_shp.shp,WA_LOCAL_2 "WEMBLEY DOWNS"
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
2020-08-29 15:52:49 Configured osmnx
2020-08-29 15:52:49 optiburb.py:__init__:86 [WARNING] WARNING - this program does not consider the direction of one-way roads or other roads that may be not suitable for your mode of transport. You must confirm the path safe for yourself
2020-08-29 15:52:49 optiburb.py:<module>:724 [INFO] shapefile=/Users/jgc/Projects/gis/WA_LOCALITY_POLYGON_shp/WA_LOCALITY_POLYGON_shp.shp, key=WA_LOCAL_2
2020-08-29 15:52:49 collection.py:crs:207 [INFO] Failed to auto identify EPSG: 7
2020-08-29 15:52:50 optiburb.py:load_shapefile:538 [INFO] df=     LC_PLY_PID   DT_CREATE DT_RETIRE LOC_PID  WA_LOCALIT WA_LOCAL_1      WA_LOCAL_2 WA_LOCAL_3 WA_LOCAL_4 WA_LOCAL_5 WA_LOCAL_6 WA_LOCAL_7                                           geometry
0         21062  2019-05-22      None  WA1702  2018-05-05       None          MARDIE       None       None          G       None          5  POLYGON ((115.47186 -21.07534, 115.47215 -21.0...
1         21053  2019-05-22      None  WA1702  2018-05-05       None          MARDIE       None       None          G       None          5  POLYGON ((116.03249 -20.91230, 116.03209 -20.9...
2         21033  2019-05-22      None  WA1704  2012-07-27       None   BARROW ISLAND       None       None          G       None          5  POLYGON ((115.46461 -20.79073, 115.46461 -20.7...
3         21056  2019-05-22      None  WA1704  2012-07-27       None   BARROW ISLAND       None       None          G       None          5  POLYGON ((115.49201 -20.75366, 115.49168 -20.7...
4         21072  2019-05-22      None  WA1707  2015-11-09       None       EMU POINT       None       None          G       None          5  POLYGON ((117.95232 -34.98519, 117.95252 -34.9...
...         ...         ...       ...     ...         ...        ...             ...        ...        ...        ...        ...        ...                                                ...
1732      20504  2019-05-22      None  WA3366  2018-08-03       None       NAREMBEEN       None       None          G       None          5  POLYGON ((118.39423 -32.07007, 118.39355 -32.0...
1733      19346  2019-05-22      None  WA3368  2013-08-06       None  MAHOMETS FLATS       None       None          G       None          5  POLYGON ((114.61875 -28.79658, 114.61657 -28.7...
1734      20453  2019-05-22      None  WA3370  2017-11-01       None  MOUNT TARCOOLA       None       None          G       None          5  POLYGON ((114.62517 -28.78666, 114.62516 -28.7...
1735      19460  2019-05-22      None  WA3384  2010-04-27       None    MILLARS WELL       None       None          G       None          5  POLYGON ((116.82340 -20.73641, 116.82340 -20.7...
1736      20747  2019-05-22      None  WA3388  2014-08-01       None     GLENDALOUGH       None       None          G       None          5  POLYGON ((115.82340 -31.91651, 115.82348 -31.9...

[1737 rows x 13 columns]
2020-08-29 15:52:51 optiburb.py:load_shapefile:539 [INFO] df.crs=GEOGCS["GDA94",DATUM["GDA_1994",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]]
2020-08-29 15:52:51 optiburb.py:get_shapefile_polygon:126 [INFO] shapefile=     LC_PLY_PID   DT_CREATE DT_RETIRE LOC_PID  WA_LOCALIT WA_LOCAL_1      WA_LOCAL_2 WA_LOCAL_3 WA_LOCAL_4 WA_LOCAL_5 WA_LOCAL_6 WA_LOCAL_7                                           geometry
0         21062  2019-05-22      None  WA1702  2018-05-05       None          MARDIE       None       None          G       None          5  POLYGON ((115.47186 -21.07534, 115.47215 -21.0...
1         21053  2019-05-22      None  WA1702  2018-05-05       None          MARDIE       None       None          G       None          5  POLYGON ((116.03249 -20.91230, 116.03209 -20.9...
2         21033  2019-05-22      None  WA1704  2012-07-27       None   BARROW ISLAND       None       None          G       None          5  POLYGON ((115.46461 -20.79073, 115.46461 -20.7...
3         21056  2019-05-22      None  WA1704  2012-07-27       None   BARROW ISLAND       None       None          G       None          5  POLYGON ((115.49201 -20.75366, 115.49168 -20.7...
4         21072  2019-05-22      None  WA1707  2015-11-09       None       EMU POINT       None       None          G       None          5  POLYGON ((117.95232 -34.98519, 117.95252 -34.9...
...         ...         ...       ...     ...         ...        ...             ...        ...        ...        ...        ...        ...                                                ...
1732      20504  2019-05-22      None  WA3366  2018-08-03       None       NAREMBEEN       None       None          G       None          5  POLYGON ((118.39423 -32.07007, 118.39355 -32.0...
1733      19346  2019-05-22      None  WA3368  2013-08-06       None  MAHOMETS FLATS       None       None          G       None          5  POLYGON ((114.61875 -28.79658, 114.61657 -28.7...
1734      20453  2019-05-22      None  WA3370  2017-11-01       None  MOUNT TARCOOLA       None       None          G       None          5  POLYGON ((114.62517 -28.78666, 114.62516 -28.7...
1735      19460  2019-05-22      None  WA3384  2010-04-27       None    MILLARS WELL       None       None          G       None          5  POLYGON ((116.82340 -20.73641, 116.82340 -20.7...
1736      20747  2019-05-22      None  WA3388  2014-08-01       None     GLENDALOUGH       None       None          G       None          5  POLYGON ((115.82340 -31.91651, 115.82348 -31.9...

[1737 rows x 13 columns], key=WA_LOCAL_2, name=WEMBLEY DOWNS
2020-08-29 15:52:51 optiburb.py:get_shapefile_polygon:132 [INFO] suburb=    LC_PLY_PID   DT_CREATE DT_RETIRE LOC_PID  WA_LOCALIT WA_LOCAL_1     WA_LOCAL_2 WA_LOCAL_3 WA_LOCAL_4 WA_LOCAL_5 WA_LOCAL_6 WA_LOCAL_7                                           geometry
882      20194  2019-05-22      None  WA3322  2018-08-03       None  WEMBLEY DOWNS       None       None          G       None          5  POLYGON ((115.76615 -31.91078, 115.76615 -31.9...
2020-08-29 15:52:51 optiburb.py:load:520 [INFO] fetching OSM data bounded by polygon
2020-08-29 15:52:51 Projected GeoDataFrame to +proj=utm +zone=50 +ellps=WGS84 +datum=WGS84 +units=m +no_defs +type=crs
2020-08-29 15:52:51 Projected GeoDataFrame to epsg:4326
2020-08-29 15:52:51 Projected GeoDataFrame to +proj=utm +zone=50 +ellps=WGS84 +datum=WGS84 +units=m +no_defs +type=crs
2020-08-29 15:52:51 Projected GeoDataFrame to epsg:4326
2020-08-29 15:52:51 Requesting network data within polygon from API in 1 request(s)
2020-08-29 15:52:51 Retrieved response from cache file "cache/a610e58b149b8f9903291e3c8b05f842.json"
2020-08-29 15:52:51 Got all network data within polygon from API in 1 request(s)
2020-08-29 15:52:51 Creating graph from downloaded OSM data...
2020-08-29 15:52:51 Created graph with 3469 nodes and 6131 edges
2020-08-29 15:52:51 Added edge lengths to graph
2020-08-29 15:52:51 Identifying all nodes that lie outside the polygon...
2020-08-29 15:52:52 Created r-tree spatial index for 3469 points
2020-08-29 15:52:52 Identified 2928 nodes inside polygon
2020-08-29 15:52:52 Removed 541 nodes outside polygon
2020-08-29 15:52:52 Truncated graph by polygon
2020-08-29 15:52:52 Identifying all nodes that lie outside the polygon...
felt2129 commented 4 years ago

Thanks for the quick reply.

After upgrading the packages, I still get the same error.

I notice that I have [1738 rows x 5 columns] while your out put has [1737 rows x 13 columns]. Could that be leading to the KeyError?

jgc234 commented 4 years ago

@felt2129 - I'd double-check that shapefile. Open it up in QGIS or something similar and check to ensure the column exists, and is the name we're expecting.