Error Executing the Build Networks Tool

[aagervasi]

An error has occurred while executing Python code:

IndexError 
Traceback (most recent call last):
  File "C:/Users/Arielle/.qgis2/python/plugins\GNAT\build_network_dialog.py", line 167, in calc_subnetwork_id
    UG = network.calc_network_id(list_SG)
  File "C:/Users/Arielle/.qgis2/python/plugins\GNAT\network.py", line 177, in calc_network_id
    raise IndexError  # not sure about this... will probably change later
IndexError

Python version: 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] 
QGIS version: 2.18.4 Las Palmas, 59c1d21 

Python Path:
C:/OSGEO4~1/apps/qgis/./python/plugins\processing
C:/OSGEO4~1/apps/qgis/./python
C:/Users/Arielle/.qgis2/python
C:/Users/Arielle/.qgis2/python/plugins
C:/OSGEO4~1/apps/qgis/./python/plugins
C:\OSGEO4~1\apps\Python27\lib\site-packages\matplotlib-1.3.1-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\nose-1.3.3-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\tornado-4.0.1-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\backports.ssl_match_hostname-3.4.0.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\certifi-14.05.14-py2.7.egg
c:\osgeo4~1\apps\python27\lib\site-packages\python_dateutil-2.1-py2.7.egg
C:\Python27\ArcGIS10.1
C:\OSGEO4~1\bin\python27.zip
C:\OSGEO4~1\apps\Python27\DLLs
C:\OSGEO4~1\apps\Python27\lib
C:\OSGEO4~1\apps\Python27\lib\plat-win
C:\OSGEO4~1\apps\Python27\lib\lib-tk
C:\OSGEO4~1\bin
C:\OSGEO4~1\apps\Python27
C:\OSGEO4~1\apps\Python27\lib\site-packages
C:\OSGEO4~1\apps\Python27\lib\site-packages\PIL
C:\OSGEO4~1\apps\Python27\lib\site-packages\jinja2-2.7.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\markupsafe-0.23-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\pytz-2012j-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\win32
C:\OSGEO4~1\apps\Python27\lib\site-packages\win32\lib
C:\OSGEO4~1\apps\Python27\lib\site-packages\Pythonwin
C:\OSGEO4~1\apps\Python27\lib\site-packages\Shapely-1.2.18-py2.7-win-amd64.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8-msw-unicode
C:\OSGEO4~1\apps\Python27\lib\site-packages\xlrd-0.9.2-py2.7.egg
C:\OSGEO4~1\apps\Python27\lib\site-packages\xlwt-0.7.5-py2.7.egg
C:/Users/Arielle/.qgis2//python

[aagervasi]

Does the shapefile need to be projected before attempting to run this tool?

[jesselangdon]

There seems to be an issue regarding the size of this network shapefile and NetworkX running out of memory when creating a graph.

[aagervasi]

I will try with a smaller network (tried with 1706 which is a HUC4)

On Mon, Jun 19, 2017 at 5:30 PM, Jesse Langdon notifications@github.com wrote:

There seems to be an issue regarding the size of this network shapefile and NetworkX running out of memory when creating a graph.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SouthForkResearch/pyGNAT/issues/27#issuecomment-309610231, or mute the thread https://github.com/notifications/unsubscribe-auth/AX5oVoABEcWLshOi17EP9tT3jPDXaqZeks5sFxLBgaJpZM4N6fEp .

[jesselangdon]

So the network shapefile in question is pretty big (the whole Snake River basin, i.e. a "HUC 2"), with ~260,000 stream reaches. Within the network.py/import_shp function, after importing this shapefile as a networkx graph, then sending the DiGraph to the get_subgraphs function, I get a Memory Error (I did this manually using PyCharm's python interpreter):

  File "<input>", line 1, in <module>
  File "C:\Python27\ArcGIS10.1\Lib\site-packages\networkx\algorithms\components\connected.py", line 111, in connected_component_subgraphs
    yield G.subgraph(c).copy()
  File "C:\Python27\ArcGIS10.1\Lib\site-packages\networkx\classes\graph.py", line 1448, in copy
    return deepcopy(self)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\OSGeo4W\apps\Python27\lib\copy.py", line 192, in deepcopy
    memo[d] = y
MemoryError

My local OSGeo4W install of Python is 32bit, but the original error that Arielle reported occurred on her local machine, which uses a 64bit OSGeo4W install. So maybe this is a limitation of the NetworkX library? This is the first time I've encountered this issue with NetworkX, but this is also the largest shapefile we've ever tried to process...

[MattReimer]

I haven't looked into it but I think networkx is a red herring. Everything I've seen leads me to believe that networkx can handle as big a network as you give it.

We might be running out of memory but if that's the case maybe we should investigate how many attributes we're storing. For example, if all we do is load the linkages and then do lookups for the attributes later that may save us quite a bit of breathing room.

Failing that we can look at storing the networks in a file format (like hdf5) and querying them on disk instead of in memory.

[jesselangdon]

For what it's worth, there are only 15 attribute fields associated with this particular shapefile. I also imported the shapefile to graph in a 'simplified' format, so no intermediate nodes included...