search

SEL-Columbia / sequencer

Python library for sequencing the output of Network Planner csv's and shape file outputs
Other
4 stars 4 forks source link

MemoryError on large networks #41

Closed chrisnatali closed 9 years ago

chrisnatali commented 9 years ago

Copying from https://github.com/SEL-Columbia/modelrunner/issues/23

From @carbz

Confirming this issue still exists. Tested full KEDCO dataset via networker found here http://23.253.225.19:8888/worker_data/d75771fb-f073-4c88-95c8-a82f368720ec/job_log.txt

Similar error message results...

reading input from /home/mr/modelrunner/worker_data/d75771fb-f073-4c88-95c8-a82f368720ec/input discarding /home/mr/miniconda/envs/modelrunner/bin from PATH prepending /home/mr/miniconda/envs/sequencer/bin to PATH 2015-05-14 15:59:08,747 : NetworkPlan [INFO] : Asserting Input Projections Match 2015-05-14 15:59:08,751 : NetworkPlan [INFO] : Aligning Network Nodes With Input Metrics 2015-05-14 15:59:25,884 : NetworkPlan [INFO] : Computing Pairwise Distances 2015-05-14 15:59:25,884 : NetworkPlan [INFO] : Using haversine Distance 2015-05-14 15:59:35,243 : NetworkPlan [INFO] : Directing Network Away From Roots 2015-05-14 15:59:36,022 : NetworkPlan [INFO] : Directing SUBGRAPH 1 / 6 2015-05-14 15:59:36,075 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,076 : NetworkPlan [INFO] : Directing SUBGRAPH 2 / 6 2015-05-14 15:59:36,109 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,109 : NetworkPlan [INFO] : Directing SUBGRAPH 3 / 6 2015-05-14 15:59:36,132 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,132 : NetworkPlan [INFO] : Directing SUBGRAPH 4 / 6 2015-05-14 15:59:36,170 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,171 : NetworkPlan [INFO] : Directing SUBGRAPH 5 / 6 2015-05-14 15:59:36,225 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,226 : NetworkPlan [INFO] : Directing SUBGRAPH 6 / 6 2015-05-14 15:59:36,239 : NetworkPlan [INFO] : DONE! 2015-05-14 15:59:36,834 : Sequencer [INFO] : Converting The Network to a HashMap (Warning: this is time consuming for large Networks) Traceback (most recent call last): File "/home/mr/modelrunner/scripts/mvmax_sequencer.py", line 22, in results = model.sequence() ... File "/home/mr/miniconda/envs/sequencer/lib/python2.7/site-packages/scipy/sparse/base.py", line 793, in _process_toarray_args return np.zeros(self.shape, dtype=self.dtype, order=order) MemoryError

chrisnatali commented 9 years ago

This appears to be due to the recursive construction of the adjacency dict representation of the network here https://github.com/SEL-Columbia/Sequencer/blob/master/sequencer/NetworkPlan.py#L230. Looks like the adj_matrix is allocated for each call...probably putting a lot of memory on the stack when the network is large. Need to instantiate it once for this.

chrisnatali commented 9 years ago

Duplicate of #42