search

gflow / GFlow

Software for modeling circuit theory-based connectivity
GNU General Public License v3.0
22 stars 5 forks source link

gflow stops unexpectedly #22

Open jmsigner opened 7 years ago

jmsigner commented 7 years ago

In the attached example (https://dl.dropboxusercontent.com/u/5554895/ex1.zip) gflow unexpectedly stops after varying number of iterations (usually 100 to 300). The interesting thing is, that if I remove the second line from nodes_dummy everything works well (i.e. all iterations are completed).

Any hints on what I am doing wrong would be greatly appreciated.

pbleonard commented 7 years ago

@jmsigner can you please check access to the link? Is it private? Does not work for me.

jmsigner commented 7 years ago

@pbleonard: does this link work: https://www.dropbox.com/s/wleyyihc0l88o6x/ex1.zip?dl=0.

pbleonard commented 7 years ago

@jmsigner : you're attempting to write the output to a directory that may not exist. We have updated GFlow to handle this but here is what I would recommend. In your output_sum_density_filename flag please either use "./filepath" to reference the current working directory or change the current working directory with the flag OUTPUT_DIR on line 19 of the execute script. This should fix your crash problem.

jmsigner commented 7 years ago

@pbleonard: the directory exists. If remove the seconds line from nodes_dummy it runs fine.

pbleonard commented 7 years ago

@jmsigner The problem is not with the second line of nodes_dummy. To test, I simply clipped the nodes file (after the first 2 unique pairs -- so that each pair is unique) then randomly selected another node to calculate for 3 unique pairs. Please see the log of me running your inputs as well as your edited nodes_dummy file. Please try this input.

log.txt

nodes_dummy.txt

jmsigner commented 7 years ago

@pbleonard Surely, the second line is not the problem. But consider the following nodes files (nodes_dummy):

69 131
13 44
69 131
69 131
69 131
69 131
69 131

When I run gflow it sometimes works:

/usr/bin/mpiexec
Mon Sep 11 16:39:16 UTC 2017
Mon Sep 11 16:39:16 2017 >> Effective resistance will be written to R_eff.csv.
Mon Sep 11 16:39:16 2017 >> (rows,cols) = (228,200)
Mon Sep 11 16:39:16 2017 >> Removed 0 islands (0 cells).
Mon Sep 11 16:39:16 2017 >> 7 points in nodes_dummy
Mon Sep 11 16:39:16 2017 >> Max distance: 20000.00 pixels
Mon Sep 11 16:39:16 2017 >> 21 pairs generated.  0 skipped.
Mon Sep 11 16:39:16 2017 >> Number of unknowns: 45600
Mon Sep 11 16:39:16 2017 >> Solving pair 0 (1 of 21): 2[12,43] to 3[68,130].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> R_eff = 2,3,18692.018075
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 1 (2 of 21): 2[12,43] to 7[68,130].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 0 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 0.000000e+00 (0-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 2,7,18692.018075
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 2 (3 of 21): 1[68,130] to 2[12,43].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 1 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,2,18692.017911
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 3 (4 of 21): 3[68,130] to 4[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 2 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 9.999999e-01 (6-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 3,4,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 4 (5 of 21): 4[68,130] to 6[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 3 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 4,6,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 5 (6 of 21): 1[68,130] to 3[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 4 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,3,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 6 (7 of 21): 3[68,130] to 5[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 5 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 3,5,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 7 (8 of 21): 4[68,130] to 5[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 6 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 4,5,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 8 (9 of 21): 1[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 7 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,7,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 9 (10 of 21): 1[68,130] to 6[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 8 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,6,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 10 (11 of 21): 2[12,43] to 6[68,130].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 9 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 2,6,18692.018075
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 11 (12 of 21): 4[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 10 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (8-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 4,7,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 12 (13 of 21): 2[12,43] to 4[68,130].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 11 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 2,4,18692.018075
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 13 (14 of 21): 1[68,130] to 5[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 12 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (8-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,5,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 14 (15 of 21): 2[12,43] to 5[68,130].  206.93 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 13 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 2,5,18692.018075
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 15 (16 of 21): 1[68,130] to 4[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 14 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (8-N)
Mon Sep 11 16:39:16 2017 >> R_eff = 1,4,0.000000
Mon Sep 11 16:39:16 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:16 2017 >> Solving pair 16 (17 of 21): 3[68,130] to 6[68,130].    0.00 Km apart
Mon Sep 11 16:39:16 2017 >> Solution to iteration 15 discarded.
Mon Sep 11 16:39:16 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> R_eff = 3,6,0.000000
Mon Sep 11 16:39:17 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:17 2017 >> Solving pair 17 (18 of 21): 5[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:39:17 2017 >> Solution to iteration 16 discarded.
Mon Sep 11 16:39:17 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> R_eff = 5,7,0.000000
Mon Sep 11 16:39:17 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:17 2017 >> Solving pair 18 (19 of 21): 5[68,130] to 6[68,130].    0.00 Km apart
Mon Sep 11 16:39:17 2017 >> Solution to iteration 17 discarded.
Mon Sep 11 16:39:17 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> R_eff = 5,6,0.000000
Mon Sep 11 16:39:17 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:17 2017 >> Solving pair 19 (20 of 21): 3[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:39:17 2017 >> Solution to iteration 18 discarded.
Mon Sep 11 16:39:17 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> R_eff = 3,7,0.000000
Mon Sep 11 16:39:17 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:17 2017 >> Solving pair 20 (21 of 21): 6[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:39:17 2017 >> Solution to iteration 19 discarded.
Mon Sep 11 16:39:17 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> R_eff = 6,7,0.000000
Mon Sep 11 16:39:17 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:39:17 2017 >> Solution to iteration 0 discarded.
Mon Sep 11 16:39:17 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:39:17 2017 >> Result ./results_1.asc written.

but sometimes randomly fails after x runs. Failure occurs roughly every other run

Mon Sep 11 16:37:49 UTC 2017
Mon Sep 11 16:37:49 2017 >> Effective resistance will be written to R_eff.csv.
Mon Sep 11 16:37:49 2017 >> (rows,cols) = (228,200)
Mon Sep 11 16:37:49 2017 >> Removed 0 islands (0 cells).
Mon Sep 11 16:37:49 2017 >> 7 points in nodes_dummy
Mon Sep 11 16:37:49 2017 >> Max distance: 20000.00 pixels
Mon Sep 11 16:37:49 2017 >> 21 pairs generated.  0 skipped.
Mon Sep 11 16:37:49 2017 >> Number of unknowns: 45600
Mon Sep 11 16:37:49 2017 >> Solving pair 0 (1 of 21): 3[68,130] to 6[68,130].    0.00 Km apart
Mon Sep 11 16:37:49 2017 >> R_eff = 3,6,0.000000
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 1 (2 of 21): 1[68,130] to 3[68,130].    0.00 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 0 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 0.000000e+00 (0-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 1,3,0.000000
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 2 (3 of 21): 2[12,43] to 6[68,130].  206.93 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 1 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 0.000000e+00 (0-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 2,6,18692.018075
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 3 (4 of 21): 3[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 2 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 0.000000e+00 (0-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 3,7,0.000000
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 4 (5 of 21): 3[68,130] to 4[68,130].    0.00 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 3 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 3,4,0.000000
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 5 (6 of 21): 2[12,43] to 4[68,130].  206.93 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 4 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 2,4,18692.018075
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 6 (7 of 21): 2[12,43] to 3[68,130].  206.93 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 5 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 2,3,18692.018075
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> Solving pair 7 (8 of 21): 6[68,130] to 7[68,130].    0.00 Km apart
Mon Sep 11 16:37:49 2017 >> Solution to iteration 6 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:37:49 2017 >> R_eff = 6,7,0.000000
Mon Sep 11 16:37:49 2017 >> Estimated time remaining: 00:00:00
Mon Sep 11 16:37:49 2017 >> 1.000000 > 1.000000; converged.
Mon Sep 11 16:37:49 2017 >> Solution to iteration 7 discarded.
Mon Sep 11 16:37:49 2017 >> convergence-factor = 1.000000e+00 (-2147483648-N)
Mon Sep 11 16:37:49 2017 >> Result ./results_1.asc written.

It's not big deal, it seems occur even rarer if there are no duplicated nodes, and I have found away to work around it (through repeated calls). But it might be of interest.

pbleonard commented 7 years ago

@jmsigner Thank you for pointing this out. I'm not exactly sure why this is happening but my immediate guess is that its related to repeating pairs. Also, it appears your convergence factor values are spurious.

pbleonard commented 6 years ago

@jmsigner Can you confirm that you are using Hypre as your preconditioner? Did your installation of hypre have problems? I have recently seen a similar problem when Petsc defaulted to a different package.