Error running many models in parallel

[algchyhao]

Hi Aerispaha,

I found that when I run SWMM in parallel, it fails randomly. For example, if I run 500 cases in parallel, 0-20 out of 500 may not getting any results and prompt 303 error: not compatible hot start file. The number of failed runs differs every try. What could be the problem please?

[aerispaha]

Hi @algchyhao, apologies for the delay!

I'm not sure why you are having this issue. Possibly this may be due to the fact that EPA SWMM is not thread safe. Currently there are no unit tests for the run_models module which does not help. The plan is to build out the tests for the v0.4.0 release.

However, on a broader degree, I think the scope of swmmio should be more focused on reading, writing, and manipulating SWMM models, rather than being used for running models programmatically. I would suggest that the community leverage pyswmm to run models (indeed maybe swmmio should leverage pyswmm to manage running simulations).

That said, to help us debug, what type of environment are you running your models within?

[bemcdonnell]

@aerispaha, I have modified a version of SWMMIO to use PySWMM. When i try to run a pool of simulations, it seems to default to the hotstart file process. It actually goes and adds 4 [FILES] blocks to my original INP file and looks like this:

I ran: python -m swmmio -sp /Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper

Output:

  ERROR 200: one or more errors in input file.
params to use hotstart2 and not save anything
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
SWMMExtCntrlModel.rpt failed to initialize
local variable 'swmm_version' referenced before assignment
running /Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/SWMMExtCntrlModel.inp

 o  Retrieving project dataTraceback (most recent call last):
  File "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/pyswmm_wrapper.py", line 66, in <module>
    run_model()
  File "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/pyswmm_wrapper.py", line 55, in run_model
    sim = pyswmm.Simulation(inp_file, rpt_file, out_file)
  File "/Users/bryant/PROJECTCODE/swmmio/venv/swmmio_develop/lib/python3.8/site-packages/pyswmm/simulation.py", line 69, in __init__
    self._model.swmm_open()
  File "/Users/bryant/PROJECTCODE/swmmio/venv/swmmio_develop/lib/python3.8/site-packages/pyswmm/swmm5.py", line 217, in swmm_open
    solver.swmm_open(inpfile, rptfile, binfile)
  File "/Users/bryant/PROJECTCODE/swmmio/venv/swmmio_develop/lib/python3.8/site-packages/swmm/toolkit/solver.py", line 509, in swmm_open
    return _solver.swmm_open(f1, f2, f3)
Exception: 
  ERROR 200: one or more errors in input file.
swmmio has completed running 1 models
(swmmio_develop) bryant@Bryants-MacBook-Pro swmmio % 

Original File with added lines:

[FILES]
;;  [FILES]                                                                                    
0   USE HOTSTART "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/SWMMExtCntrlModel_hot2.hsf"

[FILES]
;;  [FILES]                                                                                    
0   USE HOTSTART "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/SWMMExtCntrlModel_hot2.hsf"

[FILES]
;;  [FILES]                                                                                    
0   USE HOTSTART "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/SWMMExtCntrlModel_hot2.hsf"

[FILES]
;;  [FILES]                                                                                    
0   USE HOTSTART "/Users/bryant/PROJECTCODE/swmmio/swmmio/wrapper/SWMMExtCntrlModel_hot2.hsf"

I will submit my PR so you can take a look at my code. I added a simple PY wrapper to replace you execution stuff. It's not completely generic yet for any instance of Python... and it doesn't have unit tests as of this message.... they're to follow soon.

[aerispaha]

@bemcdonnell thanks for looking into this! This is a feature of swmmio that has not been actively maintained for a while, so I'm not surprised that there are some issues.

As I mentioned above in my comment above, I think it makes sense to leverage pyswmm for running models, rather than using a custom approach developed in swmmio. As such, I think we should set up a couple unit tests to ensure we're getting the behavior we want while using pyswmm as the "engine" to run the simulations.

I think we can use swmmio to handle the model creation tasks, and pyswmm to run the simulations. Looking forward to getting this impplemented!