MESA example

[TabernaA]

Dear all, I am trying to connect the EMA workbench framework to my MESA model. I have followed the tutorial for python. Yet, I encounter some errors and I cannot make it run ('Simple model' object is not subscriptable- when running perform_experiment). Is there any chance you could upload an example with a known MESA model (like the MESA version of Wolf Sheep https://github.com/projectmesa/mesa/tree/main/examples/wolf_sheep ). Since I am still a beginner in Python I am pretty sure it is a very stupid mistake that I am doing to not run the model. Thanks in advance A

[quaquel]

are you trying to extend an AbstractModel class, or are you simply wrapping the mesa model run within a python function?

if you have some code to share, I am happy to take a look.

In fact, I have been meaning to write some documentation on how to write your own connectors by using MESA as an example.

[TabernaA]

Hi, thanks for the answer. Actually I only tried to apply the simplest example to my model mesa object: but of course it does not work so I don' really know what to do. An example to see how to build your connector with MESA would be great. Thanks

[steipatr]

Hi Alessandro, Our files from SocSimFest are available here: https://github.com/BROSE-Uninc/SSF2021 and include an example with the EMA Workbench and a Mesa model. Or are you trying to do something else?

@quaquel I can add this as an MVP example for EMA and mesa if you would like, connected just through Python.

[TabernaA]

Hi, thanks, that is great! Is there any chance i could find the recording somewhere? it would be very helpful

[steipatr]

Unfortunately, there were some technical issues on ESSA's side, so the recording is not available.

[TabernaA]

Okay thanks. I managed to connect my model to EMA. Yet it is very slow, I remembered there was an option for multiprocessing, is there any example of how to implement that?

Also, which SA (that in in EMA workbench), would you recommend for a non-ergodic model? (2 final possible equilibria)

Thanks in advance!

[quaquel]

depends a bit on whether your model needs access to the file system and how you made it work with MESA.

If done correctly, all you need to do is

from ema_workbench import MultiprocessingEvaluator

with MultiprocessingEvaluator(model) as evaluator:
    experiments, outcomes = evaluator.perform_experiments(100)

[TabernaA]

Thanks. I actually did adapt my model to be the same as the tutorial:

This is my code:

   def model_EEG_ABM(F1=5,
                 F2 =25,
                 H =350 ,
                 B =1 ,
                 T = 2,
                 S = 0,
                 seed = 4,
                 steps=200):

        import model

        EEG_ABM = model.KSModel(F1=50,
                        F2 =250,
                        H =3500 ,
                        B =1 ,
                        T = 0.02 ,
                        S =0,
                        seed = 4)

        EEG_ABM.run_model(steps)

        outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()

        return {'TIME' : list(range(steps + 1)),
               "Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
               "Real GDP internal" : outcomes["Real GDP internal"].tolist(),
               "Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
               "Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}

Importantly, if i test it like this it works fine:

    F1=50
    F2 =250
    H =3500
    B = 1 
    T = 2 
    S =0
    seed = 4
    steps = 10
    model_EEG_ABM(F1= F1,
                           F2 =F2,
                           H =H ,
                           B = B ,
                           T = T ,
                           S =S ,
                           seed = seed,
                          steps= steps)

Outputs: Out[18]: {'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750], 'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127], 'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21], 'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}

However I get some issues when I run the experiments:

  model = ReplicatorModel('KSModel', function=model_EEG_ABM)

  model.uncertainties = [RealParameter("T", 0.01, 0.1),
                   RealParameter("S", 0.02, 0.15)]

  model.constants = [Constant('F1', F1),
               Constant('F2', F2),
               Constant('H', H),
               Constant('B', B),
               Constant('seed', seed),
               Constant('steps', steps)]

   model.outcomes = [TimeSeriesOutcome('TIME'),
              TimeSeriesOutcome('Real GDP coastal'),
              TimeSeriesOutcome('Unemployment rate coastal')]

   model.replications = 1

results = perform_experiments(model, 10, uncertainty_sampling='sobol')

I get this error:

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 230, in _store_outcomes self.results[outcome][case_id, ] = outcome_res

KeyError: 'TIME'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "", line 25, in results = perform_experiments(model, 10, uncertainty_sampling='sobol')

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 468, in perform_experiments evaluator.evaluate_experiments(scenarios, policies, callback)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 230, in evaluate_experiments callback(experiment, outcomes)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 265, in call self._store_outcomes(experiment.experiment_id, outcomes)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 244, in _store_outcomes self.results[outcome][:] = np.nan

ValueError: cannot convert float NaN to integer

Any clue? Thanks in advance for the support I am almost there.

[quaquel]

which version of the workbench are you using? I recently addressed a bug related to this.

[TabernaA]

I am using 2.0.9

[quaquel]

the problem is presently TIME is a range of integers. If this were a range of floats, the problem would be gone.

more fundamentally, this is a bug that needs fixing on the workbench side.

[TabernaA]

I have changed time in: 'TIME' : [float(i) for i in range(steps +1)] But still I get the same error. I have also tried to remove 'TIME' so using only the other variables but still I get the same error.

Then I changed also the other variables into floats and it works. So the bottom line is: only outcomes with float. thanks!

[quaquel]

and the fact that outcomes need to be floats is indeed the bug that needs fixing

[TabernaA]

I see, perfect! Thanks. I still got problems with parallel runs but I guess is something on my end that I got to fix.

[quaquel]

regarding the parallel running a few questions because the code seems fine.

1. what OS are you using?
2. Are you in a .py file or a notebook?
3. What is the nature of the error?

A fix is being tested for the other bug, so hopefully it will be more permanently fixed in the future.

[TabernaA]

At the bottom of the code I have added:

   from ema_workbench import MultiprocessingEvaluator

   with MultiprocessingEvaluator(model) as evaluator:
   experiments, outcomes = evaluator.perform_experiments(10)

1) I am using macOS Big Sur 11.3.1 2) I am in a .py file 3) The error is that I keep receiving this error in a loop:

[MainProcess/INFO] pool started [MainProcess/INFO] performing 10 scenarios 1 policies 1 model(s) = 10 experiments /usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/optimization.py:48: ImportWarning: platypus based optimization not available

Traceback (most recent call last): File "", line 1, in File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ataberna/Documents/Theoretical ABM/code/Compiler ready to run version/untitled0.py", line 125, in with MultiprocessingEvaluator(model) as evaluator: File "/usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/evaluators.py", line 95, in enter self.initialize() File "/usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/evaluators.py", line 279, in initialize self._pool = multiprocessing.Pool(self.n_processes, initializer, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in init self._repopulate_pool() File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static w.start() File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

And then it starts again... Traceback (most recent call last): File "", line 1, in File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel)

....and so on

[quaquel]

hmmmm.... You are using brew for your python distribution if I am not mistaken. And python 3.9 as well.

I am on a Mac as well, Big Sur also, but anaconda distribution and python 3.8. I have not seen your error before.

[TabernaA]

Interesting. I think it might be something wrong with brew. I have to test Thanks for the help!

[TabernaA]

Hi guys, just a follow-up. I made the general ema workbench working (I had a problem with brew and conda directory mixed up). However I can't make it work with the multiprocessor. It does not give any error yet it nevers finish the run. (it basically says "n experiments running sequentially". Anyone noticed that before (the code is the same as before)? Cheers

[quaquel]

if it says running sequentially, you are not using multiprocessing. Can you try one of the pure python examples that come with the workbench (e.g., the lake problem)?

[TabernaA]

I am sorry, it says 'pool started' not running sequentially. I tried to copy and paste the sobol lake problem, it stayed like this the for 3 hours: (I am running in spyder, in a server that has 40 CPUs so it is usually very fast)

C:\Users\TabernaA\Anaconda3\lib\site-packages\ema_workbench\em_framework\optimization.py:48: ImportWarning: platypus based optimization not available warnings.warn("platypus based optimization not available", ImportWarning) [MainProcess/INFO] pool started [MainProcess/INFO] performing 12000 scenarios 1 policies 1 model(s) = 12000 experiments

[TabernaA]

I closed by mistake sorry

[quaquel]

you really need to provide much more context. For example, are you using notebooks, a normal script, can you show the exact code? Also why not try to make it work on your Mac first.

[TabernaA]

Hey thanks for the quick reply. As I said I am using Spyder, the code is the same as the one I posted before so:

def model_EEG_ABM(F1=5, F2 =25, H =350 , B =1 , T = 2, S = 0, seed = 4, steps=200):

    import model

    EEG_ABM = model.KSModel(F1=50,
                    F2 =250,
                    H =3500 ,
                    B =1 ,
                    T = 0.02 ,
                    S =0,
                    seed = 4)

    EEG_ABM.run_model(steps)

    outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()

    return {'TIME' : list(range(steps + 1)),
           "Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
           "Real GDP internal" : outcomes["Real GDP internal"].tolist(),
           "Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
           "Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}

F1=50
F2 =250
H =3500
B = 1 
T = 2 
S =0
seed = 4
steps = 10
model_EEG_ABM(F1= F1,
                       F2 =F2,
                       H =H ,
                       B = B ,
                       T = T ,
                       S =S ,
                       seed = seed,
                      steps= steps)

Outputs: Out[18]: {'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750], 'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127], 'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21], 'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}

  model = ReplicatorModel('KSModel', function=model_EEG_ABM)
 model.uncertainties = [RealParameter("T", 0.01, 0.1),
               RealParameter("S", 0.02, 0.15)]
 model.constants = [Constant('F1', F1),
           Constant('F2', F2),
           Constant('H', H),
           Constant('B', B),
           Constant('seed', seed),
           Constant('steps', steps)]
   model.outcomes = [TimeSeriesOutcome('TIME'),
          TimeSeriesOutcome('Real GDP coastal'),
          TimeSeriesOutcome('Unemployment rate coastal')]
    model.replications = 1

If I do individual run it works:

 results = perform_experiments(model, 10, uncertainty_sampling='sobol')

While if I do:

 from ema_workbench import MultiprocessingEvaluator  
 with MultiprocessingEvaluator(model) as evaluator:
        evaluator.perform_experiments(100, uncertainty_sampling='sobol'))

I get:

In[4]
with MultiprocessingEvaluator(model) as evaluator:
experiments, outcomes = evaluator.perform_experiments(100, uncertainty_sampling='sobol')
[MainProcess/INFO] pool started
[MainProcess/INFO] performing 600 scenarios * 1 policies * 1 model(s) = 600 experiments

But it stays like this forever, without never finishing...I would like to run in the server because with MESA batch run I can 40 in parallel while in my mac only 12 and the server can run overnight without problems.

Also if I try with the lake model (the sobol one), just copy and paste in a .py file in spyder, I get the same problem.

[quaquel]

please show the full code, not snippets. My hunch is that you miss

if __name__ == '__main__':
    # model definition and run code needs to go here

[EwoutH]

@quaquel I think we should try not to rescope this one again, but get it in 2.5.

If not a tutorial on connectors, just a simple example.

[EwoutH]

Don't have a computer at hand so just writing this thought down: Can we make the EMAworkbench leverage or interface with the Mesa batchrunner?

[quaquel]

I'm unsure what that would mean or why you would want to do that. The batch runner, in my understanding, specifies a full factorial over the provided parameter values. The workbench offers you much more flexibility.