parallel computation / efficient way of copying model

I'm thinking about the best way to make use of the multi-core processor I've on my computer to speed up the model calculations. It seems that modelx is programmed to be single-threaded. Do I understand correctly that it is expected to stay this way? I see the following ways to use more computer resources:

Perform larger matrix operations within the cell. Pandas and numpy array operations benefit from multiple cores. I try to write most of my code this way, but the actual execution of the model doesn't seem use materially more than a single logical processor capacity.
Run multiple scenarios using multiple copies of the model and write code, which will evaluate it in parallel. When running multiple scenarios it is essential to benefit from modelx feature of only evaluating the parts of the model affected by the changed parameter. There are 2 questions associated with this approach:
1. How to efficiently copy the model? An easy, but slow way is to do model.save; mx.open_model() . This method gets slower, when opening model with lots of results in it. Another method is to copy a model: import copy; model2 = copy.deepcopy(model). This method seems to be 5 times faster, but the copy of the model for some reason doesn't work:
```
import modelx as mx
import copy
m = mx.new_model(); mx.new_space()
@mx.defcells
def a():
return 1
```

m2 = copy.deepcopy(m) m2.Space1.a()

Traceback (most recent call last): File "...\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 9, in m2.Space1.a() File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 77, in call return self._impl.get_value(args, kwargs) File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 568, in get_value value = self.system.executor.eval_cell(node) AttributeError: 'CellsImpl' object has no attribute 'system'


2. How to run the copies of the model in parallel? I've tried using `multiprocessing`  module, but so far wasn't successful. I'll keep trying.

I'm thinking about the best way to make use of the multi-core processor I've on my computer to speed up the model calculations. It seems that modelx is programmed to be single-threaded. Do I understand correctly that it is expected to stay this way?

modelx uses 2 threads internally. one is the main thread, the other is for formula evaluation. The use of the other thread is only for getting a larger stack size, as Windows doesn't allow changing the stack size of the main thread. See #7 for this.

modelx does not have built-in parallel computation support, though I want to work on it someday.

I see the following ways to use more computer resources:

Perform larger matrix operations within the cell. Pandas and numpy array operations benefit from multiple cores. I try to write most of my code this way, but the actual execution of the model doesn't seem use materially more than a single logical processor capacity.

Pandas and numpy make use of CPU instruction-level parallelism. They don't automatically make use of multiple cores.

Run multiple scenarios using multiple copies of the model and write code, which will evaluate it in parallel. When running multiple scenarios it is essential to benefit from modelx feature of only evaluating the parts of the model affected by the changed parameter. There are 2 questions associated with this approach:

Due to CPython's notorious GIL, multi-threading in one Python's (CPython's) process doesn't help increasing the speed of processing. You need to consider multi-processing instead.

How to efficiently copy the model? An easy, but slow way is to do model.save; mx.open_model() . This method gets slower, when opening model with lots of results in it. Another method is to copy a model: import copy; model2 = copy.deepcopy(model). This method seems to be 5 times faster, but the copy of the model for some reason doesn't work:
import modelx as mx
import copy
m = mx.new_model(); mx.new_space()
@mx.defcells
def a():
    return 1

m2 = copy.deepcopy(m)
m2.Space1.a()

Traceback (most recent call last):
  File "...\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-f474ed443ec3>", line 9, in <module>
    m2.Space1.a()
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 77, in __call__
    return self._impl.get_value(args, kwargs)
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 568, in get_value
    value = self.system.executor.eval_cell(node)
AttributeError: 'CellsImpl' object has no attribute 'system'
How to run the copies of the model in parallel? I've tried using multiprocessing module, but so far wasn't successful. I'll keep trying.

Having copies of the same model within the same process doesn't solve the problem because of the GIL mentioned above, so the possible way would be to have multiple python processes. If you don't need to exchange data between processes, then you can simply write a code to invoke multiple Python processes and let each Python work on a subset of the whole calculations (such as 1-1000 scenarios on process1, 1001-2000 scenarios on process2, etc.)

If you need to share data between process, the link below is helpful. https://docs.python.org/3.7/library/multiprocessing.html#sharing-state-between-processes

Python 3.8 just introduced https://docs.python.org/3.8/library/multiprocessing.shared_memory.html I haven't read it, but it looks relevant.

I'm thinking about the best way to make use of the multi-core processor I've on my computer to speed up the model calculations. It seems that modelx is programmed to be single-threaded. Do I understand correctly that it is expected to stay this way?

modelx uses 2 threads internally. one is the main thread, the other is for formula evaluation. The use of the other thread is only for getting a larger stack size, as Windows doesn't allow changing the stack size of the main thread. See #7 for this.

modelx does not have built-in parallel computation support, though I want to work on it someday.

I guess the easiest way to do it would be to implement parallel computing for dynamic spaces (only run dynamic spaces, which are a copy of the same space in parallel)

I see the following ways to use more computer resources:

Perform larger matrix operations within the cell. Pandas and numpy array operations benefit from multiple cores. I try to write most of my code this way, but the actual execution of the model doesn't seem use materially more than a single logical processor capacity.

Pandas and numpy make use of CPU instruction-level parallelism. They don't automatically make use of multiple cores.

I suspected something like that. What I don't get is why the following code utilises all cores:

import numpy as np
a=np.random.rand(10000,10000)
a.dot(a)

Run multiple scenarios using multiple copies of the model and write code, which will evaluate it in parallel. When running multiple scenarios it is essential to benefit from modelx feature of only evaluating the parts of the model affected by the changed parameter. There are 2 questions associated with this approach:

Due to CPython's notorious GIL, multi-threading in one Python's (CPython's) process doesn't help increasing the speed of processing. You need to consider multi-processing instead.
How to efficiently copy the model? An easy, but slow way is to do model.save; mx.open_model() . This method gets slower, when opening model with lots of results in it. Another method is to copy a model: import copy; model2 = copy.deepcopy(model). This method seems to be 5 times faster, but the copy of the model for some reason doesn't work:
import modelx as mx
import copy
m = mx.new_model(); mx.new_space()
@mx.defcells
def a():
    return 1

m2 = copy.deepcopy(m)
m2.Space1.a()

Traceback (most recent call last):
  File "...\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-f474ed443ec3>", line 9, in <module>
    m2.Space1.a()
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 77, in __call__
    return self._impl.get_value(args, kwargs)
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 568, in get_value
    value = self.system.executor.eval_cell(node)
AttributeError: 'CellsImpl' object has no attribute 'system'
Could you give an advice about the best way to copy the model, please?

How to run the copies of the model in parallel? I've tried using multiprocessing module, but so far wasn't successful. I'll keep trying.
Having copies of the same model within the same process doesn't solve the problem because of the GIL mentioned above, so the possible way would be to have multiple python processes. If you don't need to exchange data between processes, then you can simply write a code to invoke multiple Python processes and let each Python work on a subset of the whole calculations (such as 1-1000 scenarios on process1, 1001-2000 scenarios on process2, etc.)

I'll try doing this.

If you need to share data between process, the link below is helpful. https://docs.python.org/3.7/library/multiprocessing.html#sharing-state-between-processes

Python 3.8 just introduced https://docs.python.org/3.8/library/multiprocessing.shared_memory.html I haven't read it, but it looks relevant.

Pandas and numpy make use of CPU instruction-level parallelism. They don't automatically make use of multiple cores.

I suspected something like that. What I don't get is why the following code utilises all cores:
import numpy as np
a=np.random.rand(10000,10000)
a.dot(a)
I wasn't accurate. Numpy indeed makes use of multiple cores, thanks to MKL. But this is still a lower (hardware) level parallelism than multi-threading.
How to efficiently copy the model? An easy, but slow way is to do model.save; mx.open_model() . This method gets slower, when opening model with lots of results in it. Another method is to copy a model: import copy; model2 = copy.deepcopy(model). This method seems to be 5 times faster, but the copy of the model for some reason doesn't work:
import modelx as mx
import copy
m = mx.new_model(); mx.new_space()
@mx.defcells
def a():
    return 1

m2 = copy.deepcopy(m)
m2.Space1.a()

Traceback (most recent call last):
  File "...\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-f474ed443ec3>", line 9, in <module>
    m2.Space1.a()
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 77, in __call__
    return self._impl.get_value(args, kwargs)
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 568, in get_value
    value = self.system.executor.eval_cell(node)
AttributeError: 'CellsImpl' object has no attribute 'system'
Could you give an advice about the best way to copy the model, please?

A quick and dirty way of doing this is to see here: https://github.com/fumitoh/modelx/blob/master/modelx/core/model.py#L261-L262. You see pickle.dump is used to write to a file. A similar function pickle.dumps is available , but this returns a byte object. Similary, see here: https://github.com/fumitoh/modelx/blob/master/modelx/core/system.py#L361-L362. pickle.load is used, but pickle.loads is also available. So using pickle.dumps and pickle.loads bypasses writing to a file.

Do you know, why the following code breaks?

import modelx as mx
import copy
m = mx.new_model(); mx.new_space()
@mx.defcells
def a():
    return 1

m2 = copy.deepcopy(m)
m2.Space1.a()

Traceback (most recent call last):
  File "...\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-f474ed443ec3>", line 9, in <module>
    m2.Space1.a()
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 77, in __call__
    return self._impl.get_value(args, kwargs)
  File "...\Anaconda3\lib\site-packages\modelx\core\cells.py", line 568, in get_value
    value = self.system.executor.eval_cell(node)
AttributeError: 'CellsImpl' object has no attribute 'system'

Calling this after deepcopy should solve the error (at least). Pass mx.core.mxsys to system. https://github.com/fumitoh/modelx/blob/00a12f02c458230e10285d592c987c0901c8d398/modelx/core/model.py#L330

fumitoh / modelx

parallel computation / efficient way of copying model #17