search

KratosMultiphysics / Kratos

Kratos Multiphysics (A.K.A Kratos) is a framework for building parallel multi-disciplinary simulation software. Modularity, extensibility and HPC are the main objectives. Kratos has BSD license and is written in C++ with extensive Python interface.
https://kratosmultiphysics.github.io/Kratos/
Other
1.02k stars 245 forks source link

Python GIL Ownership #5033

Closed roigcarlo closed 4 years ago

roigcarlo commented 5 years ago

Dear all.

During the development of the ExaQUte project, we have been suggested by @spirali (please feel free to correct me if anything I expose here is not accurate) from IT4I to review the policy of Kratos regarding the ownership of the GIL.

@RiccardoRossi and I have been looking for pros and cons about this but our conclusions are incomplete at best.

Introduction

The Gil

Without delving into details, python memory manager has problems, every object in python has an internal counter that needs to be keep track off and the process that does it is not thread safe, which means that only one thread can be in execution at the same time to maintain the coherence. Think of it as our classical problem of writing into a shared variable in parallel.

The mechanism which python uses to ensure that only one thread is in execution at the same time is called GIL (Global interpreter lock).

For more information i think a good and quick article to read is https://realpython.com/python-gil/

The performance problem

As you may imagine, this kills every possible parallelism from python and heavily damages all performance that could be achieve from running into multi-core. Luckily, since python 3 there are some notable exceptions for the GIL in which can be automatically released at certain intervals if no one is using it. This potentially allows some degree of parallelism between computation and I/O for multi-threaded python.

Pybind11, C++, Kratos and the GIL

As you know Kratos is a code that is written mainly on C++ and makes use of python as its scripting language. To do so, we expose a series of classes and functions, some of them very CPU intensive, which can be called from python. The library used for this is called Pybind11.

I think also hat is very important to remark some important points:

Once exposing our C++ code to python, one must chose the policy for the GIL. It could be released (and re-acquired at the end of the function) or it could be kept. the default policy in python11 is to held the GIL for the functions and is what we use right now.

The main implication for this is that basically, two python threads cannot call a C++ function from Kratos at the same time.

It is also worth to mention that most of the codes that have C++ based CPU intensive tasks seem to release the GIL when calling such procedures.

Releasing the GIL

So at the sight of what I exposed ones question could be: Why do we not release the GIL for Kratos procedures? From my side I cannot answer because I exactly don't know what will happen in a couple of situations, and I would like to discuss my list of concerns.

I would ask you please to consider that Kratos is running in a multi-threaded python for the situations below, even if this is not the use case you are used to. I will tackle this point later.

Consider a simple multi-threaded presudo-code in C++


class kratos_object {
public:
    int global_var = 0;

    void do_iter_ser() {
        for(auto in in 1..1000) {
            global_var++;
        }
    }

    void do_iter_par() {
        #pragma omp parallel for
        for(auto in 1..1000) {
            perform_cpu_bound_task();
        }
    }

    void print_me() {std::cout << "Local var: " << global_var << std::endl;}
}

pybind11::module& m;
py::class_<kratos_object, kratos_object* >(m,"kratos_object")
.def("do_iter_ser")
.def("do_iter_par")
.def("print_me")

Multi-threading over non thread safe classes:

Now first consider this code:

Import KratosMultiphysics as KM

def threaded_procedure(kratos_object):
    kratos_object.do_iter_ser()

my_kratos_object = KM.create_object()

thread.start_new_thread ( threaded_procedure, my_kratos_object )
thread.start_new_thread ( threaded_procedure, my_kratos_object )

my_kratos_object.print_me()

What will print_me() print? If the GIL is held, the answer clearly is 2000, but if the GIL is released, does any mechanism ensures that there are not race conditions over global_var?

Multi-threading over already parallel classes:

Consider now a second snippet:

Import KratosMultiphysics as KM

def threaded_procedure(kratos_object):
    kratos_object.do_iter_par()

my_kratos_object = KM.create_object()

for _ in range(0, number_of_cores)
    thread.start_new_thread ( threaded_procedure, my_kratos_object )

While running on a machine with 4 physical cores (OMP_NUM_THREADS=4), and being the C++ loop bound by CPU, not releasing the GIL will ensure that that task will run 4 times, with 4 threads every time.

What would happen with the GIL released? are 16 threads trying to run at the same time? only 4 but switching context?

I would like to remark that @spirali said that pinning is possible with Loom, but does this can be generalized?

Running python code in C++ classes

As we prohibit this it is not relevant, anyway, the answer is clear here. Routines with the GIL not released would not have problems, routines with the GIL released would have to temporally lock it to run the code.

Use Cases

Finally, concerning the use of multithreading from Kratos. Should we allow it in the first place? I understand that this could be a main topic while speaking of libraries with Django for example, but is Kratos aiming to be run in multi-threaded python codes?

Conlusion and Opinion

I hope I exposed correctly all the problems and concerns with the GIL and what is the status of Kratos today.

Answering my own questions (this is my opinion: 1 - I can't see there anything that prevents race conditions while multithreading from python, and while we could take the effort to go class by class to release the GIL on those who do not have problems, in general is not worth the effort.

2 - While there are platforms that would ensure correct threading pinning in CPU, i am not expert in this topic and I would not make any conclusion, but my feeling is that in general this will not be the situation.

3 - Finally I will admit that I could be beneficial to be able to run in threaded environment if we aim to use kratos as a web-service for example, but I also think that there are other ways to achieve the same goal and currently I don't see any benefit on allowing or incentive this possibility.

As I said @RiccardoRossi and I would really be glad if you could bring your points of view here.

RiccardoRossi commented 5 years ago

i jist would like yo add that there is one exception to python not being called from c++:

we actually use tjat to allow the user to provide one function from the input. i did try to use tjst in parallel in c++ And it was segfaulting

RiccardoRossi commented 5 years ago

i also personally like the async (concurrent futures) approach to parallelism, although it implies using different memory spaces.

spirali commented 5 years ago

My point is that if your library is not thread-safe than declare it or introduce your own lock. Do not abuse GIL. Your are preventing behavior like using Kratos in GUI app, asynchronous logging, or even using it with async Python that Riccardo mentions. Moreover you gain absolutely nothing for such approach. Moreover, it is surprising behavior of Python module that binds C/C++ library, so I strongly suggest to document this behavior as I am not aware of any other library that do the same.

(But is it really true that you are fully not-thread-safe? Is it still undefined behavior to run Kratos methods on completely separated instances?)

philbucher commented 5 years ago

A few thought from my side, since I dealt with exactly this only a few days ago with a different code (same structure as Kratos; C++ core & python interface). I think this is a really cool feature but I have some concerns:

RiccardoRossi commented 5 years ago

@philbucher i share all of your concerns...

@spirali the big problem here is that kratos is not a single function exported to python, but rather thousands of small functions. all of these fucctions can be run in parallel as long as it is guaranteed that the memory they act on is separate. for example there should be absolutely no problem in using the "concurrency" module of python (any async thread has a different GIL) so in a sense they are like separate programs.

what is absolutely NOT threadsafe is to have tho python thrrads to touch concurrently kratos internals) which to my understanding is what the GIL attempts to prevent

spirali commented 5 years ago

@RiccardoRossi Again, if your library is not thread-safe than declare it or introduce your own lock. Do not abuse GIL.

GIL is not here to make your program threads safe. Use your own mutex if you must.

Btw: You are still mixing several terms together. There is no module named "concurrency" in Python. There is "threading" module which will not work. There is "multiprocessing" which will work. And there is "asyncio" wich will not work. So sentence "any async thread has a different GIL" is a total nonsense. Multiprocessing does not use threads, threads does not have own GIL, and async use own single thread event loop.

But all this is irrelevant to fact that your are misusing the GIL for your own thread safety. Use your own lock if you must.

RiccardoRossi commented 5 years ago

dear @spirali i am afraid i do not agree with you. the GIL is there EXACTLY not to make me define and use my own locking mechanism.

this is also the reason for which BOTH pybind11 and boost.python use it the same way.

having said this, this thread is about discussing if (provided it is doable) we should or not release tje GIL on each function call. i still did not hear of any fundamental reason for doing it.

the module i was speaking about is called "concurrent.futures"

https://docs.python.org/3/library/concurrent.futures.html

the idea is that each async task (process? does this term satisfy you?) has its own gil, do that they can run concurrently. that is also mentioned in the page @roigcarlo pointed to. as far as i understand this shall also be happening for pycompss (to be verified)

Kobzol commented 5 years ago

I think that what @spirali is trying to say is that the GIL is being misused in Kratos to provide thread-safety. While it works because of the way you use Kratos, it has several disadvantages.

As far as I understand the pybind11 behaviour, it locks the GIL when entering C++ from Python, but it should be possible to release it during long-running C++ operations (which is what the documentation gives as an example - https://pybind11.readthedocs.io/en/master/advanced/misc.html#global-interpreter-lock-gil).

spirali commented 5 years ago

@RiccardoRossi No, GIL is not here to supply locking mechanism for your module. GIL is here to prevent two threads going into Python interpreter. They are two different things and mixing them have only drawbacks.

I obviously failed to deliver a message and so far it seems that it is not a blocking issue for Exaqute, I am not going to elaborate more on this.

RiccardoRossi commented 5 years ago

@kobzol i think i am expressing myself badly. it would be possible to run kratos from c++ alone (we do ot in our tests for example), nevertheless the kind of SMP parallelism we use is based on openmp (or mpi).

kratos is not threadsafe in the sense that if you divide our main into threads acting on the same data you ll definitely break it.

it would nevertheless be possible to asyncronously launch some c++ tasks, as long as we do it wisely. (for example accessing to different data). so far this was not how we extracted parallelism since our way was to have a single "serial" main, and to express all parlalelosm (with all available threads).

to my understanding the GIL is essentially obliging our main to run serially. this imploes that if we use tje threadpoolexecutor or directly python threads we have effectively no parallelism. on tje contrary if we were to ise the the "processpoolexecutor" of the conccurent fitures module this should work flawlessily.

the point for discussion here is if we should or not rrlax the serial constraint on running the main.

the big problem here is that kratos is not a single function call to c++ which we could make non blocking but rather a sequence of thousands of calls to smaller functions. pybind11 (the mechanism we use for our python interface) woukd allow is to make the calls non blocking nevertheless we would have to do that per each of the functions in our API (and we have many thousnads) one problem here is that i am not clear if we could do that for functions that accept as input or give as output new python objects.

the only feature i see as completely unsafe in this context is the openmp.SetMaxThreads method (or whatever is the name we gove to it). this is potentially a big issue since i am not clear of how we would jave thread parallelism and openmp parallelosm to interact.

anyway...i ll add this thread to the TODOs of the technicalcommittee

pooyan-dadvand commented 5 years ago

@spirali and @Kobzol, I understand your concern about using GIL and how it can affect on freezing a GUI which runs some functions of Kratos in the background. What I don't understand (sorry for my ignorance as I am mainly a C++ programmer) is what we can do on that respect. Considering the following sudo code in python:

Import KratosMultiphysics as kratos

model = kratos.Model() # please note that this is common for entire process
model_part = model.CreateModelPart("test")
io = kratos.ModelPartIO("input_file")
io.ReadModelPart(model_part) 
import SomeSolver
solver = SomeSolver.Create(model_part) # this should be done after model part is read
solver.solve() # we cannot run this before solver is created

I understand that the order of following code would be the same if we release the GIL because python thread this script as one interpretation thread. But if some other thread calls a similar script for other model_part the creation in model would fail because is not thread safe in C++. Isn't it? The situation become worse if we have another solver running on the same model_part. Then it would really a pain to deal with concurrency in an efficient way. Repeating, the efficiency here is the key point.

Do you have any suggestion on that?

Just to note that the data structure of Kratos is rather complex and imposing thread safety to it is not only complex but also make it really in efficient. For instance in OpenMp we made some part of the code thread safe as long as the performance impact is small and putting debug checks in the other parts which tells you about possible concurrency. of course I am open to suggestions.

Kobzol commented 5 years ago

Do I understand it correctly that model = kratos.Model() cannot be executed by multiple threads, i.e. the model is basically a singleton for the entire process? In that case you couldn't run more than one Kratos task in the same process anyway, so our use case with multiple Kratos tasks running concurrently doesn't hold.

Even then, it would be nice if there was an internal lock instead of misusing the GIL, but it's not so important for HyperLoom if Kratos is not thread-safe. If Kratos cannot run concurrently, you could still release the GIL and allow other Python threads to run concurrently (i.e. other HyperLoom tasks on the same worker). Again, there might not be a use case for this at this moment.

RiccardoRossi commented 5 years ago

model is not a singleton. we could eventually have one model per thread (killed when the thread finishes)...same as one stratrgy, etc.

however the only sane approach to this would be that threads share nothing ...like independent processes

the usecase for your use of threads is more to allow some interactivity (for example plotting something while the kratos runs).

this is possible even today, however the two threads would be serialized

Riccardo

On Wed, Jun 5, 2019, 1:16 PM Jakub Beránek notifications@github.com wrote:

Do I understand it correctly that model = kratos.Model() cannot be executed by multiple threads, i.e. the model is basically a singleton for the entire process? In that case you couldn't run more than one Kratos task in the same process anyway, so our use case with multiple Kratos tasks running concurrently doesn't hold.

Even then, it would be nice if there was an internal lock instead of misusing the GIL, but it's not so important for HyperLoom if Kratos is not thread-safe. If Kratos cannot run concurrently, you could still release the GIL and allow other Python threads to run concurrently (i.e. other HyperLoom tasks on the same worker). Again, there might not be a use case for this at this moment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/KratosMultiphysics/Kratos/issues/5033?email_source=notifications&email_token=AB5PWEOLB6QWGTZD42B7UODPY6OAPA5CNFSM4HS234F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW7MPYQ#issuecomment-499042274, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5PWEIKYGEYQ4SURIHPISTPY6OAPANCNFSM4HS234FQ .

pooyan-dadvand commented 5 years ago

Actually the case of Model and ModelPart is the easiest one as there are only two classes in the core in which we can tune it more or less well. The one of the solver is rather complex because it may call different elements written by other developers accessing to many data.

Meanwhile, about putting a lock, we have already OpenMP locks in may places and we don't want the intrusive use of python or pybind11 in our code for python locks. When you are talking about locks are you talking about POSIX compatible threads? or we should import the Python ones?

Kobzol commented 5 years ago

So in HyperLoom the tasks on a single worker run inside a single process, therefore if we wanted to have multiple independent share-nothing Kratos tasks, they would be different threads, not different processes. This is not possible due to the way Kratos uses the GIL (even though otherwise it could work if Kratos supports it).

If you wanted to have Kratos Python-independent, it should have it's own locking using whatever locks you want (for example pthread mutexes). Our whole point was to replace the GIL lock with C++ locks, so definitely not importing any additional Python locks inside the C++ code.

pooyan-dadvand commented 5 years ago

Thanks @Kobzol! it is clear now. As @RiccardoRossi mentioned it is already in the @KratosMultiphysics/technical-committee todo list. I also ping @KratosMultiphysics/design-committee and @KratosMultiphysics/implementation-committee

loumalouomega commented 5 years ago

Added to @KratosMultiphysics/implementation-committee TODO @pooyan-dadvand

ramonamela commented 5 years ago

Hi all, I would like to only add a couple of things to the discussion concerning the impact that the decision you adopt here will have on exaQUte project.

First of all, I agree with @RiccardoRossi when he says that GIL is the way to go to ensure thread-safety. Indeed, this is why multiprocessing is there. Since there is this 'memory sharing' problem, in the last python they have added a new functionality regarding this: https://docs.python.org/3.8/whatsnew/3.8.html

multiprocessing Added new multiprocessing.shared_memory module. (Contributed Davin Potts in bpo-35813.)

As far as I know, the intended and more general use of multi-threading is I/O operations while the use of multi-processing is encouraged for computational operations (which is the case here). Indeed, all the Python developments are going in this line.

This is, in fact, fully coherent with what @Kobzol has said. I agree that this should be said in the Kratos documentation but at the end they are just doing more or less standard things, so it was predictable. On the other side, the behavior @RiccardoRossi was describing of "concurrent.futures" is the one corresponding to "processpoolexecutor" as he pointed out in a later post. The point here is that, under my point of view, you are doing an advanced non really heterodox use of the Python interpreter.

I said all this because we have done some tests on how to free the GIL and when doing this we have had lots of problems. Hence, and since we already run with multiprocessing, releasing the GIL is not a big deal for us since we run in sequential in the worker process. Not having shared memory is a pain in the ass, yes, but at least the executions are quite robust (and apparently this will be solved pretty soon).

Finally, there is the OpenMP problem. There is an OpenMP runtime per process. In this project we plan to execute heterogeneous tasks. For me the example given by @roigcarlo in the first post is perfect. 16 or 4? I would appreciate you opinion regarding this, because in our case (running in different processes, which should be much more easy) I don't see an easy way to proceed to, for example, run a task with 4 OpenMP threads and and other in the same node with 8 OpenMP threads. It is impossible to control at which moment the threads are opened (so when the OpenMP runtime will check for the environmental variable). And it is clear that the global value cannot be 4 and 8 at the same time. How do you plan to proceed with this in a single program case?

Kobzol commented 5 years ago

You're right that for compute-bound tasks, having more Python threads running concurrently doesn't help the performance scaling, we don't argue that. But if the tasks don't hold the GIL, the interpreter can switch to bookkeeping, I/O, and other potential Python computation tasks that should be executed once in a while. If a Kratos task holds the GIL for 5 minutes, then during those 5 minutes there could be no additional Python tasks running, pretty much anything concerning Python would be blocked.

The OpenMP problem is a bit more subtle. This is indeed a problem, either way you can run too many OpenMP threads and oversubscribe your node and it doesn't really matter whether you use processes or threads. What is different though is the behaviour of OMP_NUM_THREADS. AFAIK, the threadpool is global for the whole process, which is problematic if you run more threads with Kratos tasks in parallel, as you say. You can of course control the number of launched threads per parallel region, but that is not very useful if you do not know how many tasks will run in parallel. You can partially solve this using OMP_DYNAMIC (`https://docs.microsoft.com/en-us/cpp/parallel/openmp/reference/openmp-environment-variables?view=vs-2019#omp-dynamic), but that is not a silver bullet.

If you want to avoid the thread conflicts, then you should use a global lock for Kratos, so that no two Kratos tasks could be launched in parallel. Now we get to the core of the issue that @spirali was talking about. Obviously the current behaviour of Kratos is what you want (i.e. a global lock on entry points). We don't have an issue with that, that is perfectly fine. However we argue that you should use your own lock for this and don't misuse the Python GIL. In this way you would: 1) Get the same behaviour as you get now 2) Make Kratos impossible to use incorrectly (regarding thread-safety) from all other scenarios, not just from Python 3) Enable other Python threads to run while a Kratos function is executing when called from Python

Again, to reiterate, our point is not to change the threading behaviour of Kratos, our point is that the GIL should not be used to enable thread-safety of Kratos, as that is not its intended usage.

philbucher commented 5 years ago

@Kobzol do you have a link to the documentation on how to use custom locks? I got a bit lost now. Thanks

Kobzol commented 5 years ago

You can use pretty much whatever you want, pthread mutexes, std::mutex or you can even lock from Python directly.

spirali commented 5 years ago

I am reading this discussion only occasionally and not carefully. So I would like to just correct Kobzol. HyperLoom core works even you are holding GIL (worker core is not written in Python), so worker communicate with server, etc. What is blocked is deserialization of Python objects and run of Python task in worker itself.

However, if you would like to use Dask/distributed for parallelization (another tool with task graphs, that has probably more users than HyperLoom and PyCOMPs together), holding GIL would be serious problem, since you completely block Dask worker. Of course you can always move a task to another process, but it is not for free (performance lost for extra serialization and constantly moving objects, and also introducing another kind of problems connected with running multiprocessing itself).

ramonamela commented 5 years ago

You're right that for compute-bound tasks, having more Python threads running concurrently doesn't help the performance scaling, we don't argue that. But if the tasks don't hold the GIL, the interpreter can switch to bookkeeping, I/O, and other potential Python computation tasks that should be executed once in a while. If a Kratos task holds the GIL for 5 minutes, then during those 5 minutes there could be no additional Python tasks running, pretty much anything concerning Python would be blocked.

The OpenMP problem is a bit more subtle. This is indeed a problem, either way you can run too many OpenMP threads and oversubscribe your node and it doesn't really matter whether you use processes or threads. What is different though is the behaviour of OMP_NUM_THREADS. AFAIK, the threadpool is global for the whole process, which is problematic if you run more threads with Kratos tasks in parallel, as you say. You can of course control the number of launched threads per parallel region, but that is not very useful if you do not know how many tasks will run in parallel. You can partially solve this using OMP_DYNAMIC (`https://docs.microsoft.com/en-us/cpp/parallel/openmp/reference/openmp-environment-variables?view=vs-2019#omp-dynamic), but that is not a silver bullet.

If you want to avoid the thread conflicts, then you should use a global lock for Kratos, so that no two Kratos tasks could be launched in parallel. Now we get to the core of the issue that @spirali was talking about. Obviously the current behaviour of Kratos is what you want (i.e. a global lock on entry points). We don't have an issue with that, that is perfectly fine. However we argue that you should use your own lock for this and don't misuse the Python GIL. In this way you would:

  1. Get the same behaviour as you get now
  2. Make Kratos impossible to use incorrectly (regarding thread-safety) from all other scenarios, not just from Python
  3. Enable other Python threads to run while a Kratos function is executing when called from Python

Again, to reiterate, our point is not to change the threading behaviour of Kratos, our point is that the GIL should not be used to enable thread-safety of Kratos, as that is not its intended usage.

Only for the OpenMP part and considering that, in fact, this will be the common case. Several Kratos instances running in the same node. I get you point with OMP_DYNAMIC for, let's say, omp for or regions where you don't use explicitly the amount of threads. What do you do if, for example, you consider that each thread access to a part of an array and, in order to compute the regions you need to access to, you check the maximum_omp_threads. Once you start the computation, this maximum changes. I think we can end up easily with a segfault or a non expected behavior. What I had in mind is to change this dynamically between executions but guarantee that, during a given task Kratos task execution, this value does not change. For me it was already a great challenge how to change the amount of threads in the OpenMP runtime in 'live' between one task execution an another. Hence, and as I understand it, even the OMP_DYNAMIC does not seem to fit well in this use case.

Kobzol commented 5 years ago

Sure, that may not a solution for your usecase. Nevertheless, introducing a global lock would keep the current behaviour.

ramonamela commented 5 years ago

My point is that, if we want to have two Kratos executing at the same time in the same node in whatever way you can agree to, they should be in two different processes if we don't want to have serious problems with OpenMP. I'd love some confirmation by someone on the Kratos team regarding this because maybe I'm being too conservative.

roigcarlo commented 5 years ago

@ramonamela Is basically that. If we don't run in different memory spaces we will have undefined behavior.

Kobzol commented 5 years ago

Ok, that is orthogonal to GIL usage though. I guess that there's not much else to discuss here, for Exaqute HyperLoom can be modified to support multiprocessing and we already presented our arguments why we think that GIL is a bad idea. The Kratos technical committee can decide whether you want to replace the GIL with your own lock or not.

RiccardoRossi commented 5 years ago

Just to summarize my current understanding of the discussion:

to add a personal point of view, i should say that we do not have one single function keeping computations blocked for very long periods (well, strategy.SolveSolutionStep maybe...).

as of now i am thinking that we could release the GIL for very selected functions, for example IO, serialization and "strategy". To be discussed though...not sure if this would actually work

if you see something important missing let me know and i'll update this post

Kobzol commented 5 years ago

I don't think that CPython 3.8 (or any other future CPython version) could solve the current situation - I still think that we have a slight misunderstanding. We didn't suggest you to rewrite Kratos, change its thread-safety, change how it behaves with respect to OpenMP or anything like that. What we suggested was simply to unlock the GIL and lock your own global lock instead, literally something like this:

releaseGIL();
lockGlobalKratosLock();
callLongCPPFunction();
unlockGlobalKratosLock();
takeGIL();

Therefore the behaviour of Kratos would stay exactly the same, it would just unblock other Python threads (which would have no effect on Kratos itself, even if there were other concurrent Kratos tasks).

But as I stated earlier, HyperLoom should be able to deal with the current situation and this would probably require changes to most Kratos entry points, so if you do not consider this critical, just keep the current state.

pooyan-dadvand commented 5 years ago

@KratosMultiphysics/technical-committee is currently oriented to:

  1. Allow releasing the GIL for some specific methods (like SolveSolutionStep)
  2. Implement a cpp leve locking mechanism to ensure no concurrence instances of such methods in a second step
  3. There is option for having mutex per category like Processes, Solvers, etc.

It is not consider as a high priority right now but this comment is intended as roadmap.

roigcarlo commented 4 years ago

We understand that this is needed for the the future but we will reopen we are about to implement it.