Can we have multiple instances of dss_engine?

[rayanelhelou]

Is it possible to create more than one dss_engine object at once without them conflicting with one another? I ask because for the following lines of code (from the examples in your README):

import dss
dss_engine = dss.DSS

dss_engine is not an instance, right? However, in contrast, using the COM interface:

import win32com.client 
dss_engine = win32com.client.gencache.EnsureDispatch("OpenDSSEngine.DSS")

the dss_engine object is indeed an instance, and you can have many of them simultaneously.

Correct me if I'm wrong, but would the solution to creating multiple instances have something to do with generating multiple instances of CffiApiUtil?

I appreciate your time and clarification on this matter.

[PMeira]

Hi @rayanelhelou,

the dss_engine object is indeed an instance, and you can have many of them simultaneously.

With COM, not really. Yes, you can create an multiple instances of the COM interface, but not of the underlying OpenDSS engine. If you try it with something like this:

import win32com.client 
dss_engine = win32com.client.gencache.EnsureDispatch("OpenDSSEngine.DSS")

dss_engine.Text.Command = 'new circuit.test123'
print(dss_engine, dss_engine.ActiveCircuit.Name)

dss_engines = [win32com.client.gencache.EnsureDispatch("OpenDSSEngine.DSS") for _ in range(10)]
for engine in dss_engines:
    print(engine, engine.ActiveCircuit.Name)

You can see that even though each instance has a different address, they point to the same circuit:

<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971853234720> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971853929440> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854123744> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854123840> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854123936> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124032> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124128> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124224> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124320> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124416> test123
<win32com.gen_py.OpenDSS Engine.IDSS instance at 0x2971854124512> test123

(Of course, there is the actor mechanism, but from the API side we're still restricted to a single instance and thread)

Correct me if I'm wrong, but would the solution to creating multiple instances have something to do with generating multiple instances of CffiApiUtil?

Nope, until DSS Python 0.10.x, we had the same limitation of the official COM version (plus the lack of actors). In the official/original OpenDSS codebase, all circuit and element states are kept in global variables. For the parallel-machine from v8+, they're still global but became arrays of things. If you'd like to see a bit more: https://sourceforge.net/p/electricdss/code/HEAD/tree/trunk/Version8/Source/Common/DSSGlobals.pas#l183

But, for DSS Extensions, we implemented a different approach and we're almost ready to release a new version of DSS Python which does support multiple instances, among many other things: https://github.com/dss-extensions/dss_capi/pull/109

It will work like this:

from dss import DSS
DSS.AllowChangeDir = False # when using multiple instances, it's better to forbid the engine to change the working directory of the process
dss_engines = [DSS.NewContext() for _ in range(10)]

Then you can use the engines in threads and so on. But remember that Python is a bit pathetic in terms of threading due to the GIL. Still, if your simulations are not instantaneous, that's fine, since the GIL is released when running a solve.

For backwards compatibility, the base instance (internally called "DSSPrime") always exists, but you don't need to use it after creating the separate contexts.

I can try to prepare a beta release now -- if it doesn't work for some reason, I can do it on the weekend. I'll reply here again to notify you.

By the way, if you're open to share a bit of info on how you intended to use the multiple instances, we can try to provide some comments about that or maybe address it in the documentation.

[PMeira]

@rayanelhelou, beta builds available. If you'd like to test the new context feature:

pip install dss-python==0.12.0b1

To illustrate that they're indeed separate instances:

from dss import dss as prime_engine

prime_engine.AllowChangeDir = False
prime_engine.Text.Command = 'new circuit.test0'
print(prime_engine, prime_engine.ActiveCircuit.Name)

dss_engines = [prime_engine.NewContext() for _ in range(10)]
for i, engine in enumerate(dss_engines, start=1):
    engine.Text.Command = f'new circuit.test{i}'

for engine in dss_engines:
    print(engine, engine.ActiveCircuit.Name)

The parallel-machine API (DSS.ActiveCircuit.Parallel), commands and options should be available (again) too (new actor, solveall, wait, set parallel=..., etc.).

As a bonus, the new ZIP features are also there, from help(DSS.ZIP):

 |  Close(self)
 |      Closes the current open ZIP file
 |      
 |      (API Extension)
 |  
 |  Open(self, Value)
 |      Opens and prepares a ZIP file to be used by the DSS text parser
 |      
 |      (API Extension)
 |  
 |  Redirect(self, Value)
 |      Runs a "Redirect" command inside the current (open) ZIP file.
 |      In the current implementation, all files required by the script must
 |      be present inside the ZIP, using relative paths.
 |      
 |      (API Extension)
 |  

So you can do something like dss.ZIP.Open("somefile.zip"); dss.ZIP.Redirect("path_inside_the_zip/master.dss") to run a script inside a ZIP file directly.

A few more items may be added to the final release. The relevant PR is https://github.com/dss-extensions/dss_python/pull/43

[rayanelhelou]

@PMeira, thank you so much for your prompt and rich reply, and thanks for pointing out that indeed COM is not parallelizable - fortunately, I didn't yet rely on that, but it's good to know!

I really look forward to the DSS.NewContext() feature. Would all usual syntax still work the same, (e.g. .ActiveCircuit, etc.) for this DSS.NewContext() object as it does now for just DSS?

As for anticipated version updates, does it mean I have to somehow update the installation of OpenDSS itself on my computer (windows), or is your package completely separate from the OpenDSS App?

Frankly, I'm holding off before trying the example with 0.12.0b1 that you showed, just because I don't want to mess up the installation I have now, and I'm not exactly sure how to test one without touching the other. Perhaps using another venv, right?

As for the use case, I'm hoping to use this for reinforcement learning purposes wherein it's helpful to have access to multiple instances, each with their own "state". Ideally, off-the-shelf RL packages like stable-baselines3 work better if your environment is parallelizable.

[PMeira]

that indeed COM is not parallelizable - fortunately, I didn't yet rely on that, but it's good to know!

The official version (either through COM or the other DLL alternative) allows some parallelism through the actors, but the number of actors is limited to the number of CPU cores/threads, and you can only activate one circuit for edition at a time. Besides that, if you ever need to use HPC clusters, typically Linux is preferred. The official implementation only supports Windows.

Would all usual syntax still work the same, (e.g. .ActiveCircuit, etc.) for this DSS.NewContext() object as it does now for just DSS?

Yes, exactly the same, only in different instances now. If you don't produce text reports and other files that may clash, you don't even need to change the casenames.

or is your package completely separate from the OpenDSS App?

Completely separate binaries, yes. We try to follow the official implementation closely where it matters (component models) and cross-verify the results for each release, but we changed almost every single file of the original codebase to achieve the new features (especially the separate contexts and the new property system).

Perhaps using another venv, right?

A venv (or a conda env, etc.) should be enough. But the dependencies are very minimal and you can easily switch versions by reinstalling too if you installed it with pip (pip install dss-python==0.12.0b1 to install the beta, and pip install dss-python==0.10.7.post1 to restore the previous version).

off-the-shelf RL packages like stable-baselines3 work better if your environment is parallelizable.

Nice, indeed it seems ideal for that. The new batch API (not exposed to Python in this first beta) might be useful to reduce the Python overhead too. With the batch API, you can select a group of objects, and read or update their properties in bulk (akin to the classic batchedit command, but more versatile and bypasses the text parser).

[rayanelhelou]

Thanks again for clarifying everything - it makes perfect sense! Please feel free to close this issue, as you see fit.

In case I had any questions or suggestions in the future, do you prefer that I raise new issues here or that I contact you elsewhere?

[PMeira]

OK, closing it then.

In case I had any questions or suggestions in the future, do you prefer that I raise new issues here or that I contact you elsewhere?

Up to you, my email is on the README. In the next month or so, we'll probably enable the Discussions tab in the new docs repo (private for now) to consolidate a bit the discussions around all DSS Extensions, while feature requests or bug reports will be kept on the Issues.