Loading Framework and Core based library at the same time.

[kalpeshdlad]

I have few library/module created in .NETFramework which I am using in python by loading via pythonnet. Most of them are 64bit but 1 or 2 is 32bit as there were no other option. I am using msl loadlib to do that using via rpa, between 64bit python to 32 bit library. This is working fine so far.

I have a requirement to upgrade all libraries to latest core base due to increased performance and improvements. At the same time I need to ensure backward compatibility. Given this scenario, I learned that I can load either .NETFramework on Core at a time in a python via pythonnet.

Due to this constraint, I am wandering is there any way I can load Core based 64bit library in RPC mode (just like 32bit). msl loadlib lacks 2 thing to make this solution possible,

1. mslload lib can not start/available 64 bit based server like Server32
2. mslload lib itself works only with .NETFramework only, so I can not load Core as far as I am loading msl loadlib.

Please help me to resolve this problem. or is there any other alternative I can try.

Thanks in advance.

[jborbely]

Perhaps I am focusing too much on one of your comments

mslload lib itself works only with .NETFramework only

to truly grasp the issue, but msl-loadlib will work with any library that pythonnet supports, since pythonnet does all the hard work.

At the top of the module where your Server32 subclass is defined that should use .NET Core to load the library, add the following lines

import os
os.environ['PYTHONNET_RUNTIME'] = 'coreclr'

This will tell pythonnet to load .NET Core (instead of the default .NET Framework) when import clr is used by msl-loadlib. For more details see the pythonnet docs.

Since every instance of the 32-bit server will run in a separate process, each module that runs on the 32-bit server that wraps a 32-bit .NET library can have control over the .NET backend. For example, to force pythonnet to load a library using .NET Framework, you can use

import os
os.environ['PYTHONNET_RUNTIME'] = 'netfx'

[kalpeshdlad]

I have installed these modules : pythonnet ==3.0.3 msl loadlib ==0.9.0

When I try to execute below line on server.py

from pythonnet import load
load("coreclr",runtime_config=r"runtimeconfig.json")
import clr

I getting below mentioned error. This error is not there if I just load Framework

Traceback (most recent call last): File "", line 1, in File "C:\Company\Studio\python39\lib\site-packages\MyLibrary\MyKeywords.py", line 46, in init super(MyKeywords, self).init(module32='my_server') File "studio-installer\Bundles\installation-files\src\scripts\site-packages\msl\loadlib\client64.py", line 199, in init File "studio-installer\Bundles\installation-files\src\scripts\site-packages\msl\loadlib\utils.py", line 282, in wait_for_server msl.loadlib.exceptions.ConnectionTimeoutError: Timeout after 10.0 seconds. Could not connect to 127.0.0.1:59000 [24592] Failed to execute script start_server32 Traceback (most recent call last): File "C:\Company\Studio\python39\lib\site-packages\pythonnet__init__.py", line 77, in _create_runtime_from_spec return clr_loader.get_coreclr(**params) File "C:\Company\Studio\python39\lib\site-packages\clr_loader__init.py", line 117, in get_coreclr from .hostfxr import DotnetCoreRuntime File "C:\Company\Studio\python39\lib\site-packages\clr_loader\hostfxr.py", line 5, in from .ffi import ffi, load_hostfxr File "C:\Company\Studio\python39\lib\site-packages\clr_loader\ffi\init.py", line 11, in ffi = cffi.FFI() # type: ignore File "C:\Company\Studio\python39\lib\site-packages\cffi\api.py", line 48, in init__ import _cffi_backend as backend ModuleNotFoundError: No module named '_cffi_backend'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "start_server32.py", line 223, in File "start_server32.py", line 147, in main File "importlib__init.py", line 127, in import_module File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "C:\Company\Studio\python39\lib\site-packages\extlib\MyLibrary\my_server.py", line 10, in load("coreclr",runtime_config=r"runtimeconfig.json") File "C:\Company\Studio\python39\lib\site-packages\pythonnet__init__.py", line 135, in load set_runtime(runtime, **params) File "C:\Company\Studio\python39\lib\site-packages\pythonnet__init__.py", line 29, in set_runtime runtime = _create_runtime_from_spec(runtime, params) File "C:\Company\Studio\python39\lib\site-packages\pythonnet\init__.py", line 93, in _create_runtime_from_spec ) from exc RuntimeError: Failed to create a .NET runtime (coreclr) using the parameters {'runtime_config': 'runtimeconfig.json'}.

[jborbely]

Try updating msl-loadlib to version 0.10.0

In version 0.9.0 the 32-bit server has pythonnet 2.5.2 embedded, in 0.10.0 it has pythonnet 3.0.1. The version 3.0.3 that you installed would be used by the 64-bit client, not by the server.

[jborbely]

In msl-loadlib 0.9.0 the line

from pythonnet import load

should actually raise ModuleNotFoundError: No module named 'pythonnet'. The reason it does not is that the site-packages directory from the 64-bit client gets appended to sys.path on the server. Therefore, the 32-bit server is trying to use the 64-bit version of pythonnet that you have installed since it cannot find a 32-bit version of pythonnet (via clr_loader -- running import clr works on the server, but that's not what is being importing on that line above).

You can test this by using msl-loadlib 0.9.0 and just before from pythonnet import load is executed call the static Server32.remove_site_packages_64bit() method

from msl.loadlib import Server32
Server32.remove_site_packages_64bit()
from pythonnet import load  # should raise ModuleNotFoundError

or by temporarily uninstalling pythonnet 3.0.3 and then running your script.

[kalpeshdlad]

@jborbely Thanks for the help. I guess knowing embedded version of pythonnet and removing sitepackage did the trick. With few investigation we are able make this work.

Now with this, is there anything we can do for another issue to load 64-bit assemblies in server side. or any other alternatives I can try.

[jborbely]

Do you actually want to have a frozen 64-bit server (which would run in an isolated process) or just mock a server (the client and server are running in the same process, perhaps use the threading or using multiprocessing module)?

Using a frozen server has the following overhead: first the request is serialized using the pickle module by the client, then written to a file, then the request is read from the file by the server, then unpickled, then processed on the server, then the reply is serialized, then written to a file, then read from the file by the client, then unpickled and finally the data is returned.

Creating a frozen 64-bit server would require a few fairly easy changes that I can help you with, but I probably wouldn't include it in a release, so it's a one-off solution for you. This could be done fairly quickly.

I think a mock 64-bit server could be useful in general and I could maybe see it being included in a release, but this would require some thought.

[kalpeshdlad]

What we bundle currently 32-bit is frozen one right? If yes, then I can take that route of 64-bit frozen server. Because mocking(with thread) a server under same process will lead to same problem where I will not be able to load core and .net framework at the same time, correct?

[jborbely]

Yes, when you install msl-loadlib the server you get is frozen.

I do not know if it is possible to load Core and Framework simultaneously. It is best to ask the pythonnet developers.

Here are the instructions for how to create and use a frozen 64-bit server.

1. Create and activate a virtual environment.
2. pip install pyinstaller pythonnet
3. git clone --depth 1 --branch v0.10.0 https://github.com/MSLNZ/msl-loadlib.git
4. cd msl-loadlib\msl\loadlib
5. Open freeze_server32.py in a text editor, delete (or comment) lines 63-65.

   # if loadlib.IS_PYTHON_64BIT:
   #    print('Must run {} using a 32-bit Python interpreter'.format(os.path.basename(__file__)))
   #    return

6. python freeze_server32.py --ignore-comtypes
7. Copy server32-windows.exe and server32-windows.exe.config (only if the library you want to load is from .NET < 4.0) that are located in the current working directory (msl-loadlib\msl\loadlib) to a different directory. Do not rename the files.
8. When you initialize your Client64 subclass, pass in the directory where you chose to copy the 64-bit server in step 7 as a server32_dir keyword argument. Deciding whether to use the 32-bit or the 64-bit server depends on whether you specify a value for server32_dir. It may be confusing that the text server32 also corresponds to a 64-bit server, but that's the way it needs to be for a quick solution.

   client_32lib = YourClient64Subclass("your_module32_value", ...)
   client_64lib = YourClient64Subclass("your_module32_value", server32_dir="directory\from\step\7", ...)

   Your Server32 subclass will also have to handle which DLL file to load. It is probably easiest to check the value of sys.maxsize on the server to decide.