conflict on Windows with ca.dll / Com.dll from wheel installer with pyepics

[newville]

We've had reports that there is a conflict on Windows between pyepics and pcaspy - see https://github.com/pyepics/pyepics/issues/176

It looks to me like import pcaspy causes the _cas.pyd to be loaded -- at import time. It also looks like this pyd expects ca.dll and Com.dll to be found in the same folder, or at least to be "loadable" from somewhere.

This loading can cause conflicts with pyepics as it too needs to load ca.dll and Com.dll and comes with its own set of libraries. If the two sets of libraries are not of the same generation, I believe the loading of the second one will fail.

As it turns out, pyepics does not load ca.dll at import time, but waits until as late as possible (the first real call that actually needs to do CA communication or epics.ca.initialize_libca()) to load the libraries.

I'm not sure the best way to avoid these conflicts. One possibility would be for pcaspy to import and initialize libca from pyepics. Then those libraries would be loaded and you would not have to package ca.dll and Com.dll into the Windows wheel. Another possibility would be to set PYEPICS_LIBCA environment variable to point to a single ca.dll. That is, if pcaspy loads ca.dll through _cas.pyd it could set that environmental variable to point to the ca.dll it imported. Then a subsequent epics.ca.initialize_libca() would use the version specified with PYEPICS_LIBCA.

I'm sure there are other options. Do you have any suggestions for how to avoid such conflicts?

[xiaoqiangwang]

That is a tricky problem. Because the PyPI package cannot assume the existence of epics base libraries.

I used to build with epics static libraries on Windows. But ever since I use the anaconda environment, I simply copied the dlls. In this I don't have to build epics base for the various MSVC version used by Python.

Might be for PyPI package I have to build against static epics libraries.

[newville]

@xiaoqiangwang For pyepics, we include ca/Com shared libs for all platforms. These are placed within the epics python package so as to avoid conflicts with system-installed versions. Pyepics definitely needs to load these directly as dlls -- they cannot be statically linked.

To be clear, this dynamic loading with ctypes works fine for pyepics: I think this is one of the first problems we've run into with Windows. (FWIW, there are people who complain about us supplying default shared libraries on Linux, but we do test these and I do not know that there are actual conflicts). Anyway, one can set PYEPICS_LIBCA to point to the full path of libca that should be used.

But I think it will only ever work to load one library called ca.dll in any application. So, if someone wants to use pcaspy and pyepics in the same program and if those are using different versions of the shared libraries, I believe there will be a conflict.

Do you supply libca for MacOS and Linux as well or do you rely on the user having installed epics base?

[xiaoqiangwang]

On MacOS and Linux, epics base creates static libraries by default and I always link to them.

On Windows, because I use anaconda as the building environment, I have to first change the epics-base recipe to build win32-x86-static and windows-x64-static variants.

For now, if the pyepics users want to use pcaspy, they have to inconveniently point PYEPICS_LIBCA to those dlls inside the pcaspy package. I assume the following hack during the import could work,

import os
import pcaspy
os.environ['PYEPICS_LIBCA'] = os.path.dirname(pcaspy.__file__)
import epics

[ghost]

@xiaoqiangwang thanks for the hack, for me it had to be this line though (pyepics expects a full path to the dll file):

os.environ['PYEPICS_LIBCA'] = os.path.join(os.path.dirname(pcaspy.__file__), "ca.dll")

[vstadnytskyi]

Having exactly the same problem. The fix here is not working for me. I am getting an error in importing pscpy

Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\pcaspy\cas.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "C:\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Z:\All Projects\LaserLab\Software\caproto_sandbox\random_walk_gui.py", line 5, in <module>
    import pcaspy
  File "C:\Python37\lib\site-packages\pcaspy\__init__.py", line 1, in <module>
    from .driver import Driver, SimpleServer, PVInfo, SimplePV
  File "C:\Python37\lib\site-packages\pcaspy\driver.py", line 1, in <module>
    from . import cas
  File "C:\Python37\lib\site-packages\pcaspy\cas.py", line 17, in <module>
    _cas = swig_import_helper()
  File "C:\Python37\lib\site-packages\pcaspy\cas.py", line 16, in swig_import_helper
    return importlib.import_module('_cas')
  File "C:\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_cas'

I have installed a new version of pcaspy just minutes ago

C:\Users\username>pip3 install pcaspy
Collecting pcaspy
  Downloading https://files.pythonhosted.org/packages/2c/fd/cb7390a6585710ee1bd4
2d241c9e852a6e8cc54fd62015bcb9511b00eb0b/pcaspy-0.7.2-cp37-cp37m-win32.whl (556k
B)
    100% |████████████████████████████████| 563kB 10.2MB/s
Installing collected packages: pcaspy
Successfully installed pcaspy-0.7.2

[newville]

@vstadnytskyi What makes you conclude this is the same problem? You did not provide a script that shows the problem.

As described above, if you are mixing pcaspy and pyepics, you must import pcaspy first and then set PYEPICS_LIBCA to point to the same LIBCA before using pyepics. This is because pcaspy loads LIBCA on import and only one version of a dynamic library can be loaded in an application.

[vstadnytskyi]

@vstadnytskyi What makes you conclude this is the same problem? You did not provide a script that shows the problem.

As described above, if you are mixing pcaspy and pyepics, you must import pcaspy first and then set PYEPICS_LIBCA to point to the same LIBCA before using pyepics. This is because pcaspy loads LIBCA on import and only one version of a dynamic library can be loaded in an application.

the traceback shows that line 5 failed. Repeating the traceback for your convenience.

OS: Windows 7

File "Z:\All Projects\LaserLab\Software\caproto_sandbox\random_walk_gui.py", line 5, in <module>
    import pcaspy

Here is the code.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os
import pcaspy
os.environ['PYEPICS_LIBCA'] = os.path.join(os.path.dirname(pcaspy.__file__), "ca.dll")

import wx
import epics
import epics.wx
from logging import debug,warn,info,error

import wx

__version__ = "0.0.0" #initial

class PanelTemplate(wx.Frame):

        title = "GUI Panel Template"

        def __init__(self):
            wx.Frame.__init__(self, None, wx.ID_ANY, title=self.title, style=wx.DEFAULT_FRAME_STYLE)
            self.panel=wx.Panel(self, -1, size = (200,75))
            self.Bind(wx.EVT_CLOSE, self.OnQuit)

            self.initialize_GUI()
            self.SetBackgroundColour(wx.Colour(255,255,255))
            self.Centre()
            self.Show()

        def OnQuit(self,event):
            """
            orderly exit of Panel if close button is pressed
            """
            self.Destroy()
            del self

        def initialize_GUI(self):
            """
            """
            sizer = wx.GridBagSizer(hgap = 5, vgap = 5)
            self.label ={}
            self.field = {}
            self.sizer = {}
            main_sizer = wx.BoxSizer(wx.VERTICAL)
            topSizer = wx.BoxSizer(wx.VERTICAL)

            self.sizer[b'x'] = wx.BoxSizer(wx.HORIZONTAL)
            self.label[b'x'] = wx.StaticText(self.panel, label= 'X value:', style = wx.ALIGN_CENTER)
            self.field[b'x'] = epics.wx.PVText(self.panel, pv='random_walk:x',minor_alarm = wx.Colour(5, 6, 7),auto_units = True)
            self.sizer[b'x'].Add(self.label[b'x'] , 0)
            self.sizer[b'x'].Add(self.field[b'x'] , 0)

            self.sizer[b't'] = wx.BoxSizer(wx.VERTICAL)
            self.label[b't'] = wx.StaticText(self.panel, label= 'Time of last update:', style = wx.ALIGN_CENTER)
            self.field[b't'] = epics.wx.PVText(self.panel, pv='random_walk:t',minor_alarm = wx.Colour(5, 6, 7),auto_units = True)
            self.sizer[b't'].Add(self.label[b't'] , 0)
            self.sizer[b't'].Add(self.field[b't'] , 0)

            main_sizer.Add(self.sizer[b'x'],0)
            main_sizer.Add(self.sizer[b't'],0)

            self.Center()
            self.Show()
            topSizer.Add(main_sizer,0)

            self.panel.SetSizer(topSizer)
            topSizer.Fit(self)
            self.Layout()
            self.panel.Layout()
            self.panel.Fit()
            self.Fit()

if __name__ == '__main__':
    from pdb import pm
    import logging
    from tempfile import gettempdir

    app = wx.App(redirect=False)
    panel = PanelTemplate()

    app.MainLoop()

[newville]

@vstadnytskyi But the traceback you posted says that the problem happens at line 5 which is

import pcaspy

and is before you import pyepics at all. I think that means what you are seeing is not related to this problem, but just that import pcaspy isn't working, probably because it cannot find the right dll.

[vstadnytskyi]

@vstadnytskyi But the traceback you posted says that the problem happens at line 5 which is

import pcaspy

and is before you import pyepics at all. I think that means what you are seeing is not related to this problem, but just that import pcaspy isn't working, probably because it cannot find the right dll.

That is why I am reporting it here instead of pyepics. I got the same error(reported in pyepics 176 ) before I tried the solution posted in this issues. I will update the issue tomorrow with the exact error I get while running the code without the fix. I will investigate more and report shortly.

[newville]

@vstadnytskyi but your problem really is not the same issue. It has nothing to do with pyepics: you get the error before you even try to import pyepics.

What you are seeing is that you cannot import pcaspy.

[xiaoqiangwang]

@newville is right. The exception is before the import of pyepics. Can you list the contents of C:\Python37\lib\site-packages\pcaspy?

Here is what I get after pip install pcaspy pyepics. And I don't see exceptions.

Directory of C:\miniconda-x86\Lib\site-packages\pcaspy

04.10.2019  08:46             1'286 alarm.py
04.10.2019  08:46            49'664 asIoc.dll
04.10.2019  08:46           271'360 ca.dll
04.10.2019  08:46           176'128 cas.dll
04.10.2019  08:46            26'972 cas.py
04.10.2019  08:46           273'408 Com.dll
04.10.2019  08:46           190'976 dbIoc.dll
04.10.2019  08:46            95'744 dbStaticIoc.dll
04.10.2019  08:46            31'379 driver.py
04.10.2019  08:46           161'280 gdd.dll
04.10.2019  08:46               858 tools.py
04.10.2019  08:46           181'248 _cas.cp37-win32.pyd
04.10.2019  08:46                74 _version.py
04.10.2019  08:46               142 __init__.py
04.10.2019  08:46    <DIR>          __pycache__

[vstadnytskyi]

@newville is right. The exception is before the import of pyepics. Can you list the contents of C:\Python37\lib\site-packages\pcaspy?

Here is what I get after pip install pcaspy pyepics. And I don't see exceptions.

Directory of C:\miniconda-x86\Lib\site-packages\pcaspy

04.10.2019  08:46             1'286 alarm.py
04.10.2019  08:46            49'664 asIoc.dll
04.10.2019  08:46           271'360 ca.dll
04.10.2019  08:46           176'128 cas.dll
04.10.2019  08:46            26'972 cas.py
04.10.2019  08:46           273'408 Com.dll
04.10.2019  08:46           190'976 dbIoc.dll
04.10.2019  08:46            95'744 dbStaticIoc.dll
04.10.2019  08:46            31'379 driver.py
04.10.2019  08:46           161'280 gdd.dll
04.10.2019  08:46               858 tools.py
04.10.2019  08:46           181'248 _cas.cp37-win32.pyd
04.10.2019  08:46                74 _version.py
04.10.2019  08:46               142 __init__.py
04.10.2019  08:46    <DIR>          __pycache__

I have jumped the gun and didn't read carefully the code from the issue poster originally pyepics/pyepics#176. I don't 'import pcaspy' but I got the same problem.

[xiaoqiangwang]

From 0.7.3, the windows wheel link to static EPICS libraries. So the dll conflicts are not there anymore.