Should OpenMM.dll be in conda/Library/bin rather than conda/Library/lib on Windows?

[chryswoods]

Comment:

Thanks for creating a conda package that is also installable on Windows - it is really useful :-)

I am creating a package that builds on OpenMM and have run into the problem on Windows that my libraries cannot find OpenMM.dll at runtime. I believe this is because OpenMM.dll is installed into conda_root\Library\lib while I believe it should be conda_root\Library\bin. I am not a Windows expert, so could be wrong, but I think the the lib directory is for the .lib files, while DLLs are treated like executables on Windows and need to be in the PATH to be loaded (so should be in the bin directory).

Moving OpenMM.dll into conda_root\Library\bin works for my package. Is this something that can be changed, or is there a reason to put the DLLs in lib?

Thanks :-)

[jaimergp]

@peastman ^

[jaimergp]

We used to add copies of the libs in %LIBRARY_BIN% to avoid overlinking errors during build, but at some point they disappeared, so we removed that workaround to reduce package size.

IIRC, OpenMM expects libraries there... Check the contents of openmm.version to double check!

[chryswoods]

Thanks :-)

[peastman]

I'm not an expert on Windows library handling. On Linux and Mac, lib is the directory where all libraries are supposed to get installed. Anything in that directory is automatically included in Python's library path. Does conda organize things differently on Windows?

[isuruf]

I tried fixing this in https://github.com/openmm/openmm/pull/3390, but it was rejected.

[isuruf]

Does conda organize things differently on Windows?

Yes, but it's not because of how conda works. It's the convention on windows for any package irrespective of conda.

[peastman]

But see https://github.com/openmm/openmm/pull/3473. What is the value of version.openmm_library_path?

[chryswoods]

(base) PS C:\Users\opc> ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:15:42) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import openmm

In [2]: openmm.version.openmm_library_path
Out[2]: 'C:/Users/opc/sire_app/Library/lib'

[chryswoods]

This is the value of %PATH% when I have activated the conda (just vanilla conda, no sub-environments).

(base) C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools>echo %path:;=&echo.%
C:\Users\opc\sire_app
C:\Users\opc\sire_app\Library\mingw-w64\bin
C:\Users\opc\sire_app\Library\usr\bin
C:\Users\opc\sire_app\Library\bin
C:\Users\opc\sire_app\Scripts
C:\Users\opc\sire_app\bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\bin\Roslyn
C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\x86
C:\Program Files (x86)\Windows Kits\10\bin\x86
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\bin
C:\Windows\Microsoft.NET\Framework\v4.0.30319
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\Tools
C:\Users\opc\sire_app\condabin
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0
C:\Windows\System32\OpenSSH
C:\Program Files\Git\cmd
C:\Users\opc\AppData\Local\Microsoft\WindowsApps
.
C:\Users\opc\AppData\Local\Programs\Microsoft VS Code\bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja

This is a pretty vanilla cloud VM with just Visual Studio 2017 build tools, and Git installed centrally, and minimamba from conda-forge installed for local user use in C:\Users\opc\sire_app.

[chryswoods]

Windows does things differently. Conda puts all of the DLLs into the bin directory, e.g.

(base) PS C:\Users\opc> ls sire_app/Library/bin

    Directory: C:\Users\opc\sire_app\Library\bin

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        5/11/2022   3:53 PM                plugins
-a----         5/7/2021   6:31 PM          20960 api-ms-win-core-console-l1-1-0.dll
-a----         5/7/2021   6:31 PM          20928 api-ms-win-core-console-l1-2-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-datetime-l1-1-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-debug-l1-1-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-errorhandling-l1-1-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-fibers-l1-1-0.dll
-a----         5/7/2021   6:31 PM          25032 api-ms-win-core-file-l1-1-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-file-l1-2-0.dll
-a----         5/7/2021   6:31 PM          20936 api-ms-win-core-file-l2-1-0.dll
...
lots of DLLs

The lib files are put in the lib directory

ls sire_app/Library/lib

    Directory: C:\Users\opc\sire_app\Library\lib

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        5/11/2022   4:00 PM                cmake
d-----        5/11/2022   3:53 PM                dde1.4
d-----        5/11/2022   4:00 PM                engines-1_1
d-----        5/11/2022   4:00 PM                icu
d-----        5/11/2022   3:53 PM                itcl4.2.2
d-----        5/11/2022   4:00 PM                libpng
d-----        5/11/2022   3:53 PM                nmake
d-----        5/11/2022   4:00 PM                pkgconfig
d-----        5/11/2022   4:00 PM                plugins
d-----        5/11/2022   3:53 PM                reg1.3
d-----        5/11/2022   3:53 PM                sqlite3.36.0
d-----        5/12/2022  10:23 AM                static
d-----        5/11/2022   3:53 PM                tcl8
d-----        5/11/2022   3:53 PM                tcl8.6
d-----        5/11/2022   3:53 PM                tdbc1.1.3
d-----        5/11/2022   3:53 PM                tdbcmysql1.1.3
d-----        5/11/2022   3:53 PM                tdbcodbc1.1.3
d-----        5/11/2022   3:53 PM                tdbcpostgres1.1.3
d-----        5/11/2022   3:53 PM                tdbcsqlite31.1.3
d-----        5/11/2022   3:53 PM                thread2.8.7
d-----        5/11/2022   3:53 PM                tk8.6
-a----        9/24/2021  11:38 PM         112500 archive.lib
-a----        9/24/2021  11:37 PM        1985064 archive_static.lib
-a----         4/2/2022   7:06 AM           7328 boost_atomic.lib
-a----         4/2/2022   7:06 AM          24058 boost_chrono.lib
-a----         4/2/2022   7:06 AM          44556 boost_container.lib
-a----         4/2/2022   7:06 AM           6060 boost_context.lib
-a----         4/2/2022   7:06 AM          41204 boost_contract.lib
...
lots of .lib files

The OpenMM DLLs are the only DLLs in the lib directory.

(base) PS C:\Users\opc> ls .\sire_app\Library\lib\*.dll

    Directory: C:\Users\opc\sire_app\Library\lib

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----         3/2/2022   3:00 AM        2942976 OpenMM.dll
-a----         3/2/2022   3:06 AM         309248 OpenMMAmoeba.dll
-a----         3/2/2022   3:06 AM         101888 OpenMMDrude.dll
-a----         3/2/2022   3:06 AM          58368 OpenMMRPMD.dll

[peastman]

Out[2]: 'C:/Users/opc/sire_app/Library/lib'

How did you install OpenMM? That isn't any kind of path conda would have set for it. It would install OpenMM inside the miniconda3 folder.

[chryswoods]

It was a simple conda install openmm. The sire_app directory is just what I called my miniconda3 directory.

[peastman]

Ok, I understand. Here's what's going on.

When you install OpenMM through any mechanism (conda, build from source, etc.), it stores the path to wherever your libraries are in version.openmm_library_path. When you import openmm on Windows, it automatically adds that location to the PATH to ensure the libraries will be found:

_path = os.environ['PATH']
os.environ['PATH'] = '%(lib)s;%(lib)s\plugins;%(path)s' % {
    'lib': version.openmm_library_path, 'path': _path}

This used to work correctly, but the CondaForge build of Python 3.10 changed the mechanism for loading libraries on Windows such that it no longer works. Instead you have to use a different mechanism, which we added in https://github.com/openmm/openmm/pull/3473.

os.add_dll_directory(version.openmm_library_path)

That change will be in the next release.

[chryswoods]

Thanks, that should work. It does feel though like it goes against the way Windows normally places its DLLs. These are normally in bin directories rather than lib directories because they are searched for in the executable path rather than the library path. I agree that this feels wrong when coming from a Mac / Linux / Unix world, but Windows is just different.

[peastman]

It's really independent of that. If building from source, the libraries might be in some totally different location, often inside C:\Program Files. This ensures they'll be found wherever they are. What directory they should be put in is a different question.

[chryswoods]

Thanks for your time and help :-)