opensim for python 3.8

[mcmips]

How can I get opensim to install in python 3.8 for google colab? Please help.

[aymanhab]

@kidzik Can you clarify what is the workflow that's supposed to work here:

1. What code base is used?
2. How is it built/compiled on what platforms?
3. How is it tested, packaged and distributed? @carmichaelong If you're familiar of this let me know so we can maintain collectively.

[nickbianco]

Based on a recent post on the Moco forum, there may be issues with installing OpenSim in Python 3.8: https://simtk.org/plugins/phpBB/viewtopicPhpbb.php?f=1815&t=12387&p=0&start=0&view=&sid=0e54eee37b6d796a511ab14df68a3750.

Moco issue: https://github.com/opensim-org/opensim-moco/issues/655.

@mcmips are you able to install on Python 3.7?

[kidzik]

There are two relatively easy ways to run OpenSim in colab.

1. Setting up a jupyter notebook locally and connect colab to it
2. Installing conda and opensim conda distribution in colab

Both installation methods are described here in the context of reinforcement learning with osim-rl https://colab.research.google.com/drive/1Kbubb9_L-MDtjt0nHVYR_zF9CK1Onz4C?usp=sharing Simbody visualizer will be available only in the first installation method. The conda distribution that is provided in the notebook above is only available for python 3.6.

@aymanhab answering your questions:

1. In the OpenSim conda build I was using this commit 399c8d57a779dd5dde2916192f8b92bfc959e269
2. It was compiled with these scripts https://github.com/opensim-org/conda-opensim/tree/master/opensim on MacOS / Ubuntu / Windows. Builds are available here https://anaconda.org/kidzik/opensim/files
3. It was built ad hoc for the NeurIPS challenge 2017 and slightly updated for 2018. There was no proper testing

It's probably good to completely scratch my builds and start off from @chrisdembia code https://github.com/conda-forge/staged-recipes/pull/4693

@mcmips to answer your question, my conda builds won't work with python 3.8 so for that version you would have to recompile OpenSim. Depending what you want to do, the easiest way to make things work could be to just use conda and my OpenSim build with python 3.6

[aymanhab]

@nickbianco do you know if this was specific to Anaconda or to the standard python 3.8? Trying to use Anaconda and 3.8 locally I get the errors below: 1>1: Test command: D:\anaconda3\python.exe "-m" "unittest" "discover" "--start-directory" "D:/src/opensim-core/Bindings/Python/tests" "--verbose" 1>1: Environment variables: 1>1: PATH=D:/build/opensim-core/Release 1>1: Test timeout computed to be: 10000000 1>1: D:\anaconda3\lib\site-packages\numpy__init.py:140: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service 1>1: from . import _distributor_init 1>1: Traceback (most recent call last): 1>1: File "D:\anaconda3\lib\site-packages\numpy\core\init.py", line 24, in 1>1: from . import multiarray 1>1: File "D:\anaconda3\lib\site-packages\numpy\core\multiarray.py", line 14, in 1>1: from . import overrides 1>1: File "D:\anaconda3\lib\site-packages\numpy\core\overrides.py", line 7, in 1>1: from numpy.core._multiarray_umath import ( 1>1: ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found. 1>1: 1>1: During handling of the above exception, another exception occurred: 1>1: 1>1: Traceback (most recent call last): 1>1: File "D:\anaconda3\lib\site-packages\numpy\init.py", line 142, in 1>1: from . import core 1>1: File "D:\anaconda3\lib\site-packages\numpy\core\init__.py", line 50, in 1>1: raise ImportError(msg) 1>1: ImportError: 1>1: 1>1: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! 1>1: 1>1: Importing the numpy C-extensions failed. This error can happen for 1>1: many reasons, often due to issues with your setup or how NumPy was 1>1: installed. 1>1: 1>1: We have compiled some common reasons and troubleshooting tips at: 1>1: 1>1: https://numpy.org/devdocs/user/troubleshooting-importerror.html 1>1: 1>1: Please note and check the following: 1>1: 1>1: The Python version is: Python3.8 from "D:\anaconda3\python.exe" 1>1: The NumPy version is: "1.18.5" 1>1: 1>1: and make sure that they are the versions you expect. 1>1: Please carefully study the documentation linked above for further help. 1>1:

@kidzik Does the opensim-rl environment depend on numpy? and do we need to be on 3.8? Thanks

[nickbianco]

The user on the forum was using standard Python 3.8. He got an error when the Python install script tries to import the simbody Python module but was unable to find it. Module imports happen using Python's importlib, which got some minor updates in 3.8, but I've not done any debugging of my own yet.

[aymanhab]

This appears to be SWIG related (release notes for SWIG 4.0.1 explicitly specifies adding support for python 3.8), so it makes no sense to try to fix this anywhere else at the moment, instead we should plan upgrading SWIG first. How time critical is this? @kidzik @nickbianco

[nickbianco]

I think most users can revert to 3.7 (as did the user on the Moco forum). Unless we have users who specifically need 3.8 features it's probably not super time critical.

[kidzik]

It's not time-critical for my use cases (NeurIPS challenge and RL workshop). Also, in the tutorial I'm listing exact versions with which everything is functioning well. Numpy is indeed needed for opensim-rl. We could make it leaner and depend only on OpenSim. Our users will almost surely use numpy anyway. So definitely it makes sense to update SWIG first.

[chrisdembia]

The next release of opensim will depend on numpy.

[adamkewley]

As an additional datapoint, I just spent a while trying to get python to work on my Windows machine for a researcher because of this issue.

The reason why I found the install pretty confusing is because using Python with OpenSim right now involves non-standard library setup steps (e.g., not using pip) and involves a few extremely specific steps, where mis-doing those steps produces a vague-sounding DLL linking error message (generated from SWIG):

• The docs do mention python2 vs. 3 (here) but ambiguate the version when they say "set the CMake variable OPENSIM_PYTHON_VERSION to the desired version (2 or 3)". Reading it a bit more carefully clarifies that, no, it must be specifically python 3.7, not at least 3.7 (which would be assumed)
• The python official website's officially recommended download for my Windows10 (64-bit) is 3.8.5 32-bit edition (this page, the link i got). The 64-bit edition is actually the non-default - despite 64-bit being mainstream for a decade or so. The majority of windows python users (ignoring anaconda, et. al) will install whatever that site suggests to them.
• python setup.py install of the opensim SDK will succeed, regardless of python version or ABI - the issue only occurs when running the binary and there are no installation-time sanity checks
• The setup also mensions editing Windows environment variables (e.g. the PATH), which I figured would be causing a runtime error (e.g. the runtime searches via the PATH), so I spent a while trying diffent PATHs
• Switching to 3.8.5 64-bit observes almost identical error messages (slightly different, if paying attention) to using 32-bit, so it's easy to confuse the issue of the python version being incorrect with the ABI being incorrect.
• On seeing the new error, I assumed I had mis-installed something and spent time adding debug messages into the SWIG bindings to find out why it couldn't load _simbody
• After exhausting all of that, I eventually just switched specifically to version 3.7.8 (64-bit). This isn't an automatic assumption for python devs because the version concern usually only applies to "is it 2 or 3". Version-specific native issues are usually handled by compiling the native bindings at install time (e.g. pip will do this for source balls), or at least having the installer produce an annoying error message that explains the problem.

[adamkewley]

To further compound the issue, after I just tried to get Anaconda working with the researcher:

• Latest Anaconda3 now uses Python 3.8, which will not work with OpenSim 4.1. Using something like conda install python=3.7 might work, but it seems to take a very long time to run, and can subsequently explode if the user has installed a dependency that can't be cleanly downgraded to 3.7 (this was our problem--something about downgrading openssl would cause an error--so we couldn't move forward)
• Anaconda2 (documented for OpenSim) defaults to Python2. Users then have to upgrade to 3.7. This took so long (>10 min) that I had to ditch the call, so I'm assuming that typical Anaconda users don't typically do this (instead, they use whatever the platform integrates in a particular version)
• Because it took too long to switch the version over, we didn't have the time to try out spyder. I'm assuming that, once python is upgraded/downgraded, spyder will correctly use 3.7, but (iirc) conda can have parallel python versions, so there might be a separate configuration step required to ensure spyder then uses 3.7 after it is installed
• From what I gathered, the reason some research teams standardize on a major Anaconda release is so that all team members can just install anaconda + libraries (via the anaconda package manager), and call it a day. The fact that OpenSim may specifically require a python downgrade and a manual installation is unlikely to go down so well with teams building software that way.

[aymanhab]

Thanks for the feedback @adamkewley You have to remember that:

1. Python changed how packages are initialized in version 3.8, the documentation was written before that change so it assumed python will maintain backward compatibility which is false in hindsight (We tested against 3.4-3.7). If you can update the documentation until we upgrade that would be great.
2. Based on your investigation, would bindings created for python 3.8 work with 3.7 and earlier? or we have to choose between the two? Ideally we don't force users on python 3.7 and earlier to upgrade just to please 3.8 users 😸
3. Not sure what needs to be changed on our side to use pip instead of the current mechanism, any clues?

[adamkewley]

I might try a 3.8 custom build out to satisfy the researcher on our side. It might be that swig generates mostly low-3 source-level bindings but the problem is likely to be the native bindings they hook into (if python 3.8 changed things). I'll report back after I try this.

For pip et al, it would potentially be enough for us to ship a very minimal package that effectively checks the python version, abi, and runtime searches for our preinstalled SDK package, followed by forwarding or installing the SDK. That would effectively be a python-friendly installer that politely asks the user to go and install opensim from simtk, set such and such a variable, "sorry, your python won't work", etc. I might be able to whip this up in an unofficial package as a prototype and then I can get back to you once I know the pitfalls.

The main issue is that typical python developers are used to their package manager railroading loads of stuff. Even if we provide a minimal package that is clear about what the problem is (by showing error messages etc) it might eradicate a bunch of stuff we otherwise need to manually document and hope people read carefully.

On Thu, Sep 24, 2020, 7:07 PM Ayman Habib notifications@github.com wrote:

Thanks for the feedback @adamkewley https://github.com/adamkewley You have to remember that:

1. Python changed how packages are initialized in version 3.8, the documentation was written before that change so it assumed python will maintain backward compatibility which is false in hindsight (We tested against 3.4-3.7). If you can update the documentation until we upgrade that would be great.
2. Based on your investigation, would bindings created for python 3.8 work with 3.7 and earlier? or we have to choose between the two? Ideally we don't force users on python 3.7 and earlier to upgrade just to please 3.8 users 😸
3. Not sure what needs to be changed on our side to use pip instead of the current mechanism, any clues?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opensim-org/opensim-core/issues/2869#issuecomment-698471638, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEC5ST26NMEACZOXYRVGBLSHN4EZANCNFSM4QPT6TPA .

[adamkewley]

As an update for this, I manually stripped all traces of other python versions from the PC and built OpenSim with anaconda's python3.8. It seems to build + install fine, but it's now having issues loading the native binaries in the .egg file that the SDK's setup.py installs to:

(base) C:\Users\ak>python
Python 3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import opensim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "C:\Users\ak\anaconda3\lib\site-packages\opensim-4.2-py3.8.egg\opensim\__init__.py", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "C:\Users\ak\anaconda3\lib\site-packages\opensim-4.2-py3.8.egg\opensim\simbody.py", line 13, in <module>
ImportError: cannot import name '_simbody' from 'opensim' (C:\Users\ak\anaconda3\lib\site-packages\opensim-4.2-py3.8.egg\opensim\__init__.py)

Which might've been similar to message I was getting when using 4.1 with python3.7, so it could be that there's some module-loading-path missing (the module is right there, in the zip package, though).

I Had similar issues with loading numpy in anaconda when configuring the custom build (to build OpenSim4.1 via anaconda /w python 3.8). To fix that, I had to set up my %PATH% with anaconda binaries etc. (e.g.). For this build of OpenSim, it seems to be that it can't load _simbody from within the .egg file the installer makes, even if my PATH is correctly occupied with opensim binaries (i can run opensim-cmd, etc.):

(base) C:\Users\ak>echo %PATH%
C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Simbody\lib;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\lib;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\bin;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install;C:\Users\ak\anaconda3;C:\Users\ak\anaconda3\Library\mingw-w64\bin;C:\Users\ak\anaconda3\Library\usr\bin;C:\Users\ak\anaconda3\Library\bin;C:\Users\ak\anaconda3\Scripts;C:\Users\ak\anaconda3\bin;C:\Users\ak\anaconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\dotnet;C:\Program Files\Git\cmd;C:\Program Files\MATLAB\R2020a\bin;C:\OpenSim 4.1\bin;C:\Users\ak\AppData\Local\Microsoft\WindowsApps;C:\Users\ak\.dotnet\tools;C:\Program Files\CMake\bin;C:\Program Files (x86)\NSIS\Bin;C:\Program Files (x86)\NSIS;C:\Users\ak\AppData\Local\GitHubDesktop\bin;C:\OpenSim 4.1\bin;.

However, if I use the package in its unpacked form (e.g. by manually setting PYTHONPATH=C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\python;%PYTHONPATH% so that python's module loader loads the unpacked SDK, rather than the installed egg), then it works fine:

(base) C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python>set PYTHONPATH=%PWD%;%PYTHONPATH%

(base) C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python>python
Python 3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import opensim
>>>

So the trick for Anaconda /w 3.8 seems to be:

• Install latest Anaconda
• Ensure numpy is installed in it (it should be, by default)
• Set up build system's PATH to be able to run Anaconda's python, ensure it definitely can import numpy
• Build OpenSim with that populated path (it should automatically detect that python by attempting things on the PATH)
• Install OpenSim somewhere
• Set system PATH to the usual bin dir
• Set system PYTHONPATH to sdk\Python
• Don't bother running OpenSim's setup.py install

You'd think that, because the Anaconda version of OpenSim is built against 3.8, opensim would "just work" with barebones Python 3.8. But going through the exact same steps yields:

C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Simbody\lib;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\lib;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Program Files\MATLAB\R2020a\bin;C:\OpenSim 4.1\bin;C:\Users\ak\AppData\Local\Microsoft\WindowsApps;C:\Users\ak\.dotnet\tools;C:\Program Files\CMake\bin;C:\Program Files (x86)\NSIS\Bin;C:\Program Files (x86)\NSIS;C:\Users\ak\AppData\Local\GitHubDesktop\bin;C:\OpenSim 4.1\bin;
""
C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\bin;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python;C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python\opensim
""
Python 3.8.6rc1 (tags/v3.8.6rc1:08bd63d, Sep  7 2020, 23:10:23) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import opensim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python\opensim\__init__.py", line 1, in <module>
    from .simbody import *
  File "C:\Users\ak\Desktop\opensim-python38\RelWithDebInfo-install\sdk\Python\opensim\simbody.py", line 13, in <module>
    from . import _simbody
ImportError: DLL load failed while importing _simbody: The specified module could not be found.

They are slightly different versions of python, and they are built with slightly different versions of MSVC. Might be that, no idea. I installed numpy into the barebones python3.8 and it imports fine, so I don't think it's missing numpy.

It could also be because (as far as I can tell) the OpenSim build requires locating NumPy at configure time, which it probably uses it for some of the python bindings (perf?). The (working) anaconda version has the advantage of being the exact version built against with the exact numpy built against.

Either way, for my purposes, I might be able to internally ship the custom build /w instructions for using it with Anaconda as a temporary bandaid.

[dembia]

@adamkewley would you be interested in picking up the work of creating an opensim conda package? This is the correct long term solution. A lot of the work is already done: https://github.com/conda-forge/staged-recipes/pull/11894 . I could answer any questions you have.

[adamkewley]

Additional update: we now have a working python3.8-based anaconda deployment of opensim internally. It seems to mostly work fine if we just hack people's bashrc computer-wide with a PYTHONPATH that points to the sdk/python dir. This probably isn't in-line with how anaconda should be used (i.e. a constrained environment pulled via its package manager), but works for us.

[aymanhab]

Thanks for the update @adamkewley The main question I have is what version of SWIG you used to create the bindings and on which platforms. If you needed to make changes to get the latest source to work with python 3.8 it would be great to have these on our master branch. Did it work also with generic python 3.8 from python.org? Thank you

[adamkewley]

@aymanhab , for our workstation's opensim (the one on which the researcher is using Anaconda /w Python 3.8):

swig3.0 -version

SWIG Version 3.0.12

Compiled with g++ [x86_64-pc-linux-gnu]

Configured options: +pcre

Please see http://www.swig.org for reporting bugs and further information

# system-wide python
/usr/bin/python3 --version
Python 3.8.5

# conda python: default environment

(base) adam:~$ which python
/opt/anaconda3/bin/python
(base) adam:~$ python --version
Python 3.8.3

The researcher is using python3.7, installed as an additional environment. opensim was working with 3.8, he just downgraded it because he uses 3.7 on his local machine.

Hacky fix I added into /etc/bash.bashrc:

# add opensim to system-wide pythonpath
export PYTHONPATH=/usr/local/lib/python3.8/site-packages/:$PYTHONPATH
export PATH=/opt/anaconda3/bin:$PATH

For the actual build, I built opensim with the scapulo joint merged. Can't remember why, could be because one of the researchers needs access to the joint for their models and I didn't want the hassle of setting up the library paths etc.:

commit 5cf9526e4f85bfbd2c1f3cac985727c37628ac77 (HEAD -> feature-merge-scapulothoracicjoint, adam/feature-merge-scapulothoracicjoint)
Author: Adam Kewley <contact@adamkewley.com>
Date:   Thu Sep 3 10:23:55 2020 +0200

    Merge ScapulothoracicJoint plugin (from https://github.com/opensim-org/ScapulothoracicJointPlugin)

Build script (not my finest work: something I had lying around from months ago):

#!/usr/bin/env bash

# (before) checkout this project
# git clone https://github.com/opensim-org/opensim-core.git
# cd opensim-core/

# (before) install package-managed dependencies
#apt-get install git cmake cmake-curses-gui \
#                freeglut3-dev libxi-dev libxmu-dev \
#                liblapack-dev swig python-dev default-jdk

# CMAKE_BUILD_TYPE: what type of build to run
#   Debug           Unoptimized (10x slower) build with debug symbols
#   Release         Optimized without debug symbols
#   RelWithDebInfo  Optimized build with debug symbols (default)
#   MinSizeRel      Optimized **for size** build without debug symbols

opensim_project="${PWD}"

# where build intermediates are placed
opensim_build_dir="${opensim_project}/opensim-core-build"

# where built binaries are placed

# same, but for vendored dependencies
dependencies_project="${opensim_project}/dependencies"
dependencies_build_dir="${opensim_project}/opensim-core-dependencies-build"

# number of build targets to build in parallel
num_build_jobs=$(nproc)

# build vendored dependencies that OpenSim uses
mkdir -p "${dependencies_build_dir}"
pushd "${dependencies_build_dir}"
      cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      "${dependencies_project}"
make -j"${num_build_jobs}"
popd

# build OpenSim
mkdir -p "${opensim_build_dir}"
pushd "${opensim_build_dir}"
cmake -DOPENSIM_DEPENDENCIES_DIR="${opensim_project}/opensim_dependencies_install" -DBUILD_JAVA_WRAPPING=OFF \
      -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      "${opensim_project}"

make -j "${num_build_jobs}"
ctest -j "${num_build_jobs}"
make install
popd

[aymanhab]

Development builds as of PR #2987 are made against python 3.7 and numpy 1.19

[jenhicks]

@aymanhab What is the status of this issue?

[aymanhab]

We used Anaconda for the Moco workshop at TGCS, using python 3.8 so I assume this is fixed already, but I haven't used the colab environment directly, maybe others did e.g. @carmichaelong @nickbianco I can try while testing 4.3

[carmichaelong]

If this issue is strictly about OpenSim working with 3.8, then we should at least have this tested well with Windows and Mac environments. @aymanhab have you tried on Linux?

I haven't tried using OpenSim with colab yet.

[kidzik]

I was using it in a workshop, but I used a workaround hack -- participants had local binaries of OpenSim (not in google's cloud) and colab connected to these local binaries https://colab.research.google.com/drive/1Kbubb9_L-MDtjt0nHVYR_zF9CK1Onz4C

[aymanhab]

Successful run in browser with 3.8 and OpenSim 4.3 Beta as shown here

[marnunez]

Could you share how did you manage to set this up @aymanhab? I've been trying for days to have the latest OpenSim 4.3 artifact python bindings running, and I still get the "DLL load failed: the specified module could not be found", even on a fresh python 3.7.9, numpy 1.19 install. I checked and double checked the environment variables:

and

and I still get:

So far I tried fresh environments with python 3.7 and 3.8 with numpy 1.19, both from anaconda and conda-forge. Also tested the nomkl versions.

[aymanhab]

@mcmips I would not try hooking jupyter until you can have a conda environment running to reduce the number of players involved. Unless you build from source and use python 3.7, the prebuilt bindings work only with python 3.8. Unfortunately recent changes to SWIG and python made it hard for the bindings to support multiple python versions as is. The instructions for installing the bindings with conda/python 3.8 are included here https://simtk-confluence.stanford.edu/display/OpenSim/Scripting+in+Python and has been tested many times for a workshop without an issue, please compare the steps and let me know if you get conda running with OpenSim 4.3 on windows. If successful I can point you to the recipe I used to wire Jupyter.

[marnunez]

Thanks @aymanhab ! I'll see if I can get it to work with a Python 3.8

[clnsmith]

@marnunez Any luck? I have just gone down the same rabbit hole with the following error:

ImportError: DLL load failed while importing _simbody: The specified module could not be found.

I will try to organize my testing, but I am on Win 10 using anaconda with python 3.8 and OpenSim 4.3 (from the standard opensim download, eventually I want to use a forked version of opensim-core for opensim-jam).

A few notes so far: 1) https://simtk-confluence.stanford.edu/display/OpenSim/Scripting+in+Python There is an inconsistency abot using OpenSim 4.2: At the top of the page it says you cannot use python 3.8 and opensim 4,2, but then under the "Special instructions for Python 3.8+ on Windows:" it has instructions for 4.2.

2) To be clear, the command prompt comands:

python setup_win_python38.py python -m pip install .

should be run before: python setup.py install

3) #2987 Here I see you are forcing binding against numpy 1.19.3... but annoyingly if I run "conda search numpy" I can install 1.19.2 or 1.19.4, but there is no source for 1.19.3

[aymanhab]

@clnsmith Thanks for reporting:

1. To be clear you can't use python 3.8 with OpenSim 4.2 because the SWIG version (3.0.9) used then was old and the resulting code didn't compile. In 4.3 we upgraded to SWIG 4.0.2 and modified interface files accordingly, so there should be no inconsistency: version 4.2 use Python 3.7, version 4.3 use Python 3.8 if we have language indicating otherwise please let me know to fix.
2. Are you sure to have set the PATH to the bin folder of the installation? While python doesn't use the PATH internally anymore, the python bindings _*.dlls do use it AFAIK.
3. The numpy version was decided based on the default anaconda install on windows but you'll likely be able to use 1.19.4 as patches are not supposed to change the interface.

Please let me know what you find out as this likely going to be a common issue for users on python 3.8

[clnsmith]

I still haven't solved this. PATH=C:\OpenSim 4.3\bin PYTHONPATH=C:\OpenSim 4.3\sdk\Python

In anaconda command prompt I created an environment conda create -n opensim_test python=3.8 numpy=1.19.4 spyder conda activate opensim_test

python setup_win_python38.py python -m pip install . python setup.py install

spyder

In my .py file I have: import os os.add_dll_directory("C:\OpenSim 4.3\bin") import opensim as osim

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

Traceback (most recent call last):

File "C:\Users\csmith\github\jam-resources\python\examples\PassiveFlexion\examplePassiveFlexion.py", line 15, in import opensim as osim

File "C:\OpenSim 4.3\sdk\Python\opensim__init__.py", line 6, in from .simbody import *

File "C:\OpenSim 4.3\sdk\Python\opensim\simbody.py", line 13, in from . import _simbody

ImportError: numpy.core.multiarray failed to import

I tried changing numpy to 1.19.2 with conda install, and to 1.19.3 using pip install. I get the same error. I verified the numpy versions in spyder using the command window.

[aymanhab]

@clnsmith This actually suggests that python3.8 is fine and loaded properly and that it's only an issue with the numpy version. I think long term we should decouple the numpy usage and put it in a separate module to avoid this issue but for now you can create a branch based on the tag 4.3 and modify the build script to use the numpy version of interest or let me know and I'll do it for you. The build script is here just in case https://github.com/opensim-org/opensim-core/blob/master/.github/workflows/continuous_integration.yml

[clnsmith]

ahh I didn't look at the actual source code to see numpy 1.20.2 is now linked :tired_face:

Everything is now working using OpenSim 4.3 following the code below. :+1:

When I build my forked repo from source locally, I still get the the following error: ImportError: DLL load failed while importing _simbody: The specified module could not be found.

However, if I use the artifact generated by the github actions for my forked repo, everything works fine with python. So I think the issue with my local build is that I am using Visual Studio 2017.

Many of my previous errors were because I was using OpenSim 4.2, and OpenSim 4.3 beta and an older version of anaconda.

Working Instructions (which do match the Python scripting docs page, just repeating shortened version here for recording, possibly not all needed)

Environemental Variables: PATH=C:\OpenSim 4.3\bin PYTHONPATH=C:\OpenSim 4.3\sdk\Python

In anaconda command prompt I created an environment conda create -n opensim_test python=3.8 numpy=1.20.2 spyder conda activate opensim_test

cd C:\OpenSim 4.3\sdk\Python python setup_win_python38.py python -m pip install . python setup.py install

spyder

In my .py file I have: import os os.add_dll_directory("C:\OpenSim 4.3\bin") import opensim as osim

[marnunez]

Excellent @clnsmith! I believe using os.add_dll_directory might not be needed in your script, setup_win_python38.py should have taken care of that.

Note: OpenSIM's 4.3 ezc3d.dll collides with conda's ezc3d package, so if you try to install ezc3d in that environment, importing opensim will fail. So don't install both in the same environment for now.

[clnsmith]

I now have the local source build working as well. I upgraded to VS 2019 and found that the "PYTHON_LIBRARY" in CMake (under Advanced in GUI) was not pointed to my Anaconda python installation.

[aymanhab]

@clnsmith @marnunez @mcmips Are these issues resolved and the documentation clear on hooking up python to OpenSim 4.3? If not please speak out so we try to address the issue(s) or close this one. Thank you

[jenhicks]

@aymanhab I think we can close this issue, but first could you point to the issue for conda, as I think commenters will be interested in following progress on that one. Thanks!

[aymanhab]

Working on related issue https://github.com/opensim-org/opensim-core/issues/2396 Steps so far:

• [x] Re-arrange build system so both numpy and python are found consistently using CMake
• [x] Add a CMake flag to avoid packaging runtime as is handled by the conda packaging system
• [x] Provide recipe for build and for specifying variants
• [ ] Create PR for conda-forge recipe including testing and have it reviewed/merged

[jenhicks]

Thanks ... is the list included elsewhere, and ok to close this issue?

[aymanhab]

No, the list is not included elsewhere I was just outlining the plan of attack. I still think this can be closed since opensim with python 3.8 is operational in 4.3 already

[clnsmith]

Slow response... but this issue is solved my perspective.

I just went through the opensim python documentation page again to update to source code changes for conda and the documentation page was clear for me.

A trick for building from source where you want OpenSim to find a custom Anaconda environment: On windows building with CMAKE GUI: "Add Entry" Python3_ROOT_DIR=C:/Users/USERNAME/anaconda3/envs/opensim_jam

where opensim_jam is the custom environment name.

[nickbianco]

I've also needed to set Python3_ROOT_DIR when building OpenSim from source to set my desired conda environment. We should document this in the updated build instructions (related: #3178).

[ParticularMiner]

I'd be grateful if someone could explain how os.add_dll_directory(...) works.

Python documentation seems to indicate that setting the PATH environment variable in Windows to "C:\OpenSim 4.3\bin" in order to enable the file-access of _simbody.lib is not required for Python versions 3.8+:

Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.

The OpenSim documentation also says the same thing:

Starting at version 3.8, Python changed how dll's are located on windows, in particular PATH is not used any more, instead every environment should set the path to locate the dlls using ...

But this is not what I observe in practice: it is still necessary to set the PATH environment variable!

I might be missing something. But at the moment, I suspect this might be causing confusion to OpenSim end-users.

[aymanhab]

My conjecture is that while python decided to invent its own mechanism for locating dlls, the underlying windows system is still using it since this how all other programs are distributed, so it's possible this is still necessary. You may want to add dll directory for simbody as well to see what happens.

Due to the wide variety of python distributions, we've created a self contained conda package in release 4.4, I'd recommend you give that a try.

On Mon, Jun 13, 2022, 7:54 AM ParticularMiner @.***> wrote:

I'd be grateful if someone could explain how os.add_dll_directory(...) works.

Python documentation https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew:~:text=Specifically%2C%20PATH,loading%20your%20library. seems to indicate that setting the PATH environment variable in Windows to "C:\OpenSim 4.3\bin" in order to enable the file-access of _simbody.lib is not required for Python versions 3.8+:

Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() https://docs.python.org/3/library/os.html#os.add_dll_directory and if it exists, use it to add your DLLs directory while loading your library.

The OpenSim documentation https://simtk-confluence.stanford.edu:8443/display/OpenSim/Scripting+in+Python#:~:text=Starting%20at%20version%203.8%2C%20Python%20changed%20how%20dll%27s%20are%20located%20on%20windows%2C%20in%20particular%20PATH%20is%20not%20used%20any%20more%2C%20instead%20every%20environment%20should%20set%20the%20path%20to%20locate%20the%20dlls%20using%20this%20snippet also says the same thing:

Starting at version 3.8, Python changed how dll's are located on windows, in particular PATH is not used any more, instead every environment should set the path to locate the dlls using ...

But this is not what I observe in practice: it is still necessary to set the PATH environment variable!

I might be missing something. But at the moment, I suspect this might be causing confusion to end-users.

— Reply to this email directly, view it on GitHub https://github.com/opensim-org/opensim-core/issues/2869#issuecomment-1154027054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6JY4HCNM25Z7EP7N3TW3TVO5DTJANCNFSM4QPT6TPA . You are receiving this because you were mentioned.Message ID: @.***>

[ParticularMiner]

Thank you very much @aymanhab for the pointers! I'm also excited to learn about the new conda release. That would certainly make things much easier.

So I executed the following commands without any problems:

> conda create -n gaitdev
> conda install -c opensim-org -n gaitdev opensim
> conda activate gaitdev
(gaitdev) > python
Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:32:50) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

But got the following error on attempting to import opensim:

>>> import opensim as osim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\runner\anaconda3\envs\gaitdev\lib\opensim\__init__.py", line 7, in <module>
    os.add_dll_directory(DLL_PATH)
NameError: name 'DLL_PATH' is not defined
>>> 

Is this expected behavior? I had expected the conda installation script to set the dll paths automatically so that I wouldn't have to do it myself (by running python setup_win_python38.py in C:\Users\runner\anaconda3\envs\gaitdev\Lib , for example).

[ParticularMiner]

@aymanhab

FYI, following my previous message, I modified the following line in my conda environment's C:\Users\runner\anaconda3\envs\gaitdev\Lib\__init__.py: https://github.com/opensim-org/opensim-core/blob/f100bf2fa725412b6ba7cd020320218f6184e2c0/Bindings/Python/__init__.py#L4

to

   if (os.path.exists(os.path.join(sys.prefix, 'Lib'))):

This cleared the runtime error above, but raised another error (the old one):

> python -c "import opensim"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\runner\anaconda3\envs\gaitdev\Lib\opensim\__init__.py", line 10, in <module>
    from .simbody import *
  File "C:\Users\runner\anaconda3\envs\gaitdev\Lib\opensim\simbody.py", line 13, in <module>
    from . import _simbody
ImportError: DLL load failed while importing _simbody: The specified module could not be found.

I also tried setting environment variable CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 as suggested in https://github.com/conda/conda/issues/10897#issue-989357732 and even manually set the PATH environment variable.

None of the above attempts worked.

Question: Do I need to install the full OpenSim 4.4 (non-python) application somewhere on my system, in addition to the conda package opensim installation?

[aymanhab]

You don't need to install OpenSim elsewhere, this is supposed to be self contained. That said, you have to use specific python versions (we have python 3.7,8,9 but no 3.10). Also the line you modified was supposed to detect if the python in use is conda based but other installation/settings could fail, I'm curious why the line failed and you needed to modify it (so we can make this detection more robust).

[ParticularMiner]

you have to use specific python versions (we have python 3.7,8,9 but no 3.10)

Thanks for the info, @aymanhab. However, removing and recreating the environment with conda create -n gaitdev python=3.9 for instance, did not solve the problem.

I'm curious why the line failed

It seems that the conda-detection line failed on my system because my conda installation does not have a subdirectory named "conda". I guess this is the default configuration for conda environments. But correct me if I'm wrong.

[aymanhab]

Some suggestions:

1. I'd make sure no other opensim installation is in the path or leftovers in your anaconda environment, etc.
2. Try installation in base environment and make sure you get the expected python version (if that's not supported yet let me know and I'll make one). You're unlikely to have multiple versions around at the same time anyway.
3. If you try installation and dll fails to load, use dependency walker to find out what dlls/libraries are missing. Let us know how that goes to document your findings. Thank you

[ParticularMiner]

Installation in the base environment yielded the same results as above:

(base) > python --version
Python 3.8.8
(base) > conda install -c opensim-org opensim
(base) > python -c "import opensim"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\runner\anaconda3\lib\opensim\__init__.py", line 7, in <module>
    os.add_dll_directory(DLL_PATH)
NameError: name 'DLL_PATH' is not defined

This is my first time using dependency walker. Could you direct me on how to use it in this case, please. My efforts to pass python -c "import opensim" to dependency walker did not produce any output:

(base) > C:\depends.exe python -c "import opensim"

[aymanhab]

Modify DLL_PATH to the folder that contains all the opensim dlls first