Can osim-rl create a new environment like Open AI Gym?

[Dsai1006]

I want to create new environment for my .osim file, and I want to do reinforcement learning in that environment. Can osim-rl create a new environment like Open AI Gym?

[kidzik]

It is possible, although you would neet to also write your own wrapper. Here is a good simple example of another .osim turned into an environment https://github.com/stanfordnmbl/osim-rl/blob/master/osim/env/arm.py

[Dsai1006]

Thank you.

And, I want to use osim model with McKibbenActuator. But, Object type McKibbenActuator is not recognized. Can't I use Object type McKibbenActuator in osim-rl?

[Dsai1006]

I can use McKibbenActuator in opensim-core. Is there a way to use it in osim-rl?

[kidzik]

You should be able to use most of classes available in OpenSim. Can you give us specifics of your error? Ideally a minimal example to reproduce the error.

[Dsai1006]

Sorry, I can use it.

By the way, when I use env.reset() in windows environment, the program is terminated. Is there a solution? To be more specific, when train_arm.py is executed, only target: [0.017934784704161454 0.4866614200109531] is displayed and the program is terminated.

[kidzik]

Are you getting any error message? Again a minimal example to reproduce the error will be very helpful.

[Dsai1006]

This is the error statement. Normally, the model should be defined, but the program terminates immediately after env.reset().

(opensim-rl) C:\Users\dsait\osim-rl\examples>python train_arm.py
Using TensorFlow backend.
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\tensorflow\python\framework\dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Updating Model file from 20303 to latest format...
Loaded model arm26 from file C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\osim\env\../models/arm2dof6musc.osim
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))

target: [0.34263648937737284 0.5514187553958187]

(opensim-rl) C:\Users\dsait\osim-rl\examples>     

[Dsai1006]

I get an error like this when activating a virtual environment, is this the cause?

(base) C:\Users\dsait\osim-rl\examples>conda activate opensim-rl

C:\Users\dsait\osim-rl\examples>SET DISTUTILS_USE_SDK=1

C:\Users\dsait\osim-rl\examples>SET MSSdk=1

C:\Users\dsait\osim-rl\examples>SET platform=

C:\Users\dsait\osim-rl\examples>IF /I [AMD64] == [amd64] set "platform=true"

C:\Users\dsait\osim-rl\examples>IF /I [] == [amd64] set "platform=true"

C:\Users\dsait\osim-rl\examples>if defined platform (set "VSREGKEY=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0" )  ELSE (set "VSREGKEY=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\14.0" )

C:\Users\dsait\osim-rl\examples>for /F "skip=2 tokens=2,*" %A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0" /v InstallDir') do SET "VSINSTALLDIR=%B"
エラー: 指定されたレジストリ キーまたは値が見つかりませんでした

C:\Users\dsait\osim-rl\examples>if "" == "" (set "VSINSTALLDIR=" )

C:\Users\dsait\osim-rl\examples>if "" == "" (
ECHO "WARNING: Did not find VS in registry or in VS140COMNTOOLS env var - your compiler may not work"
 GOTO End
)
"WARNING: Did not find VS in registry or in VS140COMNTOOLS env var - your compiler may not work"
指定されたバッチ ラベルが見つかりません - End

(opensim-rl) C:\Users\dsait\osim-rl\examples> 

[kidzik]

Since you are using the train_arm.py example, I suppose you have tensorflow installed? There are some problems with certain versions of tensorflow on windows and this might be it. Can you verify that keras and tensorflow work correctly? For example, by running one of the keras examples https://keras.io/getting-started/sequential-model-guide/

For testing just opensim RL you can try a simpler example https://github.com/stanfordnmbl/osim-rl/blob/master/examples/arm2d.py

[Dsai1006]

I can verify keras and tensorflow, but when arm2d.py is executed, program terminates in the same way.

(opensim-rl) C:\Users\dsait\osim-rl\examples>python arm2d.py
Updating Model file from 20303 to latest format...
Loaded model arm26 from file C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\osim\env\../models/arm2dof6musc.osim
C:\Users\dsait\Anaconda3\envs\opensim-rl\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))

simbody-visualizer: received Shutdown message. Goodbye.

target: [0.3446909238932005 0.5734890417695653]

(opensim-rl) C:\Users\dsait\osim-rl\examples>

[kidzik]

And that's with the original arm2dof6musc.osim model or your modified one?

Expected output is roughly like this (with some print statements removed):

It's most likely a problem with windows deployment. Which Windows version you are using? Can you post output of pip list?

[Dsai1006]

I use an original model. pip list is here.

(opensim-rl) C:\Users\dsait\osim-rl\examples>pip list
Package              Version
-------------------- -------------------
absl-py              0.9.0
astor                0.7.1
certifi              2020.4.5.1
chardet              3.0.4
click                7.1.2
cloudpickle          1.3.0
configparser         5.0.0
cycler               0.10.0
docker-pycreds       0.4.0
future               0.18.2
gast                 0.3.3
gitdb                4.0.4
GitPython            3.1.1
gql                  0.2.0
graphql-core         1.1
grpcio               1.23.0
gym                  0.17.1
h5py                 2.10.0
idna                 2.9
Keras                2.3.1
Keras-Applications   1.0.8
Keras-Preprocessing  1.1.0
keras-rl             0.4.2
kiwisolver           1.2.0
Mako                 1.1.0
Markdown             3.2.1
MarkupSafe           1.1.1
matplotlib           3.2.1
mkl-service          2.3.0
mock                 4.0.2
numpy                1.18.3
nvidia-ml-py3        7.352.0
opensim              4.0
osim-rl              3.0.11
pathtools            0.1.2
pip                  20.1
promise              2.3
protobuf             3.11.4
psutil               5.7.0
pyglet               1.5.0
pygpu                0.7.6
pyparsing            2.4.7
pyreadline           2.1
python-dateutil      2.8.1
PyYAML               5.3.1
redis                3.5.0
requests             2.23.0
scipy                1.4.1
sentry-sdk           0.14.3
setuptools           46.1.3.post20200325
shortuuid            1.0.1
six                  1.14.0
smmap                3.0.2
subprocess32         3.5.4
tensorboard          1.13.1
tensorflow           1.13.2
tensorflow-estimator 1.13.0
termcolor            1.1.0
Theano               1.0.4
timeout-decorator    0.4.1
urllib3              1.25.9
wandb                0.8.34
watchdog             0.10.2
Werkzeug             1.0.1
wheel                0.34.2
wincertstore         0.2

and I use windows10.

I ran sim_L2M2019_randomrun and it terminated in the same way.

[kidzik]

That's weird and unfortunatelly it's most likely some issue with our Windows release of the conda environment. You might be able to make things work with the official opensim-core with python wrappers or by recompiling a new version of opensim.

[Dsai1006]

OK, thank you.