MS0401_T02 Integrating python scripting in opensimQt

[idhamari]

This is the main task until 15.8/2020

• learning Qt basics (estimated time 3 days)
• learning pythonQt basics (estimated time 3 days)
• learning opensimQt basics (estimated time 3 days)
• Integrating the python version of opensim into our opensimQt interface (estimated time 3 weeks)
  • one should be able to run python code and run opensim python code
  • one should be able to modify simple interface components e.g. rename a body

[riteshsv]

• [x] learning Qt basics (estimated time 3 days) to be completed by 28.7.2020 #24
• [x] learning pythonQt basics (estimated time 3 days) to be completed by 30.7.2020
• [x] learning opensimQt basics (estimated time 3 days) to be completed by 1.8.2020

[riteshsv]

Acquired basic understanding of QT. Will start with pythonQt tomorrow

[idhamari]

Well done, the interesting part is just started, we will have an online meeting next week to discuss your ideas for python integration after you learn about PythonQt features.

[riteshsv]

I have acquired basic understanding of pythonQt and Vtk. Unless I understand opensim it will be difficult to develop a strategy for integrating python. I will start understanding opensim.

[idhamari]

@riteshsv

I have acquired basic understanding of pythonQt and Vtk.

well done :watermelon:

Unless I understand opensim it will be difficult to develop a strategy for integrating python. I will start understanding opensim.

I think before you go to openim iyou should get to know opensimQt first. I would start by trying something simple (no OpenSim involves) e.g. implementing the python console in our opensimQt application.

1. writing simple python syntax in the opensimQt scripting console e.g.

           print("Hello python")

2. access the opensimQt qt elements e.g. changing a button caption or color from the python console.

[riteshsv]

I have created a small application and accessed all ui objects from integrated python console successfully. I will do the same in opensimQt tomorrow, the way you have suggested.

[idhamari]

@riteshsv excellent :watermelon: I am looking forwared to see some results in our interface.

[riteshsv]

@idhamari I have done integration of python on openSimQt and got basic functions to work like openModel() etc...We can have a discussion tomorrow if you are available.

[idhamari]

@riteshsv sounds great, could you please push your code to your branch so I can test it? I have two long meetings tomorrow, do you have time on Friday after 10:00 German time?

[riteshsv]

@idhamari we can do it on friday 10 am German Time. I can push the code but you will have to download and install pythonQt and set paths for same in cmake. Also, I have modified UI little bit for the purpose of demo and discussion.

[idhamari]

@riteshsv great, we meet on Friday at 10:00. Please push the code I will try to install pythonQt and if I have problem, I will post here.

[riteshsv]

@idhamari I have pushed the code to branch-> integrate_python_scripting. after installing pythonQt set the path for set(PYTHONQT_PATH "/Users/ritesh/projects/idhamari/VisSimKoblenz/pythonqt") in our CMakeLists.txt

[riteshsv]

@idhamari Below are the commands that you may try on python console: openSim.openModel() openSim.newModel() openSim.reload() openSim.saveModel() openSim.saveModelAs() openSim.saveAll() openSim.exit() openSim.closeModel() openSim.closeAll() openSim.loadMotion()

runSimulation()

[idhamari]

@riteshsv thanks for your effort :watermelon: I will test it and provide feedback tomorrow.

[idhamari]

@riteshsv

I could not build pythonqt using cmake but I build it using qtcreator and it works. I was able to run a few examples. We probably need to figure out how to make it works using cmake as well as the current CMakeLists luses Qt4.

After that, I added the path as you suggested but when I tried to build opensimQt I get this error:

-- Configuring done CMake Error at CMakeLists.txt:127 (add_executable): Target "opensimQt" links to target "Python::Module" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? Do you have some suggestions on how to fix it?

@yasseerr could you please test this on Windows as well?

[idhamari]

I think the reason for the error is:

CMakeWarning at CMakeLists.txt:65 (message): Python was not found I also found out the built libraries are for python 2 not 3. To fix this, I changed the version in /src/python/python.prf

[idhamari]

Update: everything works now, the problem was multiple versions of python, I had to hardcoded some python pahs.

Quick feedback:

1.The console does not support multi-line code e.g. :

         for i in range(3):
               print(i)

2. The console does not see the system python environment e.g. I have the python version of vtk installed in my system via pip but when I try to import in opensimQt it is not recognized.

        import vtk

[yasseerr]

@idhamari i m currently testing the code, i had problems building pythonQt using cmake and msvs15, reporiting back when it's completed

[riteshsv]

@idhamari

1. pythonQt has to be built outside the project. it does not support cmake. I was planning to discuss this point with you during our meeting about options: -- I will try to build by using custom command in cmake.
2. as per cmake documentation find_package() will search for python installations and will prefer python 3 over python 2. But in your case may be you changed some default python installation paths. -- should we link python as static lib ? just an option. we can think of other options.

[riteshsv]

Update: everything works now, the problem was multiple versions of python, I had to hardcoded some python pahs.

Quick feedback:

1.The console does not support multi-line code e.g. :

         for i in range(3):
               print(i)

1. The console does not see the system python environment e.g. I have the python version of vtk installed in my system via pip but when I try to import in opensimQt it is not recognized.

        import vtk

I did not code for that. will fix the above two issues

[riteshsv]

Update: everything works now, the problem was multiple versions of python, I had to hardcoded some python pahs. Quick feedback: 1.The console does not support multi-line code e.g. :

         for i in range(3):
               print(i)

1. The console does not see the system python environment e.g. I have the python version of vtk installed in my system via pip but when I try to import in opensimQt it is not recognized.

        import vtk

I did not code for that. will fix the above two issues

I have pushed the code.

• for using multi-line, use shift+Enter and then use tab on the next line. once completed, use Enter to execute the code.
• I have modified the code to use local python site-packages.

[idhamari]

pythonQt has to be built outside the project. it does not support cmake. I was planning to discuss this point with you during our meeting about options: I will try to build by using custom command in cmake,

maybe @yasseerr can help with this.

as per cmake documentation find_package() will search for python installations and will prefer python 3 over python 2. But in your case may be you changed some default python installation paths.

Python 2 is not supported anymore so we should concentrate on python3. The problem now is when someone has multiple versions, as in my case, would produce error in cmake. Probably we should add an option like Python_Version to CMakeLists to handle this. In future, we will try to automate the build of everything using scripts for each system windows, linux and mac.

should we link python as static lib ? just an option. we can think of other options.

I am not sure about this, let's discuss this in our next meeting.

for using multi-line, use shift+Enter and then use tab on the next line. once completed, use Enter to execute the code.

we still need a way to run a block of code e.g. via copy/pate similar to the standard python console

     for i in range(3):
            print("HI");

I have pushed the code.

Excellent job :watermelon: , I will test it and report here.

Meanwhile, could you please implement the run function in Script menu, this should be simple. Here is how it works:

1. The user select Script>Run
2. An open file dialog appears to select a python file e.g. myScript.py
   • you can create any simple script file for testing.
3. The script code is loaded and run in the console.

[yasseerr]

HI! @VisSimKoblenz/opensimqtteam the console is also working on windows with some changes to the cmake file, here are some tips that may help with cmake

• for the build of pythonQt i had a problem and its fix was to add a third envirement variable for the version PYTHON_VERSION on top of the PYTHON_LIB and the PYTHON_PATH or you can change it manually from pythonqt/build/python.prf
• the version text differ from platform to another in linux it is 3.7 and 37 in windows , that is true also for library names in the find_library() where the {major}.{minor} became {major}{minor} in windows
• to find Python3 explecitly i used this line of find package find_package (Python3 COMPONENTS Interpreter Development)
• to make it work i also changed all the Python references to Python3, for exemple : Python_FOUND->Python3_FOUND
• a helpful like might be this
• for the headers i didnt find the headers in the build folder ${PYTHONQT_PATH}/src/, so i used the ones in the source folder temporarely

[idhamari]

the console is also working on windows

great! so far everything is working in all systems. Probably we need to modify the cmake files to be more flexible and save a copy of the pythonqt in our repository if needed. @yasseerr if possible, please join our meeting tomorrow at 10 German time.

[idhamari]

Meeting on 7.8.2020:

• all sources should be forked and modified when needed: opensim, vtk, pythonqt
• no hardcoded, setting should be read from a text file e.g. .ini
  • using local python or a static one.

Next ToDo:

find out how to connect between a model object in python and the same object in cpp e.g.

• access the model from python and change it should change the loaded model in cpp.
• changing the model in cpp e.g. the navigator should change it also in python.

A good example is the one from Slicer

[riteshsv]

@idhamari

1. it is taking more time to do research on this topic. Slicer code is too large and complicated. What I have found is:
   • Slicer is using PythonQt but it is heavily customised.
   • Slicer is also using Python Utilities from VTK. I will have to dig deeper to understand exact mechanism.
2. I am also studying another option which is highly recommended in industry, that is pybind11. It uses features of modern c++ , is very light weight and is well documented.

We will have to invest more time to figure out the right solution to our peculiar problem and lay the right foundation upon which we can build other functionalities.

[idhamari]

@riteshsv thanks for your effort and feedback.

1. it is taking more time to do research on this topic. Slicer code is too large and complicated. What I have found is: Slicer is using PythonQt but it is heavily customised. Slicer is also using Python Utilities from VTK. I will have to dig deeper to understand exact mechanism.

VTK has already python wrapping so we probably can use it if needed.

I am also studying another option which is highly recommended in industry, that is pybind11. It uses features of modern c++ , is very light weight and is well documented. We will have to invest more time to figure out the right solution to our peculiar problem and lay the right foundation upon which we can build other functionalities

I disagree, I think pybind11 is not an alternative as it is meant for Qt. Probably an alternative would be PySide2. Still, I believe the slicer approach is the one that already well tested so I prefer to stick to it for now. Probably more students will join the project so we can distribute the workload if needed.

My suggestion:

• check how slicer shares the object memory between python and cpp, an option is to ask in Slicer discourse forum, you may get some answer.
• we make a long term plan to write all the python functions needed for the interface similar to what you already did.

What about a group meeting on Friday for the three of us at 11:00 or 18:00? @yasseerr

[riteshsv]

@idhamari

What about a group meeting on Friday for the three of us at 11:00 or 18:00?

I am available at both timings.

[idhamari]

@yasser @riteshsv

here is a simple idea for linking the model in the navigator and the model in the python:

• all model cpp operations on a model should write the changes to temp model file e.g. modelName_temp_cpp.osim

• all model python operations on a model should write the changes to temp model file e.g. modeName_temp_py.osim

• we create these helper python functions:

  1. pullModelFromNavigator("modelName")
     • this function create a python model object from the temp_cpp file
  2. modelUpdateFromNavigator(modelObject)
     • this function update the model from the temp_cpp file e.g. after some cpp modifications
  3. modelSave(modelObject)
     • this function save the model to its original file not the temp file
  4. pushModelToNavigator(modelObject,modelNode)
     • this function update the model node in the navigator using the temp_py file e.g. after some python modifications. First we write the model to the temp file then we load it in the navigator.
  5. loadModelToNavigator(modelObject)
     • this function load a model from its temp_py file to the navigator e.g. the model is created via python from scratch

• These functions are simples, probably each one is a few lines that deal with files closing, updating and opening

• We need to create a cpp function that writing a model to its temp file and call this function after each model modification

• With the above functions, we can use both opensim cpp and python objects without conflicting and send the update from one to another.

• If the performance is good we can also call the update function after each operation so we have the two models synchronized as well, but this need some testing, especially for large complicated model.

Let's discuss this on next meeting in more technical details.

[yasseerr]

@idhamari i think the solution that you presented will work no problem, but like you said it needs testing for big/medium models as it involve a lot of InputOutput operations that will make it difficult to see changes in an acceptable delay. it also valid if you dont need to control the objects that are created in runtime like Motions & Decorations for the meeting i can attend in both hours no problem.

[idhamari]

@VisSimKoblenz/opensimqtteam we meet on Friday at 11:00

i think the solution that you presented will work no problem, but like you said it needs testing for big/medium models as it involves a lot of InputOutput operations that will make it difficult to see changes in an acceptable delay.

The above solution is better than nothing. We should make it in a flexible way so if a better solution is found, we can modify the code automatically with some script or with minimum effort.

it also valid if you don't need to control the objects that are created in runtime like Motions & Decorations

Let's discuss this in the meeting.

[riteshsv]

@idhamari

The user select Script>Run

1. An open file dialog appears to select a python file e.g. myScript.py
   • you can create any simple script file for testing.
   • The script code is loaded and run in the console.

• The Script>Run is implemented.
• Paste operation for the console implemented
• History of python statements executed on the console is displayed in history box next to the console.
• Console prompt customised.

[idhamari]

well done! @riteshsv :watermelon: I will test it and provide feedback if needed.

[riteshsv]

@VisSimKoblenz/opensimqtteam Team, I have prepared a small presentation so that we can discuss about my findings and suggestions.

[idhamari]

Team, I have prepared a small presentation so that we can discuss about my findings and suggestions.

This sounds nice, thanks for your effort.

I just tested the last commit quickly and but there was a problem in linking, I need to double-check what I missed.

when the user wants to see the updated model in GUI, he only need to reload the model. In case there is a conflict when opening the same file from opensim python and cpp, we can handle this using a temp.osim

we also need to create sub tasks e.g.:

• MS0401_T0201 Python: opensim Python CPP communication: I think there is no need to have many helper functions. I think we need only two python functions:

  • python function getModelFromNavigator("modelName")
    • call save cpp function to save the model
    • load the model to opensim python/

  • python function setModelFromPython(modelName)

    • call save python function to save the model after you check the navigator for the filename

                modelName.printToXML(modelNamePath)

    • call cpp open/reload model

    • call our python manager function vsPyManager.UpdateGui()

• MS0401_T0202 Python: opensim Python and Qt GUI communications

  • visualization:

    • call set setModelFromPython
      • simulation setting

    • write our own python function

         vsPyManager.setAccurayc()
         vsPyManager.UpdateGui()

• MS0401_T03 Python: GUI macro recording

• MS0401_T04 Python: OpenSim Python plugin: this is future work and not important for now.