Add an ISB Frame Definition Workflow UI (top-level issue)

[adamkewley]

This issue was broken out from #467 , which was originally an all encompassing issue for building an OpenSim model warping UI.

Issue Background

It turned out that creating a warping UI will require a lot of auxiliary tooling+code to do robustly, so that UI idea was broken into three separate pieces that were deemed to be handy in isolation:

• A mesh warping UI that uses the TPS algorithm (#467)
• A frame definition UI that enables users to define frames based on relationships between landmarks and meshes (this)
• An OpenSim model warping UI that takes multiple mesh warping definitions (from #467) and high-level frame definitions (this issue) to warp OpenSim models (issue number TODO)

Work List

High-Level Work List

• [x] Review how frames are defined in biomechanical models (e.g. the ISB recommendations)
• [x] Review any previous attempts at automation/UIs
• [x] Design a UI that has enough of a featureset to define frames this way
• [ ] DROPPED: Have the UI export a suitable datastructure for defining frames in mesh data
  • DROPPED due to insufficient desire: the UI can export new meshes or directly to an osim
• [ ] DROPPED: Integrate the exported data into the OpenSim model warping's inputs
  • DROPPED: insufficient time: the frame definition UI isn't 100 % compatible with the existing OpenSim infrastructure, and making it compatible will require changes to the existing warping UI, the frame definition UI, and the available components within OpenSim (i.e. we need a way of encoding Frame definition frames in an osim without compatability issues)

Detailed Work List

• [x] Tried to create a new component+property system that doesn't have as many dependencies as OpenSim
  • DROPPED: will require a large amount of engineering to work. Was recommended that we try and do everything in terms of OpenSim::Component
• [x] Created a new FrameDefinitionTab that houses this workflow
• [x] Added right-click callbacks to ModelEditorViewPanel,
  • This is so that context menu customization is possible
  • Because the context menu in the frame definition UI will not be the same as in the general model editor UI
• [x] Added layering support to ModelEditorViewPanel (big refactor)
  • Separates rendering layer, interaction layer, button+gizmo layers
  • Adds API that enables external code to push new layers on top of the viewport (osc::ModelEditorViewPanel::addLayer)
  • This is so that the frame definition UI can push "select N things", "do this", "do that" layers over a viewport in order to prompt a user for relevant information
  • It is also going to be useful in unrelated work (e.g. modal editing, muscle path point placement, etc.)
• [x] Separated ModelEditorViewerPanel's internal code for decoration generation, rendering, hittesting, etc. into separate APIs under src/OpenSimCreator/Graphics (e.g. OverlayDecorationGenerator, OpenSimDecorationGenerator, ComponentAbsPathDecorationTagger, etc.)
  • This is so that the frame definition UI is able to heavily customize what information is seen during one of those layered states (e.g. show the model, but with non-selectable elements faded and not-hittested)
  • It also enables other heavily-customized model viewers from those separate elements (and will be handy in testing, customization, etc. later)
• [x] Used the above changes to implement a basic "select N components" state in the frame definition UI, which will then be customized for "select other point to create edge" and "select 3 edges to create a frame" workflows.
• [x] Created FrameDefinitionPoint base class component
  • [x] This is the "virtual point" base API
  • [x] It enables the UI to just ask "is the given component a point?"
• [x] Created FrameDefinitionSphere component
  • [x] Similar to OpenSim::Sphere, but inherits from FrameDefinitionPoint
• [x] Created FrameDefinitionMidpoint component
  • [x] A FrameDefinitionPoint that sockets to two FrameDefinitionPoints to compute a midpoint
• [x] Created FrameDefinitionEdge component
  • [x] This is the "virtual edge" base API
  • [x] It enables the UI to ask "is the given component an edge?"
• [x] Created FrameDefinitionPointToPointEdge component
  • [x] A FrameDefinitionEdge that sockets two FrameDefinitionPoints to compute an edge vector
• [x] Created FrameDefinitionCrossProductEdge component
  • [x] A FrameDefinitionEdge that sockets two FrameDefinitionEdges to compute a new cross-product (plane) edge
  • [ ] It is still not clear where within the scene the edge should be rendered
• [x] LandmarkDefinedFrame
  • [x] Should socket two FrameDefinitionEdges
  • [x] And an origin VirtualPoint
  • [x] And ask the user which axes to assign to which edges (+cross product)
  • [x] And it should figure out any undefined edge (e.g. if it asks the user for two edges)
• [x] "Focus camera on this"
• [x] Undo/redo keybinds
• [x] Deletion support
  • [x] LIMITED: can only delete objects that have nothing attached to them, so the user has to delete one-by-one
• [x] Good default colors (i.e. frame edges are red,blue,green color-coded and other elements use non-RGB colors that are unique enough)
• [ ] Scene scale factor support
  • [x] LIMITED: the edges in the frame definition UI aren't scaled (#712)
• [x] Make landmarks not use offset frames (they are stations and do not need it)
• [x] Figure out how the frame definition UI feeds into the existing editor UI
  • [ ] LIMITED: There is an export button in the edit menu that should be moved somewhere more obvious (#713 )
• [ ] DROPPED: Add 'non participating landmark' support into warping UI?
  • [ ] DROPPED because of insufficient time

Feature Specification List

Note: this was written pre-development: see this are more of a wishlist

• [x] Can import multiple meshes:

  • [ ] And there is an obvious (e.g. a dropdown) mechanism for flipping between each mesh
  • [ ] And the user can change between meshes when (e.g.) defining frames/edges that span between meshes
  • [ ] And the user can change between meshes when defining frames
  • [x] And the user can change the mesh's scale/rotation/translation within its own cartesean coord system?
  • Original Grood paper says something like "Mesh's coordinate systems should try to align the translation reference point and reference lines along the mesh coordinate system"

• [x] Ability to define named 3D locations in mesh-space (i.e. "basic points"):

  • [ ] They can be imported from an external (e.g. CSV) file
  • [ ] DROPPED: insufficient time (#714)
  • [x] And their location can be edited by typing the (mesh-space) location in an input box in the UI
  • [x] And their location can be set on addition by raycasting the user's mouse onto the surface of the mesh
  • [ ] And a workflow exists that enables moving an existing point via raycasted placement
  • [ ] DROPPED: insufficient time
  • [x] And their location can be manually modified via a gizmo
  • [ ] And they can be (de)serialized to the frame definition file
  • [ ] DROPPED: no frame file support

• [x] Ability to define named scene elements that may host points (i.e. "virtual point hosts"):

  • [x] E.g.: user can add a Sphere and use its center point in edge definition
  • [x] E.g.2: user can define a Midpoint, which is dependent on two other virtual points, and then use that midpoint in edge definition
  • [ ] And they can be (de)serialized to the frame definition file (DROPPED: no frame file support)

• [x] Ability to define directed and named edges between two (basic/virtual) points:

  • [x] User can select two points via the 3D scene, or from a point list
  • [x] And the endpoints of the edge can be swapped
  • [x] And the edge can be renamed
  • [ ] And they can be (de)serialized to the frame definition file (DROPPED: no frame file support)

• [x] Ability to define edges by cross-producting two other edges? (might be useful for reference lines?)

• [x] Ability to define named, ordered, and specialized frames from the edges:

  • [ ] And the user can optionally specify a parent dependency
  • [x] And there is a way to define the reference line
  • [x] And there is a way to define the translation reference line (or two points)

Example Frame Definition Cases

Knee (Grood, 1983)

Tibia (Frame, Body-Fixed Axis (e3), and Reference Axis (e3r)

e3 is defined as passing midway between two intercondylar eminences proximally and through the center of the ankle distally... etc.

Rough dependency diagram (gist):

Femur (Frame, Body-Fixed Axis (e1) and Reference Axis (e1r)

In the femur, the body fixed axis is chosen so that rotations about it correspond to the clinical motion of flexion-extension.

Rough dependency diagram (gist)

Knee Joint (excl. translations)

Rough dependency diagram (gist)

Academic References

• Seminal paper (Grood/Suntay): https://doi.org/10.1115/1.3138397
• ISB Standards (Main Page): https://isbweb.org/activities/standards
  • Contains links to most of the primary resources
  • Global coordinate systems: https://media.isbweb.org/images/documents/standards/Wu%20and%20Cavanagh%20J%20Biomech%2028%20(1995)%201258-1261.pdf
  • Ankle/hip/spine: https://media.isbweb.org/images/documents/standards/Wu%20et%20al%20J%20Biomech%2035%20(2002)%20543%E2%80%93548.pdf
  • Hip commitee: https://media.isbweb.org/standards/hip.pdf
  • Spine proposal (which flips Z and Y, because): https://isbweb.org/activities/standards/450-spine-proposal-by-stokes-skalli-plamondon
  • Shoulder/elbow/wrist/hand: https://media.isbweb.org/images/documents/standards/Wu%20et%20al%20J%20Biomech%2038%20(2005)%20981%E2%80%93992.pdf
  • Comments related to the above: https://isbweb.org/activities/standards/451-comments-by-carolyn-anglin
• How to build a Dinosaur (Peter Bishop)
  • link
  • supplemantary material

Software References

• isfm_installer.zip: https://media.isbweb.org/images/documents/standards/isfm_installer.zip
  • Seems to be software for converting between various coordinate systems? (requires Matlab)
• biokinepy: https://pypi.org/project/biokinepy
  • Seems to be a python library that includes some ISB angle support
  • Specifically, biokinepy.segment_cs
  • Originally spotted: https://github.com/klevis-a/true-vs-apparent-axial-rot/blob/public/true_vs_apparent/common/analysis_utils.py#L6
• UECombined: https://github.com/orat/UECombined
  • Seems to be some kind of modelling project
  • But it uses "Nimue Platform", which seems to have some sort of ISB stuff supported? http://www.nimue-platform.de
  • Originally spotted: https://github.com/orat/UECombined/blob/2e93a0179b65b68947562d1558ffb42ddaf863ef/models/UEGeneral2%2BCluster.xml#L89
• Silvereye: https://github.com/juanmalunatic/silvereye
  • IMU tracking software+UI
  • Seems to use ISB systems internally: https://github.com/juanmalunatic/silvereye/blob/c51bcd8e49b0cdd9a66ddae7dbe30b00a541e5a8/silverapp/Silverapp/mainwindow.cpp#L596
• CusToM: https://github.com/anmuller/CusToM
  • Seems to be a custom implementation of a biomechanics engine
  • Mentions using ISB coordinates here: https://github.com/anmuller/CusToM/blob/b1cb8d411aa924c6f7d4cf41ed015959b4412d1e/Examples/VR_pick_and_place/PostProcessingVR.m#L31
• Rofunc: https://github.com/Skylark0924/Rofunc
  • Includes XSens MVN support, which requires decoding ISB angles:https://github.com/anmuller/CusToM/blob/b1cb8d411aa924c6f7d4cf41ed015959b4412d1e/Examples/VR_pick_and_place/PostProcessingVR.m#L31
• PyBiomech: https://github.com/u0078867/PyBiomech
  • Contains some ISB maths: https://github.com/u0078867/PyBiomech/blob/c2955382d54ccc7107ed6fa84e5f4723ea93fec2/src/PyBiomech/kine.py
• Kineticstoolkit: https://github.com/felixchenier/kineticstoolkit
  • Documentation mentions ISB: https://github.com/felixchenier/kineticstoolkit_doc/blob/776750cb39e8b962acc2f0454c070d43b8a8465c/html/_sources/kinematics_joint_angles.md
  • And it's an interesting project, in general
• pykinematics: https://github.com/M-SenseResearchGroup/pykinematics
  • Encodes/decodes ISB angles: https://github.com/M-SenseResearchGroup/pykinematics/blob/master/pykinematics/omc/angles.py#L12
• Spacelib: https://uk.mathworks.com/matlabcentral/fileexchange/1657-spacelib
  • Might not be relevant: seems to be Matlab code for kinematics. Might be worth a quick scan.
• ANZ: https://isbweb.org/resources/software-resources/130-movement-analysis/three-d-analysis-of-human-movement/471-anz
  • Might not be relevant: seems to be gait analysis code. Might be worth a quick scan.
• Kwon3D: http://kwon3d.com/
  • Commercial motion analysis software (old)
• Joint Model Personalization: https://simtk.org/projects/jmp
  • Seems to be an alg for refitting joints based on markers.
• myomotion: https://www.noraxon.com/our-products/myomotion/#1541097720904-fe85b033-bcdd
  • Commercial software that may include ISB stuff
• AnyBody: https://www.anybodytech.com/
  • Commercial alternative to OpenSim, probably includes frame definition workflow
  • Tutorials: https://anyscript.org/tutorials/
• SIMM: https://www.oit.va.gov/Services/TRM/ToolPage.aspx?tid=8538
  • Commercial alternative to OpenSim, probably includes frame definition workflow
• MVN-Analyze: https://www.xsens.com/products/mvn-analyze
  • Contains some references to encoding ISB angles etc.

[JuliaVanBeesel]

This is the link to Peter Bishop's paper "How to build a dinosaur": https://www.cambridge.org/core/journals/paleobiology/article/how-to-build-a-dinosaur-musculoskeletal-modeling-and-simulation-of-locomotor-biomechanics-in-extinct-animals/CE84BB804E697DF47C0A3A367CC16B22 And here is where you can access the supplementary material, including the open access matlab code, which also includes the primitive shape fitting algorithm: https://datadryad.org/stash/dataset/doi:10.5061/dryad.73n5tb2v9

[adamkewley]

Based on a couple of discussions, it looks like the UI needs to support (minimum viable product):

• Loading a mesh in ground
• Defining/importing 3D locations on that mesh (aka "basic landmarks"):
  • Must be name-able, because users will use biologically relevant names for each landmark
  • Must be importable from basic CSV files (e.g. of the type exported by the warp UI)
  • Must include the ability to manually supply the location (i.e. so that computed locations from external software can be pasted in).
  • Must include the ability to freely move the points within the scene
  • Preferably supports raycasted addition (e.g. click somewhere on the mesh to define a surface intersection as a landmark location)
• Defining landmarks that are derived in some way from either the mesh or other landmarks (aka "virtual landmarks"):
  • Must be name-able (same reasons as above)
  • Must supply at least one concrete implementation of the form "between two other landmarks" (non-recursive)
  • Must be coded for extensibility (i.e. other virtual landmark definitions, such as "fit a sphere", or similar, can be added later without requiring too much rework)
  • Preferably supports caching the computed location in the output file, so that there is a recovery route available if external data is deleted by the user (i.e. so that the UI can say "whoops, can't find the external mesh/landmark: want me to use this cached location as a landmark for now?")
• Defining named edges between basic/virtual landmarks
  • Must be name-able, so that (potentially, external) frame definitions can refer to it
  • TODO: is this necessary, or should downstream frame definitions just use the two landmark names instead?
• Defining frames that are derived from the above information:
  • Must be name-able
  • Must support the most basic case of defining two edge direction vectors to derive a plane
  • Preferably supports optionally defining parent frames, so that users can use the UI to get OpenSim-relevant quantities out of the system?

[adamkewley]

e.g. the software would ultimately be defining data that would be serialized like this:

[landmarks]

    [landmarks.l1]
    type = "basic"
    location = [20.0, 0.0, 3.7]

    [landmarks.l2]
    type = "basic"
    location = [20.0, 0.0, 3.7]

    [landmarks.middle]
    type = "midpoint"
    first_landmark = "l1"
    second_landmark = "l2"
    # cached_location = [30.0, 20.0, 10.0]  # TODO: caching for error recovery

    [landmarks.external_l3]
    type = "external"
    external_file = "some_other_landmark_file"
    landmark_name_in_external_file = "l3"
    # cached_location = [30.0, 20.0, 10.0]  # TODO: caching for error recovery

    [landmarks.external_l4]
    type = "external"
    external_file = "some_other_landmark_file"
    landmark_name_in_external_file = "l4"
    # cached_location = [30.0, 20.0, 10.0]  # TODO: caching for error recovery

    # extensibility example: not MVP
    #
    # e.g. would show a sphere in the UI that the user can use to eyeball locations
    #      but it would ultimately resolve to the sphere's origin for frame
    #      calculations
    [landmarks.sphere]
    type = "sphere"
    origin = [0.5, 5.0, 7.0]
    radius = 5.0
    # cached_location = [30.0, 20.0, 10.0]  # TODO: caching for error recovery

[frames]

    [frames.f1]
    type = "grood_suntay"
    e1_start = "external_l3"
    e1_end = "external_l4"
    e3_start = "l1"
    e3_end = "l2"
    # e2 is the common perpendicular "floating" axis (F): F = normalize(cross(e3, e1))

[adamkewley]

The core original paper:

E.S. Grood, A Joint Coordinate System for the Clinical Description of Three-Dimensional Motions: Application to the Knee, 1983

Is what most of the other ISB standards are effectively based on. The original spec appears to be "looser" than its concrete usage in the ISB standards, though, because it defines (e.g.) the two edges ("fixed" axes) potentially on different bodies in different coordinate systems, and it seems to rigidly define angles? Probably needs to be studied before going through each ISB recommendation.

Note: up to p138

[adamkewley]

The commit onslaught above effectively:

• Tried to create a new component+property system that doesn't have as many dependencies as OpenSim
  • DROPPED: will require a large amount of engineering to work. Was recommended that we try and do everything in terms of OpenSim::Component
• Created a new FrameDefinitionTab that houses this workflow
• Added right-click callbacks to ModelEditorViewPanel,
  • This is so that context menu customization is possible
  • Because the context menu in the frame definition UI will not be the same as in the general model editor UI
• Added layering support to ModelEditorViewPanel (big refactor)
  • Separates rendering layer, interaction layer, button+gizmo layers
  • Adds API that enables external code to push new layers on top of the viewport (osc::ModelEditorViewPanel::addLayer)
  • This is so that the frame definition UI can push "select N things", "do this", "do that" layers over a viewport in order to prompt a user for relevant information
  • It is also going to be useful in unrelated work (e.g. modal editing, muscle path point placement, etc.)
• Separated ModelEditorViewerPanel's internal code for decoration generation, rendering, hittesting, etc. into separate APIs under src/OpenSimCreator/Graphics (e.g. OverlayDecorationGenerator, OpenSimDecorationGenerator, ComponentAbsPathDecorationTagger, etc.)
  • This is so that the frame definition UI is able to heavily customize what information is seen during one of those layered states (e.g. show the model, but with non-selectable elements faded and not-hittested)
  • It also enables other heavily-customized model viewers from those separate elements
  • (and will be handy in testing, customization, etc. later)
• Uses the above changes to implement a basic "select N components" state in the frame definition UI, which will then be customized for "select other point to create edge" and "select 3 edges to create a frame" workflows.

[adamkewley]

@JuliaVanBeesel / @pepbos / @aseth1 initial prototyping of the frame definition UI is underway and is being continuously deployed to the main branch as a custom tab (press + in OSC's tab bar experimental, OpenSim/FrameDefinition).

The current in-development version needs some work (more actions, easier interaction when moving things around) but the current idea is:

• Use the UI to import mesh(es)
• Attach points (spheres, but can also be ellipsoids etc.) to the mesh
• Create midpoints between points (if desired)
• Create edges between points (if desired)
• Create cross products (planes) between edges
• (NOT YET IMPLEMENTED) create frames from 3 orthogonal edges
• (NOT YET IMPLEMENTED) derive OpenSim quantities from those frame definitions (position, orientation, etc)
• (STILL DREAMING UP) define reference lines, joint coordinates, etc. according to ISB definitions

Demo video:

https://github.com/ComputationalBiomechanicsLab/opensim-creator/assets/4730570/f0849ae2-5895-479a-be3a-8e23214f836e

(green spheres: "landmarks", teal spheres: "midpoints", "green plane": cross product plane - all user defined points are moveable and all derived quantities (edges, cross products, midpoints, etc.) automatically recompute while editing happens.

In-development builds are available on the actions page under the main branch:

• https://github.com/ComputationalBiomechanicsLab/opensim-creator/actions

[adamkewley]

@JuliaVanBeesel / @aseth1 this top-level feature-push issue will now be closed (it will remain here, online, but further work will be done under different issues).

I have updated the original issue post with what parts of the original feature idea have been implemented/DROPPED/LIMITED so that it's clear how development proceeded (the ~50 code commits are harder to read).

The reason some things are DROPPED/LIMITED may be due to time constraints. The delivery date of this feature was scheduled for approximately 2 weeks before TGCS/ISB, so that the overall project, which includes:

• Quick bugfixes
• Adding relevant buttons for opening the UI
• Typos, basic documentation strings
• Publishing a formal OSC release that contains the UI

Can be done in the remaining 1-2 weeks.

Rough project schedule here: https://github.com/orgs/ComputationalBiomechanicsLab/projects/9/views/1

---

I'll repost any important (but, unreleaseable within the desired timeframe) changes/fixes/steps etc. as separate issues. Those issues will then be prioritized according to the general OSC project schedule etc.

[adamkewley]

Individual nice-to-haves were broken out into new issues that will be assigned as part of project development - it might be that a few of the easier ones make this window.