Closed editorialbot closed 2 years ago
editorialbot
commented
2 years ago Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.
For a list of things I can do to help you, just type:
@editorialbot commandsFor example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@editorialbot generate pdfReference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1109/TPAMI.2019.2929257 is OK
- 10.1109/TBME.2007.901024 is OK
- 10.1109/ICCV.2017.256 is OK
- 10.1016/j.jbiomech.2021.110665 is OK
- 10.1016/j.celrep.2021.109730 is OK
- 10.1038/s41593-018-0209-y is OK
- 10.3390/s21196530 is OK
- 10.3390/s22072712 is OK
- 10.1016/j.gaitpost.2007.07.007 is OK
- 10.48550/arXiv.2012.13392 is OK
MISSING DOIs
- None
INVALID DOIs
- NoneSoftware report:
github.com/AlDanial/cloc v 1.88 T=0.69 s (784.7 files/s, 202190.9 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
XML 113 0 8367 125018
Python 22 961 1319 2171
Markdown 2 120 0 530
JSON 400 0 0 400
TeX 1 10 0 109
TOML 2 24 12 106
-------------------------------------------------------------------------------
SUM: 540 1115 9698 128334
-------------------------------------------------------------------------------
gitinspector failed to run statistical information for the repositoryWordcount for paper.md is 1027
editorialbot
commented
2 years ago :point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
jonmatthis
commented
2 years ago 
lambdaloop
commented
2 years ago 
cvhammond
commented
2 years ago 
davidpagnon
commented
2 years ago Dear reviewers,
I suppose it is not really advised to change the code once it's under review, but I wanted to make Poe2Sim available through pip. It made me slightly modify the file structure, although none of the core code has been changed. I tested it all, updated the Readme, and pushed the changes today.
I don't think it will make a big difference, but if you already cloned the repository, you should probably git pull it or clone it again to make sure you have the last version.
Best regards,
David
danasolav
commented
2 years ago @lambdaloop @jonmatthis @CVHammond How is your review going?
lambdaloop
commented
2 years ago I'm sorry, give me another week to complete the review. I'll have it done by Friday April 3rd.
cvhammond
commented
2 years ago see this issue about potential syntax issue
lambdaloop
commented
2 years ago See https://github.com/davidpagnon/pose2sim/issues/3 about an issue for running code on Linux
lambdaloop
commented
2 years ago See https://github.com/davidpagnon/pose2sim/issues/4 about underspecified community guidelines
lambdaloop
commented
2 years ago See https://github.com/davidpagnon/pose2sim/issues/5 about specifying the version of OpenSim
lambdaloop
commented
2 years ago Here is my review of the paper. It doesn't really fit as an issue in the Pose2Sim repository. Let me know if this should be placed somewhere else.
Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
The summary is currently overly technical for a non-specialist audience. In particular, OpenPose and OpenSim are very specific tools that most people would not have heard of. It would be better to describe them, such as: "from an OpenPose input, namely the 2D positions of joints from videos, to an OpenSim result, namely the position of a 3D skeletal model" (it should be rephrased by the authors to match their view of this software)
I think the reference "User/Config.toml" file is too much detail for the Summary.
Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
The paper briefly describes why markerless tracking is useful, but this is not actually the need that the package meets! The purpose of the software is in the title, to translate OpenPose outputs OpenSim outputs. This is indeed a useful contribution, but the reason is buried in the current statement. As it reads currently, it seems that the best contribution is you could obtain 3D angles from the OpenSim output. Why is this useful? It would help to detail why 3D angles are useful above 3D positions. Furthermore, as I understand, OpenSim allows you to further estimate forces, constrain movements biomechanically, produce cool animations of skeletons, and perhaps even provide a framework for interpreting the motion in terms of muscle activities. Very little of this is clear from the statement of need, and the paper would really benefit from reframing the full statement to make the advantage of bringing kinematic data into OpenSim clearer.
Do the authors describe how this software compares to other commonly-used packages?
There is really not much detail about how the other packages compare. There is a sentence about Theia3D and Anipose computing 3D angles, but no further description. As a user, why should I use Pose2Sim over these other packages? There should be a longer description comparing these packages, even at least a sentence about how Pose2Sim does something that those do not.
Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
The paper has several typos. Generally, the whole paper could benefit from another pass to make it more clear and concise. Examples of some of these issues:
Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
lambdaloop
commented
2 years ago For functionality, I was able to reproduce the run the demo after fixing the issues pointed out in the issues above.
I currently only use Linux and as far as I can tell I need to compile OpenSim from source to use it, so I wasn't able to test this part of the package. I hope one of the other reviewers can address this in particular!
I'll conclude by saying that bringing markerless 3D tracking data into OpenSim is something I've thought about for a while but didn't know how to do, so I'm excited to see this open-source pipeline to achieve this!
cvhammond
commented
2 years ago @lambdaloop makes great points, I'd like to follow up with:
The software does produce a usable TRC file that works with the given BODY_25B model. Have you checked with OpenSim regarding the best way to produce TRC files according to their specifications? I was told by Ayman Habib to use STOAdapter from OpenSim, but of course that would require OpenSim as a dependency. Manual string concatenation, as you've done, is common, but also unsafe to specification changes by OpenSim. With that being said RCNL (my lab) has also implemented a manual string concatenation function for TRC files in some of our software. Your technique for TRC file generation is likely fine.
Also, the markers in the OpenSim model don't appear to be in the correct positions for research grade use of OpenSim. The elbow marker is the most obvious example. Since the paper includes announcing the full body OpenSim model, the marker positions should better approximate the OpenPose identified locations or justify their positions in some way.
Statement of need in the GitLab documentation is not clear enough. It does not include target audience (biomechanists? researchers?). Also, "OpenPose input to an OpenSim result" is not correct. Isn't the starting point for the software OpenPose's output? The software produces a TRC file or OpenSim marker position data file rather than an "OpenSim result".
This project should include a test suite with at least unit testing, but could also include an end-to-end test to quickly identify when underlying function implementations from other libraries are changed. GitHub Actions or the GitLab equivalent can be used and the badge saying if the package is passing/failing can be added to the top of the README.
As @lambdaloop said, the community guidelines are lacking. Beyond the dual-repository issue, it's not intuitive to click on the 'issues' badge to go to the GitHub repository; a more overt indication of how to report issues should be added. The to-do list at the bottom should be linked to from the top of the README and expanded in a more actionable way to facilitate contributions. Prioritizing/reducing the number of to-dos could also be valuable. Regarding seeking support, explicitly give instructions on what to do (visit FAQ, contact me, etc).
Line 9 - OpenPose should be capitalized? Openpose is used a few times in the article Line 16 - this tool does not include functionality for scaling and running inverse kinematics via OpenSim. It only publishes a TRC file.
Lines 20-25: Commentary on if marker-based kinematics is the gold-standard in human movement tracking is beyond the scope of this paper, especially without citations. This section would do best to focus on the limitations of marker-based tracking to justify markerless motion analysis.
Theia3D has released validation studies with measured accuracy. This can be used to better state how your package compares to other commonly-used packages.
Check to see if it's appropriate to cite OpenSim everytime it is referenced, from other papers I've read, usually you cite the first usage and remove the others. OpenSim has two paper to cite, one from 2007 and one from 2018. You can view their citations on the OpenSim download page.
davidpagnon
commented
2 years ago Dear reviewers, Thank you for your insightful reviews that we are going to take into account soon. We have transferred the gitlab to this address: https://github.com/perfanalytics/pose2sim in order to make issues and contributing more coherent.
Dear @danasolav and @editorialbot, is it possible to change the address of the repository so that it is correclty referenced and in order for me to generate the pdf? Thanks in advance!
jonmatthis
commented
2 years ago I'm working on my review in the form of a comment thread in Issue#9 on the main repo - https://github.com/perfanalytics/pose2sim/issues/9
jonmatthis
commented
2 years ago For the record (regarding the JOSS review process) -
I wish I was able to open a comment thread from my own Checklist comment and include my comments there.
The method of "I'm Going to write everything up in a great big text blob and plop it in one big comment" feels inconvenient for all involved.
Maybe in the future, the check list could also auto-generate a linked Issue in the target repository?
jonmatthis
commented
2 years ago Ok, I'm finished with my review!
My specific points are listed in - https://github.com/perfanalytics/pose2sim/issues/9
And I generally echo the comments from @lambdaloop and @CVHammond
Marker-based motion capture is the gold standard for generating accurate recordings of human movement. Your concerns are completely valid, but the remains that marker-based mocap is the most accurate method currently available for recording 3d movement.
I think you should alter that section to recognize that fact, but maintain your concerns about marker placement and the potential for innacurate measurements that could arise as a result. Basically something to the effect of "Marker based mocap is the most accurate method currently available, but it's got all theses problems so we should be trying to find better alternative solutions"
A lot of the formatting in the main document is a bit sloppy. A lot of things that are supposed to be 'bullet points' but are in fact just hyphens, and no spaces between normal text and those not-quit-bulleted lists (e.g. Line 64-76) I'd like to see a bit more polish in the formatting before final acceptance, if possible.
Also, maybe draw a border between the images? At the moment it is very difficult to parse what I'm seeing. e.g. in the upper right panel - is that a bunch of people in the same area? or multiple images super imposed on each other?
I wouldn't mind if you broke Figure 1 up into multiple figures, if that makes it easier to clarify what is happening in each panel
davidpagnon
commented
2 years ago Dear reviewers,
Thank you for your input! I'm sorry about my late answer, I had a difficult deadline to meet. Anyway, here it is!\ I answered to the issues you have posted on the repository (feel free to follow up if you're not satisfied), now here are my answers to each of the reviewers in separate posts.
Also, I'm not sure you're going to be able to compile the paper with the @editorialbot generate pdf command since we've changed our repository to github, so please find attached the pdf.
davidpagnon
commented
2 years ago Answers to @lambdaloop
Summary
The summary is currently overly technical for a non-specialist audience. In particular, OpenPose and OpenSim are very specific tools that most people would not have heard of. It would be better to describe them, such as: "from an OpenPose input, namely the 2D positions of joints from videos, to an OpenSim result, namely the position of a 3D skeletal model" (it should be rephrased by the authors to match their view of this software)
Rephrased to:
"Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods.\
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles)."
I think the reference "User/Config.toml" file is too much detail for the Summary.
Agreed.
A statement of need
Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
The paper briefly describes why markerless tracking is useful, but this is not actually the need that the package meets! The purpose of the software is in the title, to translate OpenPose outputs OpenSim outputs. This is indeed a useful contribution, but the reason is buried in the current statement. As it reads currently, it seems that the best contribution is you could obtain 3D angles from the OpenSim output. Why is this useful? It would help to detail why 3D angles are useful above 3D positions. Furthermore, as I understand, OpenSim allows you to further estimate forces, constrain movements biomechanically, produce cool animations of skeletons, and perhaps even provide a framework for interpreting the motion in terms of muscle activities. Very little of this is clear from the statement of need, and the paper would really benefit from reframing the full statement to make the advantage of bringing kinematic data into OpenSim clearer.
I removed the section about gold standards in kinematics, which was beyond the scope of this paper indeed.
The statement of need has been completely rephrased: " For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement.
The emergence of markerless kinematics opens up new possibilities. Indeed, the interest in deep learning pose estimation neural networks has been growing fast since 2015 [@Zheng_2022], which makes it now possible to collect accurate and reliable kinematic data without the use of physical markers. OpenPose, for example, is a widespread open-source software which provides 2D joint coordinate estimations from videos. These coordinates can then be triangulated in order to produce 3D positions. Yet, when it comes to biomechanic analysis of human motion, it is often more useful to obtain joint angles than absolute positions. Indeed, joint angles allow for better comparison among trials and individuals, and they represent the first step for other analysis such as inverse dynamics. OpenSim is an other widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed muskuloskeletal model, scale it to each individual subject, and perform inverse kinematics with customizable biomechanical constraints. It provides other features such as net joint moments calculation or individual muscle forces resolution, although this is out of the scope of our contribution.
The goal of Pose2Sim is to build a bridge between the communities of computer vision and biomechanics, by providing a simple and open-source pipeline connecting the two aforementioned state-of-the-art tools, OpenPose and OpenSim.
Pose2Sim has already been used and tested in a number of situations (walking, running, cycling, balancing, swimming, boxing), and published in peer-review scientific publications [@Pagnon_2021; @Pagnon_2022] assessing its robustness and accuracy. The combination of its ease of use, customizable characteristics, and robustness and accuracy makes it promising, especially for "in-the-wild" sports movement analysis.
So far, little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, two softwares are worth being mentionend. Anipose [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. Theia3D [@Kanko_2021] provides a software for human gait kinematics from videos. Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022]. "
State of the field
Do the authors describe how this software compares to other commonly-used packages?
There is really not much detail about how the other packages compare. There is a sentence about Theia3D and Anipose computing 3D angles, but no further description. As a user, why should I use Pose2Sim over these other packages? There should be a longer description comparing these packages, even at least a sentence about how Pose2Sim does something that those do not.
This paragraph has been rephrased to: "Little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, two softwares are worth being mentionend. Anipose [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. Theia3D [@Kanko_2021] provides a software for human gait kinematics from videos. Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022.]"
Quality of writing
Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
The paper has several typos. Generally, the whole paper could benefit from another pass to make it more clear and concise. Examples of some of these issues:
- "fall-off" on line 23 should be "fall off".
- "...networks are has been growing" on line 27 should be "has been growing"
- "it is allegedly easy to learn" on line 37 -- remove "allegedly" for clearer sentence
- "the 3D point coordinates are finally constrained" on lines 37/38 --> remove "finally"
- "AniPose" should be "Anipose" (line 31)
Thank you for pointing out these typos! I made an other pass, with the help of your and the other reviewers' comments.
References
Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
- The Pagnon et al 2022 paper seems to be missing a title?
Fixed.
- I think it would help to cite a paper or two about how someone would estimate 3D angles with a marker-based approach in the statement of need.
OpenSim is now further explained and covers this. "OpenSim is an other widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed muskuloskeletal model, scale it to each individual subject, and perform inverse kinematics with customizable biomechanical constraints. "
- The OpenCV paper should be cited
Done.
Functionality
For functionality, I was able to reproduce the run the demo after fixing the issues pointed out in the issues above. I currently only use Linux and as far as I can tell I need to compile OpenSim from source to use it, so I wasn't able to test this part of the package. I hope one of the other reviewers can address this in particular! See davidpagnon/pose2sim#3 about an issue for running code on Linux See davidpagnon/pose2sim#4 about underspecified community guidelines See davidpagnon/pose2sim#5 about specifying the version of OpenSim
Thank you! I took these issues into account. It should now work on linux, as I wrote a Github Action to test the code on all platforms. It is possible to build OpenSim from source for Linux, but it is quite a hassle indeed. \ Community guidelines have been specified, as well as compatible OpenSim versions.
I'll conclude by saying that bringing markerless 3D tracking data into OpenSim is something I've thought about for a while but didn't know how to do, so I'm excited to see this open-source pipeline to achieve this!
Thank you for your valuable and appreciated feedback!
davidpagnon
commented
2 years ago Answers to @CVHammond
Functionality:
The software does produce a usable TRC file that works with the given BODY_25B model. Have you checked with OpenSim regarding the best way to produce TRC files according to their specifications? I was told by Ayman Habib to use STOAdapter from OpenSim, but of course that would require OpenSim as a dependency. Manual string concatenation, as you've done, is common, but also unsafe to specification changes by OpenSim. With that being said RCNL (my lab) has also implemented a manual string concatenation function for TRC files in some of our software. Your technique for TRC file generation is likely fine.
I did follow these guidelines on how to produce a TRC file, however I wasn't aware of the STOAdapter class, thanks! I'm not sure I should use it though, as it is written in C++ and embedding it would require quite some heavy modifications to the code, which I would like to keep light. This seems to be a rather solid standard now, however you're right that if it does change, I'll have to change my string concatenation accordingly!
Also, the markers in the OpenSim model don't appear to be in the correct positions for research grade use of OpenSim. The elbow marker is the most obvious example. Since the paper includes announcing the full body OpenSim model, the marker positions should better approximate the OpenPose identified locations or justify their positions in some way.
This is a very valid remark, however this has been done deliberately.\ OpenPose has likely not been annotated by researchers with anatomical domain knowledge, as Needham points out: "Crowd-sourced datasets are unlikely to have been labeled with the underlying anatomical structures in mind, which may reduce the quality of the ground-truth data" Needham 2022: The accuracy of several pose estimation methods for 3D joint centre localisation.\ I also noticed it while displaying marker-based 3D joints (bright green) in comparison to triangulated OpenPose 2D keypoints (pink). In particular, the elbow was quite off, probably because a non trained person will tend to think the joint center is at the center of the arm section, while it is actually off-centered: there is generally more mass in the front than in the back of the elbow. As a consequence, I had to take this into account in order to obtain accurate kinematics, and to offset some 3D keypoints on the model as regard to "true" joint centers. \ Of course, results would probably be greatly improved if images were correctly labelled in the first place! This is a known issue and will probably be fixed sometimes in the future.
Statement of Need - Documentation:
Statement of need in the GitLab documentation is not clear enough. It does not include target audience (biomechanists? researchers?).
This has been rephrased to make it clearer. I believe that it is now self-explanatory that the target audience is biomechanists and researchers, but feel free to tell me otherwise.
"Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods.\
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles)."
Also, "OpenPose input to an OpenSim result" is not correct. Isn't the starting point for the software OpenPose's output? The software produces a TRC file or OpenSim marker position data file rather than an "OpenSim result".
The tool provides not only a TRC file, but also a full-body .osim model, as well as scaling and inverse kinematics setup files. It also provides explanations on how to run these steps and get OpenSim .mot results. As triangulated keypoints are not dependent on the operator nor on the subject (unlike with physical markers), there is no obvious need for adjusting model markers, nor for computing different scale factors. So these setup files should be valid in any other experiment and can be taken as is. Accuracy has been tested against marker-based methods Pagnon 2022: Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy
However, I do agree that the OpenSim section was buried too far down: I bumped it up to the "Demo" section.
Automated Testing:
This project should include a test suite with at least unit testing, but could also include an end-to-end test to quickly identify when underlying function implementations from other libraries are changed. GitHub Actions or the GitLab equivalent can be used and the badge saying if the package is passing/failing can be added to the top of the README.
Thank you for suggesting it! A github action has been added to test the code on push and pull request on all platforms, from python 3.7 to python 3.10.
Community Guidelines:
As @lambdaloop said, the community guidelines are lacking. Beyond the dual-repository issue, it's not intuitive to click on the 'issues' badge to go to the GitHub repository; a more overt indication of how to report issues should be added. The to-do list at the bottom should be linked to from the top of the README and expanded in a more actionable way to facilitate contributions. Prioritizing/reducing the number of to-dos could also be valuable. Regarding seeking support, explicitly give instructions on what to do (visit FAQ, contact me, etc).
I totally agree. We initially wanted to host the code on our lab's gitlab, but it doesn't allow external users for creating issues or offering any contributions. The code is now hosted on github, and guidelines have been specified. The To-do list has been linked to the top of the Readme, and prioritized. See answer to @labdaloop's review for more details.
Summary:
Line 9 - OpenPose should be capitalized? Openpose is used a few times in the article Line 16 - this tool does not include functionality for scaling and running inverse kinematics via OpenSim. It only publishes a TRC file.
The typo is now corrected.\ Concerning OpenSim, please refer to the answer above.
Statement of Need - Paper:
Lines 20-25: Commentary on if marker-based kinematics is the gold-standard in human movement tracking is beyond the scope of this paper, especially without citations. This section would do best to focus on the limitations of marker-based tracking to justify markerless motion analysis.
Thank you for your remark, it has now been fully rewritten.
"For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement.
The emergence of markerless kinematics opens up new possibilities. [...]"
State of the field:
Theia3D has released validation studies with measured accuracy. This can be used to better state how your package compares to other commonly-used packages.
"Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022]. "
References:
Check to see if it's appropriate to cite OpenSim everytime it is referenced, from other papers I've read, usually you cite the first usage and remove the others. OpenSim has two paper to cite, one from 2007 and one from 2018. You can view their citations on the OpenSim download page.
Thank you, I did as you suggested.
see this issue about potential syntax issue: https://github.com/davidpagnon/pose2sim/issues/2
Thank you, this has been fixed accordingly.
davidpagnon
commented
2 years ago Answers to @jonmatthis.
I answered the rest of your feedback directly on this issue: https://github.com/perfanalytics/pose2sim/issues/9
Specifically regarding the commentary on the use of marker-based motion capture as a "gold standard" - Marker-based motion capture is the gold standard for generating accurate recordings of human movement. Your concerns are completely valid, but the remains that marker-based mocap is the most accurate method currently available for recording 3d movement. I think you should alter that section to recognize that fact, but maintain your concerns about marker placement and the potential for innacurate measurements that could arise as a result. Basically something to the effect of "Marker based mocap is the most accurate method currently available, but it's got all theses problems so we should be trying to find better alternative solutions"
I rephrased this part to: "For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement. The emergence of markerless kinematics opens up new possibilities."
Please put a bit of polish into the formatting before publication A lot of the formatting in the main document is a bit sloppy. A lot of things that are supposed to be 'bullet points' but are in fact just hyphens, and no spaces between normal text and those not-quit-bulleted lists (e.g. Line 64-76) I'd like to see a bit more polish in the formatting before final acceptance, if possible.
Bullet points now replace hyphens.
Figure 1 needs a caption, ideally providing a brief summary of what is happening in each panel.
The caption is now more explanatory: "Pose2Sim full pipeline: (1) OpenPose 2D joint detection; (2i) Camera calibration; (2ii–iv) Tracking the person of interest, Triangulating his coordinates, and Filtering them; (3) Constraining the 3D coordinates to a physically consistent OpenSim skeletal model."
Also, maybe draw a border between the images? At the moment it is very difficult to parse what I'm seeing. e.g. in the upper right panel - is that a bunch of people in the same area? or multiple images super imposed on each other? I wouldn't mind if you broke Figure 1 up into multiple figures, if that makes it easier to clarify what is happening in each panel
I drew a clearer border, and with the caption I believe the figure is clear enough. Feel free to tell me if you still think we need to break it into several figures!
danasolav
commented
2 years ago @davidpagnon Thank you for the detailed responses to the reviewer's comments. @lambdaloop @jonmatthis @CVHammond , could you please take a look to see if you are satisfied with these answers and revisions, and update your checklists accordingly or raise new questions/comments if you have any? Thanks!
jonmatthis
commented
2 years ago I am satisfied with these answers. I have no additional requests and am happy to approve this project.
danasolav
commented
2 years ago @jonmatthis thank you for confirming. Could you please check the missing boxes in your checklist? @lambdaloop and @CVHammond , could you please take a look to see if you are satisfied with @davidpagnon's answers and revisions, and update your checklists accordingly or raise new questions/comments if you have any?
jonmatthis
commented
2 years ago Done! Sorry about that, I thought I already did that 😅
cvhammond
commented
2 years ago The changes made look great! The new GitHub repository looks professional and I like the Pose2Sim icon (whether it was made in jest or not, I won't know :sweat_smile:). Your responses to my concerns were more than satisfactory and I've updated my checklist.
lambdaloop
commented
2 years ago I am satisfied with the answers and would like to approve this project. Thank you for contributing this to the open source community!
davidpagnon
commented
2 years ago Thank you all for your feedback, it was truly useful and pertinent!
@CVHammond, to be honest, I know close to nothing about Javascript, and I didn't even know about Jest. Do you refer to the testing, or to the icon? I wrote the tests with a github action, and I drew the Icon with Inkscape (trying to assemble the O of OpenSim with the gymnastic figure of OpenPose 🙂 ).
danasolav
commented
2 years ago danasolav
commented
2 years ago @editorialbot generate pdf
editorialbot
commented
2 years ago :warning: An error happened when generating the pdf. Paper file not found.
danasolav
commented
2 years ago @editorialbot set https://github.com/perfanalytics/pose2sim as repository
editorialbot
commented
2 years ago Done! repository is now https://github.com/perfanalytics/pose2sim
danasolav
commented
2 years ago @editorialbot generate pdf
editorialbot
commented
2 years ago :point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
danasolav
commented
2 years ago Hi @davidpagnon,
I have a few comments regarding the paper. Please address them, and then we can proceed to acceptance:
Thanks, Dana
davidpagnon
commented
2 years ago Thank you!
Version number is v2.0.1, and here is the Zenodo DOI: 
I will address your points sometime this week!
davidpagnon
commented
2 years ago Thank you Dana, you raised very good points, and you may be right that the internal functioning may need to be more detailed. I elaborated much more on the "Pose2Sim core" section, which is now called "Pose2Sim method details".
- According to the OpenPose page, they allow for single-person 3D pose reconstruction and estimation, and not only 2D. Could you please explain in the paper, how your solution differs from their 3D reconstruction and why it is necessary?
This is a good question, thanks for asking! It is now broached in the statement of need:
"These coordinates can then be triangulated in order to produce 3D positions. Aside from Pose2Sim, a number a tools are available in this regard: the experimental OpenPose 3D reconstruction module [@Hidalgo_2021], the FreeMoCap Python and Blender toolbox [@Matthis_2022], or the pose3d Matlab toolbox [@Sheshadri_2020]. Yet, when it comes to the biomechanical analysis of human motion, it is often more useful to obtain joint angles than joint center positions in space."
and in the Triangulation section:
"Pose2Sim triangulation is robust, largely because instead of using classic Direct Linear Transform (DLT) [@Hartley_1997], we propose a weighted DLT, i.e., a triangulation procedure where each OpenPose keypoint coordinate is weighted with its confidence score [@Pagnon_2021]. Other parameters can be specified, such as: [..]"
- Line 26: Do you have a reference to the statement that markers hinder natural movement
[Colyer, Sports Med - Open, 2018] is now cited.
- Line 33: The XYZ positions of what? the joint centers?
Replaced with "joint center positions in space"
- Can you elaborate more on the camera calibration procedure? Which calibration method is used (with references)?
This has been detailed in the Calibration section: "The user can indicate whether cameras are going to be calibrated with a checkerboard, or if a preexisting calibration file such as provided by a Qualisys system will simply be converted.
If checkerboard calibration is chosen, the number of corners and the size of the squares have to be specified. In this case, the operator needs to take at least 10 pictures or one video per camera of the checkerboard, covering as much as the field of view as possible, with different orientations. Corners are then detected and refined with OpenCV. Detected corners can optionally be displayed for verification. Each camera is finally calibrated using OpenCV with an algorithm based on [@Zhang_2000]. The user can choose the index of the image which they want to be used as a reference for calculating extrinsic parameters."
- Can you clarify what "Tracking of the person viewed by the most cameras" means? What happens if one person is viewed by most cameras in some time frames of the video and another in other time frames?
It has been changed to "Tracking the person of interest", and further explained in the "Tracking" section. Pose2Sim does not currently support multi-person triangulation, although it's in the to-do-list.
"One needs to differentiate the people in the background from the actual subject. The tracking step examines all possible triangulations of a chosen keypoint among all detected persons, and reprojects them on the image planes. The triangulation with the smallest reprojection error is considered to be the one associating the right person from all cameras. If the reprojection error is above a predefined threshold, the process is repeated after taking off one, or several cameras. This happens, for example, if the person of interest has exited the field of a camera, while another person is still in the background. We recommend choosing the neck point or one of the hip points. Indeed, in most cases they are the least likely to move out of the camera views."
- Figure 1: shouldn't the 2.i camera calibration sub-figure show the calibration process (e.g., with checkerboard plates)?
I chose to represent calibration by displaying the cameras views correctly situated in the 3D scene, rather than by showing a checkerboard procedure. Indeed, here calibration can either be done with a checkerboard, or simply by converting a preexisting calibrated file. For instance, the Qualisys system makes it possible to calibrate video cameras with a classic wand procedure.
- Could you please refer to other open-source markerless 3D reconstruction tools, if relevant (e.g., pose3d, freemocap, opencap)?
Added them in the statement of need:
"These coordinates can then be triangulated in order to produce 3D positions. Aside from Pose2Sim, a number a tools are available to do so: the experimental OpenPose 3D reconstruction module [@Hidalgo_2021], the FreeMoCap Python and Blender toolbox [@Matthis_2022], or the pose3d Matlab toolbox [@Sheshadri_2020]. Yet, when it comes to the biomechanical analysis [...]"
"three software applications are worth mentioning. [...] OpenCap [@Ulrich_2022] has recently been released, and offers a user friendly web application working with low-cost hardware. It predicts the coordinates of 43 anatomical markers from 20 triangulated keypoints, and imports them in OpenSim. However, the source code has not been released yet."
- Can you elaborate on the keypoints that need to be tracked in order to successfully reconstruct an OpenSim model, and provide more details on the interpolation and filtering processes?
This is now explicited in the "2D keypoint detection" section: "Only 21 of the 25 keypoints detected by the default OpenPose models are tracked, since eye and ear keypoints would be redundant in the determination of the head orientation."
Interpolation and filtering processes have been detailed in the "Triangulation" and "Filtering" sections. "Once all frames are triangulated, the ones with missing keypoint coordinates are interpolated. The interpolation method can also be chosen among linear, slinear, quadratic, and cubic."
"Different filters can be chosen, and their parameters can be adjusted. The user can choose among a zero-phase low-pass Butterworth filter [@Butterworth_1930] that they can apply either on keypoint positions or on their speeds, a LOESS filter [@Cleveland_1981], a Gaussian filter, or a median filter. Waveforms before and after filtering can be displayed and compared."
- Line 71: it is not clear what you mean by "these setup files can be taken as is". After OpenSim performs the scaling and calculation of joint angles, what are the output results? What about the errors associated with this step (e.g., if the distance between the knee and ankle keypoints changes that is an obvious error, for example)?
I am not sure I understand this point. The output results are given as a .mot file, which is basically a table listing the time-series of the joint angles. OpenSim comes with a biomechanically constrained model, which is more than just joint limits. Bone dimensions are fixed, which means that in your example, the knee flexion/extension angle will be optimized as regards to keypoints positions, but the bone length will not be affected. This is why model-based inverse kinematics is particularly robust, as compared to geometry-based kinematics. In any case, keypoints may occasionally be mislabelled, but this also happens in marker-based capture, in particular due to skin artifact movements.
Maybe you thought I was referring to result files instead of setup ones? This sentence has been deleted, and the OpenSim procedure has been detailed in the next section.
"OpenSim allows for much more accurate and robust results [@Pagnon_2022], since it constrains kinematics to an individually scaled and physically accurate skeletal model. [...] Unlike in marker-based capture, and despite the aforementioned systematic errors, keypoints detection hardly depends on the subject, the operator, nor the context. For this reason, the scaling and the inverse kinematic steps are straight-forward, and the provided setup files require little to no adjusting."
- Line 80: delete "even".
Done.
- Line 86: replace "won't" with "will not".
Done.
- In the bulleted lists, please replace the commas with semicolons and do not capitalize the first word, or alternatively place a period at the end and capitalize the first word.
Done.
- The options described in Pose2Sim core should be better explained in the main text. For example, it might be unclear for the reader what is the connection between OpenPose, AlphaPose, and DeepLabCut. Similarly with regard to the calibration.
As quoted ahead, this has been completely rewritten.
- Some of the Pose2Sim utilities have no context in the text, which makes the list hard to understand. For example, where do you get a c3d file from? what is matplotlib? It would be helpful if you included a diagram of the possible workflows with the file types, so that a reader who has not yet used the software can better understand these lists of tools and features. Otherwise, more explanation is needed when describing these tools.
I trimmed this part a lot, integrated it in the previous section, and added a diagram for clarity.
- The way you refer to file type is not consistent throughout the text (sometimes it's in parentheses, sometimes with a . before the file type, capitalized or not). For example: (json), (h5), .toml, .c3d, trc, .trc, TRC. Please use a consistent method for file types/extensions.
Corrected.
- Please refer in the text to the OpenCV tools you used in your software and reference them, before the acknowledgments.
Done.
danasolav
commented
2 years ago @editorialbot generate pdf
editorialbot
commented
2 years ago :point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
danasolav
commented
2 years ago @editorialbot check references
editorialbot
commented
2 years ago Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1109/TPAMI.2019.2929257 is OK
- 10.1109/TBME.2007.901024 is OK
- 10.1109/ICCV.2017.256 is OK
- 10.1016/j.jbiomech.2021.110665 is OK
- 10.1016/j.celrep.2021.109730 is OK
- 10.1038/s41593-018-0209-y is OK
- 10.3390/s21196530 is OK
- 10.3390/s22072712 is OK
- 10.1371/journal.pcbi.1006223 is OK
- 10.21105/joss.01849 is OK
- 10.1101/2022.07.07.499061 is OK
- 10.1016/j.gaitpost.2007.07.007 is OK
- 10.48550/arXiv.2012.13392 is OK
MISSING DOIs
- 10.1080/10255842.2018.1564819 may be a valid DOI for title: Validation of an OpenSim full-body model with detailed lumbar spine for estimating lower lumbar spine loads during symmetric and asymmetric lifting tasks
- 10.2307/2683591 may be a valid DOI for title: LOWESS: A program for smoothing scatterplots by robust locally weighted regression
- 10.1186/s40798-018-0139-y may be a valid DOI for title: A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system
- 10.1038/s41598-021-00212-x may be a valid DOI for title: The accuracy of several pose estimation methods for 3D joint centre localisation
- 10.1109/tbme.2016.2586891 may be a valid DOI for title: Full-body musculoskeletal model for muscle-driven simulation of human gait
- 10.1109/34.888718 may be a valid DOI for title: A flexible new technique for camera calibration
INVALID DOIs
- None@davidpagnon Thank you for responding to my comments. Can you please check the references with missing DOIs and add them?
davidpagnon
commented
2 years ago Sorry about this... It should be solved now.
danasolav
commented
2 years ago @editorialbot generate pdf
Submitting author: !--author-handle-->@DavidPagnon<!--end-author-handle-- (David Pagnon) Repository: https://github.com/perfanalytics/pose2sim Branch with paper.md (empty if default branch): Version: v0.2.2 Editor: !--editor-->@danasolav<!--end-editor-- Reviewers: @lambdaloop, @jonmatthis, @CVHammond Archive: 10.5281/zenodo.7064026
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@lambdaloop & @jonmatthis & @CVHammond, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danasolav know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Checklists
📝 Checklist for @lambdaloop
📝 Checklist for @jonmatthis
📝 Checklist for @CVHammond