Is this software for Openpose or ST-GCN ?

[xavigiro]

Thanks for sharing this code. We read on the related paper on ECCV 2020 SLRTP Workshop that you used this used to clean up skeletons extracted with ST-GCN, not OpenPose as mentioned in the README.md.

Does the code work for both softwares ?

[hannahbull]

Dear Xavier,

The input of this skeleton cleaning is the OpenPose 2D hand, body and face keypoints at each frame (*_keypoints.json files). The output of this skeleton cleaning are skeleton sequences used as input for the ST-GCN.

I use this to clean sign language videos with multiple people and potentially multiple signers in a scene. This allows me to extract sequences of up to N most likely signers in each scene. OpenPose does not track people across frames, and so this is useful for videos with multiple people.

Hope this helps! Let me know if you have further questions or feedback.

Best, Hannah

Le mar. 1 sept. 2020 à 18:24, Xavier Giro-i-Nieto notifications@github.com a écrit :

Thanks for sharing this code. We read on the related paper on ECCV 2020 SLRTP Workshop https://slrtp.com/papers/full_papers/SLRTP.FP.01.011.paper.pdf that you used this used to clean up skeletons extracted with ST-GCN, not OpenPose as mentioned in the README.md.

Does the code work for both softwares ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hannahbull/clean_op_data_sl/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNGEZOOYVR2SW6OBSMBQXTSDUN2TANCNFSM4QR5LCJQ .

[xavigiro]

Thanks a lot for the prompt and detailed answer.

I had understood that the research reported in the SLRTP 2020 paper was using ST-GCN only. Is this correct ? Or is it that ST-GCN was built on top of OpenPose skeletons as you described in this answer. Maybe these OpenPose skeletons are already available in the MEDIAPI-SKEL [4] dataset and this is why they are not explicitly mentioned in the SLRTP paper ?

Cheers,

Xavi

[hannahbull]

Dear Xavier,

The OpenPose skeletons for MediaPi-Skel are indeed available for download here: https://www.ortolang.fr/market/item/mediapi-skel

As a first step, I create skeleton sequences of signers from the OpenPose keypoints using: https://github.com/hannahbull/clean_op_data_sl https://github.com/hannahbull/clean_op_data_sl

As a second step, I train a ST-GCN (https://github.com/open-mmlab/mmskeleton)

• BiLSTM to recognise subtitle-units using these skeleton sequences as input.

I'll clarify this better for the camera-ready version :).

Best, Hannah

Le mar. 1 sept. 2020 à 19:01, Xavier Giro-i-Nieto notifications@github.com a écrit :

Thanks a lot for the prompt and detailed answer.

I had understood that the research reported in the SLRTP 2020 paper https://slrtp.com/papers/full_papers/SLRTP.FP.01.011.paper.pdf was using ST-GCN only. Is this correct ? Or is it that ST-GCN was built on top of OpenPose skeletons as you described in this answer. Maybe these OpenPose skeletons are already available in the MEDIAPI-SKEL [4] https://www.aclweb.org/anthology/2020.lrec-1.743/ dataset and this is why they are not explicitly mentioned in the SLRTP paper ?

Cheers,

Xavi

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hannahbull/clean_op_data_sl/issues/1#issuecomment-685001664, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNGEZNJ7IOBLXUZJ6SQFCLSDUSFXANCNFSM4QR5LCJQ .

[xavigiro]

Thanks Hannah, this makes total sense.

May I ask if you ever tried to train the BiLSTM directly on the OpenPose skeletons or, similarly, whether you measured the gain by introducing ST-GCN in the pipeline ?

Great work !

Xavi

[hannahbull]

Hi Xavier,

No, I didn't try that. I cleaned the OpenPose skeletons to overcome the major tracking issue that person #1 in frame N is not necessarily person #1 in frame N+1 in the presence of multiple people. I used ST-GCN as it also exploits the graph structure of the skeleton keypoints, rather than inputting the keypoints as a long vector and ignoring the relationships between joints. Moreover the ST-GCN paper demonstrates superior results on gesture recognition tasks with skeleton keypoints.

Best, Hannah

Le mar. 1 sept. 2020 à 20:18, Xavier Giro-i-Nieto notifications@github.com a écrit :

Thanks Hannah, this makes total sense.

May I ask if you ever tried to train the BiLSTM directly on the OpenPose skeletons or, similarly, whether you measured the gain by introducing ST-GCN in the pipeline ?

Great work !

Xavi

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hannahbull/clean_op_data_sl/issues/1#issuecomment-685048291, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHNGEZIL6O2W3CYG5TMPQMDSDU3HZANCNFSM4QR5LCJQ .

[xavigiro]

Excellent, thanks for all the details.