Pose similarity for two video seqeunces ?

[timtensor]

Hi , I am wondering if there is a way to compare two video sequences doing same actions but in one of the action some are a bit exxagerated /wrong

For example , we have reference video and test video. Lets say reference has 10 jumps and test also has 10 jumps. But out of the 10 jumps , 3 were not proper. Is it possible to extract out the 3 jumps with the method ??

[freshsomebody]

Hello @timtensor , thank you very much for discovering issues when using this package.

Please allow me to clarify the questions again. Do you mean there are some improper poses existing in your video, and you would like to ask whether posenet-similarity has a method to extract them out? OR Do you mean you got some incorrect results from posenet-similarity when you compared 2 videos?

Thank you!

[timtensor]

Hi , I was trying to use the same method to extract bad poses. That is lets say i have a reference video doing lets say 10 squats. In the test video i do the 10 squars but 3 of them are in correct. So i was wondering if it was possible to extract those three poses ? Using weighted cosine matching and giving a threshold

[freshsomebody]

Hi @timtensor , thank you! This is a great question.

posenet-similarity does not have a method to directly extract poses with low similarity from 2 videos, but you may use it to compare each frame of the 2 videos.

The possible scenario could be that you use PoseNet to get series of pose objects from video1 and video2. Assume the pose object series of video1 is pV1, and video2's is pV2. You may get the the array of pose similarities by:

const poseSimilarities = pV1.map((pose1, index) => {
  const pose2 = pV2[index]
  return (pose2 ? poseSimilarity(pose1, pose2) : null)
})

Once you have the array of pose similarities, you can easily extract indexes of those poses with similarities under the threshold you set.

let incorrectPoseIndexes = []
poseSimilarities.forEach((similarity, index)=> {
  if (similarity && similarity < THRESHOLD_YOU_SET) {
    incorrectPoseIndexes.push(index)
  }
})

Please note that the example above just showed a rough idea about how posenet-similarity might help you when comparing 2 videos. You may still need to think about problems like:

• How to sample a video and use PoseNet to estimate pose objects within
• How to align the start and end time of a pose you would like to compare from 2 videos