Failed when I ran Triangulation. ValueError: cannot reshape array of size 0 into shape (0,newaxis)

[smwscth]

hello, I completed all the previous steps and tried to run Triangulation, but I can't continue because this error: 'ValueError: cannot reshape array of size 0 into shape (0,newaxis)' I'm not sure why this error happened, should I check the resolution of my video? I need help. Thank you for your answer! this is the error.

[davidpagnon]

Hi, it probably means that the calibration was not good. Did you have a good reprojection error on both intrinsic and extrinsic calibration?

[smwscth]

Hi, it probably means that the calibration was not good. Did you have a good reprojection error on both intrinsic and extrinsic calibration?

I check the pictures. I think my extrinsic pictures aren't good, I'll try again with the new images. Thanks for your answer.

[smwscth]

Hi, it probably means that the calibration was not good. Did you have a good reprojection error on both intrinsic and extrinsic calibration?

Hello, I tried filming again and proceeded Triangulation, and there is another error. I think maybe my calibration board was too small, which caused my failure. Maybe I should change a bigger calibration board or use scene for external parameter calibration. I will try again, but I am not sure what caused the error this time. Thank you very much for your answer. This is the error, along with my calibration of external parameters.

[davidpagnon]

Hi,

Yes, the calibration board is definitely too small, it is supposed to cover as much of the scene as possible. You can measure your bed and table (length, height...), the flooring, etc.

Such a calibration board would be fine for intrinsic calibration though, as long as you are sure the camera does not autofocus or uses a different lens when it is close from the board (like most phones would do).

[smwscth]

Hello, I try to do calibration again by scene, and I think the result is OK. But I still can't run Triangulation, I don't know why this error still happen. Is there a problem when I use openpose?Or is my calibration still not good enough? Thanks for your answer very much, I will try again soon and I hope I can do better.

[davidpagnon]

This is a little surprising.

Two things that come to mind are that:

• triangulation is good, but it is done on a very small volume. It would be better if the calibration could cover about as much space as the person that will move in
• also, 2 cameras is the absolute minimum and you will not benefit from some of the features that make the process particularly robust with more cameras
• in any case, if you want to use only two cameras, your best bet is to position them at about a 30° angle. Right here they are almost looking in parallel direction, which makes it extremely difficult to accurately triangulate, especially considering the variability of current pose estimation models

[conorgren]

Hi, I am also getting a triangulation error when i run my code. Here's an example of my intrinsic and extrinsic pictures for camera 1

[davidpagnon]

Hi, As it is an error I have never encountered, I need a little more information.

• How many cameras did you use, and how were they positioned? (a rough drawing could be easiest).
• How many images were used for intrinsic detection? Did you manage to cover the whole image, including the corners? What was the error in pixel?
• Are your cameras synchronized? Are you sure the checkerboard for the extrinsics was positioned in the exact same place for both cameras? What was the error in pixel and in mm?
• Also, for the record, for extrinsic parameters, the checkerboard should cover as much of the scene as possible, so it should be bigger. That's why I generally advise to to measure stuff omin the scene and calibrate from it. In your case, you would probably have good results if you measured the coordinates of the pieces of furniture around you. You should still get a result without an error but this is not ideal.

[conorgren]

Thanks for the reply

• I used 2 cameras set up next each other like this. They weren't synchronized but i synchronized the footage that was analyzed by clapping and matching the exact frames from each video on top of each other.

• I used 8 intrinsic images for both cameras from the same frames and named them according to the demo file.

• I did not have access to a bigger printer than A4 at the time but i have since printed A3 checkerboards and so will repeat the footage with any corrections you give.
• The extrinsic images were taken from almost the same moment in time (1 frame difference). I only have 1 extrinsic frame for each camera should i add more?

• I do not know how to see the pixel error and so would appreciate that if possible thank you Thanks so much for your help

[conorgren]

here are my calibration errors

Residual (RMS) calibration errors for each camera are respectively [0.221, 0.235, 0.171, 0.191] px, which corresponds to [0.402, 0.445, 0.45, 0.505] mm

person.Association

Mean reprojection error for Neck point on all frames is 176.7 px, which roughly corresponds to 321.8 mm

Hope that helps

[davidpagnon]

So it starts being obviously bad at the personAssociation step, which means that there must be a calibration issue.

• 2 cameras is the absolute minimum and you will not benefit from some of the features that make the process particularly robust with more cameras As I said earlier, 2 cameras is the absolute minimum. But they are synchronized and set at a good angle, so it should still work.

For intrinsic parameters, don't hesitate to get closer to the camera, and to take more pictures of the checkerboard. This should still work the way you did it. What camera did you use? If it is a phone, did you take pictures instead of extracting frames from a video? This would cause a problem, since the image format is not the same in both cases on a smartphone. Sorry for the pixel error, it is a new feature I released not long ago. You can have it by running pip install pose2sim --upgrade.

If the pattern used for extrinsic calibration is not moving, it does not matter if there are frame differences. I thought you were holding it by hand. It would be better if it were either perfectly horizontal or perfectly vertical, or your triangulated person will be leaning on the side instead of standing upright. Do the clicked green points and the blue reprojected points overlay each other?

To be honest, if after checking all that it still does not work I don't know where the issue comes from, so maybe I'll just ask you to send me your project folder and I'll be able to reproduce and fix the problem.

[conorgren]

Perfect thank you, the cameras i used were an iPhone 7 and an iPad 11. Both were set up completely still. I used the ProMovie app on both to ensure the footage was captured at 30 fps and 1920x1080p The pictures for the both intrinsic and extrinsic were both extracted from the video. The checkerboard in the extrinsic pics is not moving at all. I don't see any green points or blue reprojections when running the calibration code if this is the wrong code please let me know

[davidpagnon]

I suspect it might be a genuine bug when you run extrinsic calibration on a checkerboard. Could you send me your full project folder (by Swisstransfer or wetransfer for example) with a short raw sequence sequence to triangulate?

[conorgren]

is a google drive link ok?

[davidpagnon]

Sure

[conorgren]

Hi sorry for the delay, uploading took longer than expected here is the link, my own files are in Empty_Project

https://drive.google.com/drive/folders/1rEqodTWCJMFbdgCBpKSMCAv4fiBpd1_M?usp=sharing

thanks so much let me know if you need anything else

[smwscth]

I did it! Thank you for your help! I used three cameras this time and had them at an Angle of about 30 degrees, it is better than two cameras indeed. And I found that I don't change the frame rate in config.toml, I set the frame rate 30HZ matching my video.

[davidpagnon]

Awesome @smwscth ! I'm glad it is working now :)

[davidpagnon]

Sorry @conorgren, you are two in this post and I have mistakingly closed this issue.

Thanks for sharing your link! Your problem is that you left calibration_type to convert in Config.toml, and you incidentally copied the calibration.qca.txt from the Demo folder to Empty_project. Pose2Sim did not throw any error at you since it found what it needed, however it converted a calibration file that had nothing to do with the actual properties or the placement of your cameras.

Consequently, you should change that parameter to calculate, and follow my previous instructions for best practices.

[conorgren]

ok thank you! Do i need the calibration.qca.txt file or should i delete it from the folder?

[davidpagnon]

This won't hurt if it is still there, but it would be clearer if you deleted it. You can also run pip install pose2sim --upgrade, The Empty_project structure is now preserved and it is easier to populate it.

[conorgren]

Ok I deleted it and now it makes me manually select the squares on the checker board. Sometimes it does it automatically but usually i need to click on it. Then for the extrinsics, a different 3D space image shows up like this

do i need to have something in my scene in that shape or can i put strips of tape on the floor or something?

[davidpagnon]

Hi, Yes, just follow the tutorial here: https://github.com/perfanalytics/pose2sim#calculate-from-scratch You need to either lay a large checkerboard on the floor (or vertically), or to measure the coordinates of some points and report them in object_coords_3d (in Config.toml).

[conorgren]

ok i will try that tonight thank you!

[conorgren]

ok that's working now! thanks so much

now I'm getting this error from opensim I've never received before

[davidpagnon]

Can you close OpenSim and reopen it? It has weird error messages sometimes.

If it does not work, can you open the trc file and copy-paste it here?

[conorgren]

still didnt work i'll paste the file now

This XML file does not appear to have any style information associated with it. The document tree is shown below.

69 -1 -1 Unassigned Model_Pose2Sim_Body25b.osim Unassigned true measurements manualScale true LShoulder LHip RShoulder RHip X Y Z X Y Z X Y Z X Y Z X Y Z X Y Z true LShoulder LElbow RShoulder RElbow X Y Z X Y Z true LElbow LWrist RElbow RWrist X Y Z X Y Z X Y Z X Y Z X Y Z X Y Z true LHip LKnee RHip RKnee X Y Z X Y Z X Y Z X Y Z true LKnee LAnkle RKnee RAnkle X Y Z X Y Z true LHeel LBigToe RHeel RBigToe X Y Z X Y Z X Y Z X Y Z X Y Z X Y Z true Head Nose Head Neck X Y Z true RHip LHip X Y Z X Y Z 0.95 0.95 0.95 pelvis false 0.85 0.85 0.85 patella_r false 0.85 0.85 0.85 talus_r false 0.85 0.85 0.85 toes_r false 0.85 0.85 0.85 patella_l false 0.85 0.85 0.85 talus_l false 0.85 0.85 0.85 toes_l false 0.8 0.8 0.8 torso false 0.8 0.8 0.8 head false Standing_for_Scaling.trc 0.016666666666666666 0.4166666666666667 true Unassigned Unassigned false true 0.10000000000000001 true 0.10000000000000001 true 1 true 2 true 2 true 1 true 1 true 1 true 1 true 2 true 2 true 1 true 1 true 1 true 1 true 1 true 1 true 1 true 1 true 1 true 1 true 1 default_value 0 true 1 default_value 0 true 1 manual_value 0 true 1 manual_value 0 true 1 default_value 0 Standing_for_Scaling.trc Unassigned 0.016666666666666666 0.4166666666666667 Unassigned Model_Pose2Sim_Body25b_scaled.osim Unassigned -1

[conorgren]

I'm not sure if that worked it should be the demo scaled file?

[davidpagnon]

This document should look like this: https://github.com/perfanalytics/pose2sim/blob/main/Pose2Sim/Demo/opensim/Standing_for_Scaling.trc

You should first capture the subject on a static pose (A-pose or T-pose), do the scaling with the trc result of it; and then you can ask them to do any required movement and use the corresponding trc files.

[conorgren]

i can't use the demo scaled file?

[davidpagnon]

No, because you start with a generic subject with a generic size and proportions. You will want to scale it to your own subject.

To this end, you need to take a static capture of the subject (or to select a few frames in a trial where the subject is not moving), and run the scale tool.

[conorgren]

I am the same size and weight of the generic subject though so can it work?

[davidpagnon]

Not necessarily, it also depends on your proportions (your arms could be very long or short, your torso wide or narrow, etc)

[conorgren]

is it possible to just force it and get slightly inaccurate results, just to get it working?

[davidpagnon]

Yes, normally it should not be accurate but if your proportions are not too different it should lead to a result (inaccurate though). In that case, do not run scaling and go straight to inverse kinematics.

But your trc file (output of triangulation or filtering) should not look like what you previously sent.

[conorgren]

It did not work haha

when selecting the points on the ground does it matter that they are in different planes than the checker board?

for example when selecting the intrinsic checkerboard squares they were in the XY plane with the Z axis pointing into the camera. When selecting the extrinsic points on the ground the Z axis was pointing up. Do both need to have the same plane orientation?

[davidpagnon]

This is probably not the expected result indeed 😁

It should not matter where the board is for intrinsic parameters since it takes several images in different planes. Are you sure you clicked on the extrinsic points in the right order?

You can check the output of the trc in OpenSim : File-> Preview experimental data if I recall.

[conorgren]

yes the blue dots are moving in the same motion i would expect! It's probably somthing small I'm doing wrong. Would you possibly be free for a video call perhaps on discord? no worries if not

[davidpagnon]

Sorry right now I cannot, it is late here :)

Do you use smartphones? If so, did you take photos for extrinsics, or videos? You should take videos, because the format is different when taking a picture.

Did you try to preview your trc in OpenSim?

[davidpagnon]

Oh are you talking about the blue dots in OpenSim? If this is good, than you probably did not choose the right "pose_model" in Config.toml, or not the right model in your OpenSim xml setup file.

[conorgren]

No problem, i use an iphone 7, iphone 5s, iphone 4 (lol) and an ipad 11. All of them are the same resolution. I previewed the trc file in opensim and it works how i would expect with the blue dots. The pose model i chose is Model_Pose2Sim_Body25b.osim. This is the same as the config file i believe?

I'm free at anytime if you don't mind a call. Anytime that suits you

[davidpagnon]

Let's see if we can figure it out tonight, if not we'll have a call :)

If you chose Body_25b, then you need to:

• run OpenPose with Body_25b: bin\OpenPoseDemo.exe --model_pose BODY_25B --video <PATH_TO_PROJECT_DIR>\raw-2d\vid_cam1.mp4 --write_json <PATH_TO_PROJECT_DIR>\pose-2d\pose_cam1_json
• set pose_model to BODY_25B in Config.toml
• use IK_Setup_Pose2Sim_Body25b.xml in OpenSim

If you realize you actually ran body_25 instead (the default OpenPose model), then change everything to body_25, it should not make a big difference.

[conorgren]

i added the --model_pose1 line and i got this ERROR: unknown command line flag 'model_pose1'

[conorgren]

this is the code i ran before

(base) C:\Users\conor\anaconda3\Lib\site-packages\Pose2Sim\Empty_project\openpose-1.7.0-binaries-win64-gpu-python3.7-flir-3d_recommended\openpose>bin\OpenPoseDemo.exe --video C:\Users\conor\anaconda3\Lib\site-packages\Pose2Sim\Empty_project\raw-2d\vid_cam1.mp4 --write_json "C:\Users\conor\anaconda3\Lib\site-packages\Pose2Sim\Empty_project\pose\json_cam1.json" --display 1 --render_pose 1 --net_resolution -1x320

this code seems to work and produce json files. I run it 4 times, 1 for each video and change the vid_cam1 and the json destination

[davidpagnon]

There is no 1, it is --model_pose Mor information there: https://github.com/perfanalytics/pose2sim/tree/main#with-openpose

So you ran OpenPose with the default body_25 model, then just change to body_25 in Config.toml and in OpenSim and tell me how it goes!

[conorgren]

i think openpose cant find the 25B model i get this error Prototxt file not found: models\pose/body_25b/pose_deploy.prototxt.

[davidpagnon]

Yes, you need to follow the link and install it. But for starters, just do body_25!

[conorgren]

ok this worked and i did it for all for of my videos. I will do this now?

[davidpagnon]

You can do it with either body_25 or body_25b (or another model), but you need to make sure to edit Config.toml accordingly and to use the right OpenSim setup file

[conorgren]

I'm trying to get a T pose for calibration and it has given me an error at the triangulation stage again. What i did was make a new set of videos but they are only 1 frame long in a T pose and i was going to use those to create the trc file for scaling? Is this a good idea?