Issue with Joint Calibration

[spock-the-wizard]

          Hi, thanks for using our dataset. For the second question, refer to [here](https://huggingface.co/datasets/OpenIllumination/OpenIllumination/blob/main/others/joint_calib/greenhead/README.md).

Originally posted by @ootts in https://github.com/oppo-us-research/OpenIlluminationCapture/issues/2#issuecomment-1783445293

Hi, I opened an issue a few months ago and am still have some unresolved issues. Right now I am trying to use the light pattern images for a pretraining stage and the olat for the second stage to train a SDF.

Last time I checked, the two sets of camera poses weren't compatible, which was why I am trying the joint calibrated camera pose solution you suggested.

The camera poses seem fine when I visualize them, but the SDF looks nothing like the objects at all. I'm assuming it's an issue with K.

Here's what I've been doing.

1. Use camera_normalized.json 's 'pose' for the transform matrix
2. Use camera_normalized.json's 'newK' for the intrinsic matrix.

Am I doing something wrong? Thanks in advance!

[spock-the-wizard]

...Okay Problem solved! The image resolution used for the intrinsic matrix was different from the images provided!

Scaling the 'newK' to the actual image resolution solved the issue.

Thank you!

[spock-the-wizard]

Hi sorry, problem not solved. I'm using PermutoSDF on the greenhead item. It seemed to be working, but looking more closely am getting very wrinkly results. The first one is when using the cam pose in the main repo (no joint calibration). The second image is the result I am getting when using cameras_normalized.json.

Can you give me any hints on how I should be computing the camera matrix?

For the intrinsic matrix, I am using "newK", after rescaling each row (x,y) from colmap images(1.6Kx2.4K) back to the image resolution (2.5K x 4K).

[ootts]

The first output is reasonable. For the second one, there are multiple possibilities. 1) Did you use the images from here and calibration from here? The images are with distortion. You need to perform undistortion and use newK corresponding to the correct image resolution. 2) If it is still not working, it might be because the manual color correction doesn't support high-quality reconstruction. Or the color-corrected images fail to give an accurate calibration. To verify this, you can select the DSLR images and camera parameters from this joint set, do the reconstruction on this DSLR subset, and compare to output 1. This may tell us if the DSLR calibration is affected by introducing the high-speed images.

[spock-the-wizard]

Hi,

So no, I was using the images from the main repo! How can I update the camera matrix to fit images in the main repo?

(Edited) For a sanity check, I checked if using images from the main repo was compatible. I resized the main repo image to 1600x2400 and compared them to the ones used for joint calib, and got the following results.

I expected the images used for joint calibration were resized versions of the ones in the origin, but they are quite misaligned. I'm guessing this is why the sdf wasn't learning correctly.

[ootts]

After carefully reviewing the data, we found some inconsistencies in the released data. To better address your problem, could you tell us what exactly you need? Do you only need the lighting patterns, or you need both the lighting patterns and OLAT data?

[desmondlzy]

@ootts Hi --

thanks for the releasing this dataset! following this thread of discussion, I'm bit worried I might be running into more or less the same situation. what's exactly the issue/inconsistency in the data? I need the light positions gt in high speed cam coordinate (the one in main repo is in DSLR coords), and ways to convert between DSLR and high speed cam coordinates...

it will be really helpful if you can give some hints... thanks

[ootts]

@desmondlzy Hi, thanks for your question. This issue happened a bit long time and I need some time to review what it is. Feel free to add my wechat (clhfsky) or schedule a Zoom meeting with us so that we can better resolve this issue.