Question about Multi-Camera Data Usage and Calibration in EasyVolcap

[YongmoonPark]

I've confirmed that using custom static multi-view data works well. However, I'm encountering difficulties with multi-camera data. I attempted to convert the calibration values of my 36-channel camera setup to the EasyVolcap calibration format, which resulted in unsatisfactory outcomes. I have several questions regarding this:

1. For multi-camera setups that cannot use Colmap, should I proceed with calibration using the EasyMocap method (EasyMocap calibration)? Is there a specific calibration technique recommended for EasyVolcap?

2. Looking at the sample data, the extrinsic calibration value for Rot_00 is not an identity matrix but something different.
   Our calibration is based on cam_00, which leads me to think that the relative positioning of the cameras might not significantly alter the outcome, depending on which reference point is used. However, given the unsatisfactory results, I am keeping all possibilities open. What reference point was used to determine the camera positions in the sample? Would using the EasyMocap method yield a Rot_00 value similar to the sample?

I'm looking for guidance on whether there's a specific calibration method required for EasyVolcap and how to address the calibration issues with multi-camera data effectively. Thank you for your assistance.

---

my calibration data

R_000000: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [0.0000000000, 0.0000000000, 0.0000000000]
Rot_000000: !!opencv-matrix
  rows: 3
  cols: 3
  dt: d
  data: [1.0000000000, 0.0000000000, 0.0000000000, 0.0000000000, 1.0000000000, 0.0000000000, 0.0000000000, 0.0000000000, 1.0000000000]
T_000000: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [0.0000000000, 0.0000000000, 0.0000000000]
R_000001: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [-0.0387293496, 0.2812993123, 0.1085048680]
Rot_000001: !!opencv-matrix
  rows: 3
  cols: 3
  dt: d
  data: [0.9548976038, 0.1014360964, -0.2790721852, -0.1122470035, 0.9934143119, -0.0229916312, 0.2749021215, 0.0532796701, 0.9599948439]
T_000001: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [-0.9293841881, -0.1210860311, 0.0560820123]

---

0013_01's calibration data

R_00: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [1.6823949814, 1.4320055246, -0.6942056417]
Rot_00: !!opencv-matrix
  rows: 3
  cols: 3
  dt: d
  data: [0.2076084018, 0.9741442800, 0.0891154334, 0.5334432125, -0.0363806561, -0.8450531363, -0.8199616075, 0.2229781598, -0.5272035599]
T_00: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [0.1817326248, 0.0986888334, 3.2907896042]
t_00: 0.0000000000
n_00: 0.5000000000
f_00: 20.0000000000
bounds_00: !!opencv-matrix
  rows: 2
  cols: 3
  dt: d
  data: [-0.1252000183, -0.3997000754, -0.9367001057, 0.3047999740, 0.1002999693, 0.7732999325]
R_01: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [1.5115214586, 1.2331650257, -0.7963469625]
Rot_01: !!opencv-matrix
  rows: 3
  cols: 3
  dt: d
  data: [0.2666431665, 0.9592524171, 0.0934668258, 0.3094531894, 0.0066335150, -0.9508914948, -0.9127650261, 0.2824722826, -0.2950749993]
T_01: !!opencv-matrix
  rows: 3
  cols: 1
  dt: d
  data: [0.1018460914, -0.0497235581, 3.1227753162]
t_01: 0.0000000000
n_01: 0.5000000000
f_01: 20.0000000000
bounds_01: !!opencv-matrix
  rows: 2
  cols: 3
  dt: d
  data: [-0.1252000183, -0.3997000754, -0.9367001057, 0.3047999740, 0.1002999693, 0.7732999325]

[dendenxu]

Hi @YongmoonPark , does the visualized camera match the actual setup? Looking at the calibration, it somewhat looks reasonable (the cameras are all looking at the same object and they are all aligned). Thus I assume the conversion looks alright. If the cameras don't match the actual setup, it could be possible that the calibration itself wasn't accurate? By the way, I've also recently added the conversion script from the transforms.json of NeRF and NeRFStudio, along with the existing COLMAP conversion. It might help with debugging.

The calibration method in EasyMocap is indeed a good place to start since their camera parameters' format ('intri.yml' and 'extri.yml') can be directly used here in EasyVolcap.

[xbillowy]

Hi @YongmoonPark , thank you for your interest! Here are some suggestions regarding multi-view data calibration that are relevant to your question:

1. I'm not quite sure what you mean by 'for multi-camera setups that cannot use colmap.' If you're referring to situations where there are a small number of cameras and significant variations in viewpoints, I recommend using the wild + multiple sparse calibration method provided by EasyMocap. This calibration method involves capturing an additional dense scene video using a smartphone for joint calibration. You can find detailed instructions in the Sparse Calibration section of custom_dataset.md.

2. The reference points used to determine the camera positions in the example data, or in other words, the center point of the world coordinate system obtained through calibration, are determined by the calibration chessboard we placed in the scene during the calibration process. In fact, the example data was calibrated using the wild + multiple sparse method, where we placed a calibration chessboard in the scene to determine the origin and scale of the world coordinate system.

Therefore, I believe that the calibration chessboard is crucial for obtaining a set of real-world physical scales and determining the origin of the calibration results. The calibration process provided by EasyMocap takes all of these factors into account. As a result, EasyMocap should yield similar results to the sample data for your own data. The extrinsic parameters of all cameras are expressed in the world coordinate system determined by the placement of the calibration chessboard, rather than being referenced to a specific camera.

Finally, because EasyVolcap and EasyMocap share the same camera storage specification, the calibration results obtained from EasyMocap (extri.yml and intri.yml) can be directly used in EasyVolcap without any conversion.

I hope this can help you.