Issue with only one camera in the Camera.bin file exported by `export_for_gaussian_splatting.py`

Hello everyone Recently I have been studying 4DGS, SpacetimeGaussian,(https://github.com/oppo-us-research/SpacetimeGaussians), they used colmap for alignment, but the effect was not very good, so I thought of using metashape for alignment, and then aligning with export_for_gaussian_splatting.py and exporting the dataset in colmap format. I wrote a pipeline script myself to automatically add images, automatically align them, and then automatically call the method in export_for_gaussian_splatting.py to export the dataset in colmap format. I manually checked each metashape project, and all 38 photos were aligned. In the train.py of the spacetimeGaussian project, an accident occurred during training, only two cameras were imported. So I observed the content exported by their colmap script. I found that all cameras were in their camera.bin. But in the camera.bin exported by export_for_gaussian_splatting.py, there is only one. So can the camera.bin file store all camera data? Thank you for your reply.

This screenshot is the result of a bin file read generated by their colmap aligned script. This screenshot is the result of the bin file exported using 'export_for_gaussian_splatting.py' after the Metashape alignment.

Some additional information, this script that reads the contents of bin files was written by me for debugging purposes. It may not parse all properties of each camera, but the number of cameras has been fully parsed.

Hello!

Can you please provide the output of export_for_gaussian_splatting.py script (please ensure that you use the latest version)? You can get it from the Console view in Metashape:

Also, how many images were exported with script? (see 'images' dir in the output directory)

I just pulled the latest script from the repository and ran it. The result is as follows. The script exported 38 images. There is still only one camera in the Camera.bin file.

2024-07-29 17:52:20 To execute this script press Scripts/Export Colmap project (for Gaussian Splatting)
2024-07-29 17:52:39 All chunks: True
2024-07-29 17:52:39 All frames: False
2024-07-29 17:52:39 Zero cx and cy: True
2024-07-29 17:52:39 Use local coordinate frame: True
2024-07-29 17:52:39 Image quality: 100
2024-07-29 17:52:39 Export images: True
2024-07-29 17:52:39 Export masks: False
2024-07-29 17:52:39 Confirm deletion: True
2024-07-29 17:52:39 Using pinhole model instead of simple_pinhole: True
2024-07-29 17:52:39 Using only uncropped projections: True
2024-07-29 17:53:03 F:/4DGS/Dataset/170915_toddler5/vgaVideos/train1/point/colmap_56/metshape/
2024-07-29 17:53:03 
2024-07-29 17:53:03 F:/4DGS/Dataset/170915_toddler5/vgaVideos/train1/point/colmap_56/metshape/
2024-07-29 17:53:03 Calibrations:
2024-07-29 17:53:03 0 761.1063313541606 676 492 0.0 0.0
2024-07-29 17:53:03 Found 0 cropped projections
2024-07-29 17:53:03 Exporting images.
2024-07-29 17:53:07 Undistorted 38 cameras
2024-07-29 17:53:07 Saved 1 calibrations
2024-07-29 17:53:07 Saved 38 cameras
2024-07-29 17:53:07 Saved 5945 points from 9756 tracks
2024-07-29 17:53:07 
2024-07-29 17:53:07 Done
2024-07-29 17:53:07 -----------------------------------

Thank you.

Nice, note that it is typical that there are single camera calibration (see also in Tools->Camera Calibration) - because in most cases dataset is taken using the single camera - with photos taken from different positions. In such case it is benefitial (optimiziation will be more accurate) to optimize its intrinsics calibration (focal length/etc.) sharing them through all photos. So it seems that everything works as expected.

If you need to optimize intrinsics of each photo on its own (treating it as a separate unique camera, because it seems from your screenshot - that you use array of synced different cameras) - you need to split them into separate camera calibration groups. In GUI it can be achieved with Tools->Camera Calibration:

If you need to do this via Python API - please see here.

I tested it, and from the content printed on the console, the result is what I wanted. It seems that I expressed this issue incorrectly. Indeed, 4DGS aligns images taken from different camera positions for each frame. There are indeed 38 cameras, so there should be 38 calibrations. I will implement this feature in the Python script later. I will then inform you of the final result. Thank you very much!

2024-07-29 20:16:17 Calibrations:
2024-07-29 20:16:17 0 743.9290500190335 676 492 0.0 0.0
2024-07-29 20:16:17 1 763.9239829611759 676 492 0.0 0.0
2024-07-29 20:16:17 2 765.2830936657612 676 492 0.0 0.0
2024-07-29 20:16:17 3 761.2090977174735 676 492 0.0 0.0
2024-07-29 20:16:17 4 755.7645037761096 676 492 0.0 0.0
2024-07-29 20:16:17 5 765.5153383799644 676 492 0.0 0.0
2024-07-29 20:16:17 6 763.4741813656019 676 492 0.0 0.0
2024-07-29 20:16:17 7 756.4799627960718 676 492 0.0 0.0
2024-07-29 20:16:17 8 761.1391513865219 676 492 0.0 0.0
2024-07-29 20:16:17 9 768.3725680976859 676 492 0.0 0.0
2024-07-29 20:16:17 10 755.789368204458 676 492 0.0 0.0
2024-07-29 20:16:17 11 762.35616879801 676 492 0.0 0.0
2024-07-29 20:16:17 12 756.3159287241476 676 492 0.0 0.0
2024-07-29 20:16:17 13 811.2988059469138 676 492 0.0 0.0
2024-07-29 20:16:17 14 756.6980459737267 676 492 0.0 0.0
2024-07-29 20:16:17 15 750.8976705883549 676 492 0.0 0.0
2024-07-29 20:16:17 16 759.691143742837 676 492 0.0 0.0
2024-07-29 20:16:17 17 752.5887369585022 676 492 0.0 0.0
2024-07-29 20:16:17 18 751.8232529262775 676 492 0.0 0.0
2024-07-29 20:16:17 19 768.4889093393357 676 492 0.0 0.0
2024-07-29 20:16:17 20 764.7675256619425 676 492 0.0 0.0
2024-07-29 20:16:17 21 761.7605895445415 676 492 0.0 0.0
2024-07-29 20:16:17 22 757.4956111380766 676 492 0.0 0.0
2024-07-29 20:16:17 23 754.8436044539989 676 492 0.0 0.0
2024-07-29 20:16:17 24 755.5134377890524 676 492 0.0 0.0
2024-07-29 20:16:17 25 757.269327853756 676 492 0.0 0.0
2024-07-29 20:16:17 26 774.420875479986 676 492 0.0 0.0
2024-07-29 20:16:17 27 759.5250117014781 676 492 0.0 0.0
2024-07-29 20:16:17 28 762.4558897001095 676 492 0.0 0.0
2024-07-29 20:16:17 29 764.4395250351234 676 492 0.0 0.0
2024-07-29 20:16:17 30 762.878748036579 676 492 0.0 0.0
2024-07-29 20:16:17 31 762.4631568175589 676 492 0.0 0.0
2024-07-29 20:16:17 32 760.460866332741 676 492 0.0 0.0
2024-07-29 20:16:17 33 758.5212075680884 676 492 0.0 0.0
2024-07-29 20:16:17 34 760.2238072392947 676 492 0.0 0.0
2024-07-29 20:16:17 35 755.3817195233784 676 492 0.0 0.0
2024-07-29 20:16:17 36 757.4591065169791 676 492 0.0 0.0
2024-07-29 20:16:17 37 755.1782531434086 676 492 0.0 0.0
2024-07-29 20:16:17 Found 0 cropped projections
2024-07-29 20:16:17 Exporting images.
2024-07-29 20:16:22 Undistorted 38 cameras
2024-07-29 20:16:22 Saved 38 calibrations
2024-07-29 20:16:22 Saved 38 cameras
2024-07-29 20:16:22 Saved 5899 points from 9756 tracks
2024-07-29 20:16:22 
2024-07-29 20:16:22 Done

@PolarNick239 Regarding how to split camera group in Python, the link you provided https://www.agisoft.com/forum/index.php?topic=4769.0 seems to refer to an older version of the API.

chunk = PhotoScan.app.document.chunk

for camera in chunk.cameras:
    sensor = chunk.addSensor()
    sensor.label = camera.label
    sensor.type = camera.sensor.type
    sensor.calibration = camera.sensor.calibration
    sensor.width = camera.sensor.width
    sensor.height = camera.sensor.height
    sensor.focal_length = camera.sensor.focal_length
    sensor.pixel_height = camera.sensor.pixel_height
    sensor.pixel_width = camera.sensor.pixel_width
    camera.sensor = sensor

I found this script in a GitHub repository: https://github.com/agisoft-llc/metashape-scripts/blob/master/src/split_calibration_by_order.py, which includes a method.

def split_cameras_calibration_group_by_order():
    print("Script started...")

    chunk = Metashape.app.document.chunk 
    for camera in chunk.cameras:
        if not camera.type == Metashape.Camera.Type.Regular: #skip camera track, if any
            continue

        sensor = camera.sensor
        new_sensor = chunk.addSensor()

        new_sensor.calibration = sensor.calibration
        new_sensor.fixed = sensor.fixed
        new_sensor.focal_length = sensor.focal_length
        new_sensor.height = sensor.height
        new_sensor.width = sensor.width
        new_sensor.label = camera.photo.meta['Exif/Model'] + " (" + camera.photo.meta['Exif/FocalLength'] + " mm)"#sensor.label + ", " + camera.label
        new_sensor.pixel_height = sensor.pixel_height
        new_sensor.pixel_width = sensor.pixel_width
        new_sensor.pixel_size = sensor.pixel_size
        new_sensor.type = sensor.type
        new_sensor.user_calib = sensor.user_calib

        camera.sensor = new_sensor

    print("Intermediate stage completed...")    
    for i in range(1, len(chunk.cameras)):
        camera = chunk.cameras[i]
        prev = chunk.cameras[i-1]

        if (camera.sensor.width == prev.sensor.width) and (camera.sensor.height == prev.sensor.height):
            if (camera.sensor.pixel_height == prev.sensor.pixel_height) and (camera.sensor.pixel_width == prev.sensor.pixel_width):
                if camera.sensor.focal_length == prev.sensor.focal_length:
                    camera.sensor = prev.sensor

    print("Script finished, grouped calibration groups by the order.")

Can I use this script to split cameras calibration group? The process as I understand it is: add photos -> split cameras calibration group -> align photos -> export COLMAP dataset（export_for_gaussian_splatting.py) Is that correct?

Regarding how to split camera group in Python, the link you provided https://www.agisoft.com/forum/index.php?topic=4769.0 seems to refer to an older version of the API.

It works fine - just replace Photoscan with Metashape:

chunk =  Metashape.app.document.chunk

for camera in chunk.cameras:
    sensor = chunk.addSensor()
    sensor.label = camera.label
    sensor.type = camera.sensor.type
    sensor.calibration = camera.sensor.calibration
    sensor.width = camera.sensor.width
    sensor.height = camera.sensor.height
    sensor.focal_length = camera.sensor.focal_length
    sensor.pixel_height = camera.sensor.pixel_height
    sensor.pixel_width = camera.sensor.pixel_width
    camera.sensor = sensor

I found this script in a GitHub repository: https://github.com/agisoft-llc/metashape-scripts/blob/master/src/split_calibration_by_order.py, which includes a method. Can I use this script to split cameras calibration group?

No, it groups the cameras into calibration groups according to the camera parameters (pixel size, focal length) and picture order, but you want to split photos (with same camera parameters) into separate calibration groups. The above script should work for you.

The process as I understand it is: add photos -> split cameras calibration group -> align photos -> export COLMAP dataset（export_for_gaussian_splatting.py) Is that correct?

Yes, test this pipeline with a small dataset to experiment faster. And when you are sure that the results are OK, process all the data.

Hello @PolarNick239

I have encountered some issues with my script and would appreciate your guidance.

Firstly, I have replaced the export_for_gaussian_splatting.py script in my code with the latest version from the repository and created a new method in pipeline.py. The method follows your guidance, with the only difference being that I added code to delete the first sensor. When manually operating the software, 39 camera calibration groups are generated, likely because an addSensor() operation was performed on the original sensor. Therefore, I speculated that deleting the first sensor would be reasonable.

def split_cameras_calibration_group(chunk: Metashape.Chunk):
    for camera in chunk.cameras:
        sensor = chunk.addSensor()
        sensor.label = camera.label
        sensor.type = camera.sensor.type
        sensor.calibration = camera.sensor.calibration
        sensor.width = camera.sensor.width
        sensor.height = camera.sensor.height
        sensor.focal_length = camera.sensor.focal_length
        sensor.pixel_height = camera.sensor.pixel_height
        sensor.pixel_width = camera.sensor.pixel_width
        camera.sensor = sensor
    # delete first sensor in chunk.sensors
    first_sensors = chunk.sensors[0]
    chunk.remove(first_sensors)

The current situation is that the output result count is correct, and other results appear normal. However, the indices of the camera calibration groups differ from those in the manual software operation. Additionally, I noticed that the calibration parameters for each photo are different, and the script method crops the images, resulting in different resolutions for each photo.

Cropped initial calibration due to high distortion 677x494 -> 660x494
Cropped initial calibration due to high distortion 677x494 -> 651x494
Cropped initial calibration due to high distortion 677x494 -> 643x494
Cropped initial calibration due to high distortion 677x494 -> 665x494
Calibrations:
1 786.6144697329794 660 474 0.0 0.0
2 773.8115759475369 664 486 0.0 0.0
3 760.6436066493642 680 482 0.0 0.0
4 763.4464011534268 674 484 0.0 0.0
5 756.2719570498167 670 484 0.0 0.0
6 765.2332485278239 674 482 0.0 0.0
7 769.6795161029461 670 484 0.0 0.0
8 759.3120301747183 658 480 0.0 0.0
9 761.8608988480474 668 482 0.0 0.0
10 761.8113734653209 674 478 0.0 0.0
11 758.6644111669588 660 486 0.0 0.0
12 763.9844567622933 676 494 0.0 0.0
13 756.5461786172631 672 488 0.0 0.0
14 767.3024836337091 678 492 0.0 0.0
15 757.9429940588706 676 488 0.0 0.0
16 762.9718952301699 672 476 0.0 0.0
17 771.9147732514647 664 476 0.0 0.0
18 764.0816182622544 650 468 0.0 0.0
19 815.2062774067031 664 478 0.0 0.0
20 800.0199637698058 654 490 0.0 0.0
21 834.8572496318587 660 474 0.0 0.0
22 752.5049874901977 678 490 0.0 0.0
23 753.284779000743 674 488 0.0 0.0
24 752.3965054249401 642 490 0.0 0.0
25 740.4054787881596 656 476 0.0 0.0
26 764.637050030434 666 492 0.0 0.0
27 748.7368761239122 660 488 0.0 0.0
28 749.6282343168032 668 490 0.0 0.0
29 750.8666178024731 676 490 0.0 0.0
30 751.1943951954133 660 478 0.0 0.0
31 753.1094960396898 674 488 0.0 0.0
32 755.8681173230968 678 490 0.0 0.0
33 759.0361514637809 672 490 0.0 0.0
34 753.0837922370647 676 488 0.0 0.0
35 754.4754092477001 666 484 0.0 0.0
36 754.4687284365737 668 484 0.0 0.0
37 752.996090369959 666 478 0.0 0.0
38 750.4978900629451 660 488 0.0 0.0
Found 1651 cropped projections
Exporting images.
Undistorted 38 cameras
Saved 38 calibrations
Saved 38 cameras
Saved 5811 points from 9655 tracks

Done

👆 My Code Reslut

2024-07-30 14:57:42 Calibrations:
2024-07-30 14:57:42 0 817.0239755305487 676 492 0.0 0.0
2024-07-30 14:57:42 1 803.6427206151465 676 492 0.0 0.0
2024-07-30 14:57:42 2 805.2538489716724 676 492 0.0 0.0
2024-07-30 14:57:42 3 798.0464680735437 676 492 0.0 0.0
2024-07-30 14:57:42 4 789.390491385337 676 492 0.0 0.0
2024-07-30 14:57:42 5 787.1117869039623 676 492 0.0 0.0
2024-07-30 14:57:42 6 782.2360446363808 676 492 0.0 0.0
2024-07-30 14:57:42 7 775.461806789211 676 492 0.0 0.0
2024-07-30 14:57:42 8 779.2709010688941 676 492 0.0 0.0
2024-07-30 14:57:42 9 782.9020212323782 676 492 0.0 0.0
2024-07-30 14:57:42 10 770.2430865164716 676 492 0.0 0.0
2024-07-30 14:57:42 11 768.9644554601257 676 492 0.0 0.0
2024-07-30 14:57:42 12 758.914310491596 676 492 0.0 0.0
2024-07-30 14:57:42 13 761.8599694271904 676 492 0.0 0.0
2024-07-30 14:57:42 14 803.2068755992809 676 492 0.0 0.0
2024-07-30 14:57:42 15 830.8908722476946 676 492 0.0 0.0
2024-07-30 14:57:42 16 833.7275497676806 676 492 0.0 0.0
2024-07-30 14:57:42 17 844.8281393060412 676 492 0.0 0.0
2024-07-30 14:57:42 18 782.4481993202036 676 492 0.0 0.0
2024-07-30 14:57:42 19 797.9559422964659 676 492 0.0 0.0
2024-07-30 14:57:42 20 778.721488636774 676 492 0.0 0.0
2024-07-30 14:57:42 21 800.1176943739639 676 492 0.0 0.0
2024-07-30 14:57:42 22 829.4102677868449 676 492 0.0 0.0
2024-07-30 14:57:42 23 795.0395014883592 676 492 0.0 0.0
2024-07-30 14:57:42 24 739.1259051435111 676 492 0.0 0.0
2024-07-30 14:57:42 25 733.8277272465197 676 492 0.0 0.0
2024-07-30 14:57:42 26 755.1835316870107 676 492 0.0 0.0
2024-07-30 14:57:42 27 735.0031966550686 676 492 0.0 0.0
2024-07-30 14:57:42 28 735.2059327922864 676 492 0.0 0.0
2024-07-30 14:57:42 29 736.8229345688196 676 492 0.0 0.0
2024-07-30 14:57:42 30 737.3918024786692 676 492 0.0 0.0
2024-07-30 14:57:42 31 738.1594486885062 676 492 0.0 0.0
2024-07-30 14:57:42 32 737.3866469272924 676 492 0.0 0.0
2024-07-30 14:57:42 33 736.2005386294522 676 492 0.0 0.0
2024-07-30 14:57:42 34 736.1940055031388 676 492 0.0 0.0
2024-07-30 14:57:42 35 731.3060584892614 676 492 0.0 0.0
2024-07-30 14:57:42 36 732.7466267144542 676 492 0.0 0.0
2024-07-30 14:57:42 37 728.4632338838537 676 492 0.0 0.0
2024-07-30 14:57:42 Found 0 cropped projections
2024-07-30 14:57:42 Exporting images.
2024-07-30 14:57:46 Undistorted 38 cameras
2024-07-30 14:57:46 Saved 38 calibrations
2024-07-30 14:57:46 Saved 38 cameras
2024-07-30 14:57:46 Saved 5586 points from 9750 tracks
2024-07-30 14:57:46 
2024-07-30 14:57:46 Done

Manually operating the software result 👆 My questions are as follows:

1.Is my assumption to remove the first sensor correct? 2.How can I ensure that the camera calibration group indices start from 0 in the script method? 3.Is it normal for each photo to have different calibration parameters? 4.The resolution of the calibrated photos from the script method differs from those obtained by manually operating the software. How can I ensure they are the same?

Thank you very much for your guidance and I look forward to your response.

1.Is my assumption to remove the first sensor correct?

If you want to remove it because it was the only sensor that existed just before running script "to split camera groups", and after running script - that sensor is unused, so you want to explictly delete it - then yes, your assumption is correct.

2.How can I ensure that the camera calibration group indices start from 0 in the script method?

What do you mean? After running script "to split camera groups" and removing chunk.sensors[0] - list chunk.sensors includes only newly created sensors (one per photo), so they are indexed from zero. Do you mean some other index? (not the index in this chunk.sensors list?)

Is it normal for each photo to have different calibration parameters?

It depends on your use-case:

note that it is typical that there are single camera calibration (see also in Tools->Camera Calibration) - because in most cases dataset is taken using the single camera - with photos taken from different positions. In such case it is benefitial (optimiziation will be more accurate) to optimize its intrinsics calibration (focal length/etc.) sharing them through all photos. So it seems that everything works as expected.

If you need to optimize intrinsics of each photo on its own (treating it as a separate unique camera, because it seems from your screenshot - that you use array of synced different cameras) - you need to split them into separate camera calibration groups. In GUI it can be achieved with Tools->Camera Calibration:

Probably you need to read more about "photogrammetry intrinsics and extrinsics camera calibration parameters". In general - photos taken with the same physical camera (not just the same model, but the same exemplar) with absolutely the same calibration parameters (no auto-focus, fixed lens, etc.) should share the same camera calibration group, but photos taken with different physical camera (or taken with the same camera but each photo was taken differently - lens wasn't fixed, auto-focus was used, etc.) - should be in different calibration groups.

4.The resolution of the calibrated photos from the script method differs from those obtained by manually operating the software. How can I ensure they are the same?

This is because you splitted single camera calibration group into multiple calibration groups (one calibration group per one photo). Different camera calibration leads to different image undistortion results, which leads to different undistorted resolution.

Thank you for your reply.

I understand this issue, thank you
The index I'm referring to is in the log posted above, after the Calibrations field, the index of each sensor output by the script starts from 1, while after manual operation in the software (Tools-Camera Calibration-Split Groups), the index of each sensor starts from 0.

Probably you need to read more about "photogrammetry intrinsics and extrinsics camera calibration parameters". In general - photos taken with the same physical camera (not just the same model, but the same exemplar) with absolutely the same calibration parameters (no auto-focus, fixed lens, etc.) should share the same camera calibration group, but photos taken with different physical camera (or taken with the same camera but each photo was taken differently - lens wasn't fixed, auto-focus was used, etc.) - should be in different calibration groups.

I understand this.

3 and 4 are actually the same issue. After manually operating the software(Tools-Camera Calibration-Split Groups), the calibration parameters and cropping resolution of each photo are the same。However, when using the script to split the group, the calibration parameters and cropping resolution of each photo are different. Therefore, I would like to confirm why this result occurs.

The index I'm referring to is in the log posted above, after the Calibrations field, the index of each sensor output by the script starts from 1, while after manual operation in the software (Tools-Camera Calibration-Split Groups), the index of each sensor starts from 0

They are printed from this line https://github.com/agisoft-llc/metashape-scripts/blob/b26f56826051f718358fb0626fac31e286af0451/src/export_for_gaussian_splatting.py#L395-L397

This is due to the fact that the "split all camera calibration groups" script creates new "sensors" (=new calibration group), so they are created with new key (starting with 1, because key=0 was already taken by the initial single calibration group).

In case of GUI - after you split calibration group - you have N calibration groups and N-1 of them have keys starting from 1 up to N-1 (they are newly created), but the first one of calibration groups - is the one that existed initially (just lots of cameras were moved from that calibration group to new ones).

Probably you can try to add this behaivour into the script that "splits all camera calibratino groups", but it will be more complicated. Why do you need these keys to start from 1? Why you can't, for example, use key-1 instead of key everywhere where you use them in your following pipline?

agisoft-llc / metashape-scripts

Issue with only one camera in the Camera.bin file exported by `export_for_gaussian_splatting.py` #93