Understanding camera rig import

[pnuehrenberg]

I am trying to use the camera rig functionality, but unfortunately the instructions from https://github.com/alicevision/meshroom/wiki/Multi-Camera-Rig did not solve my issues. I am extracting frames from synchronized videos, so no exif metadata are available for the input images.

It seems like using any prefix for the image name (before the index) yields following warning: [15:57:14.438564][warning] Invalid rig structure for view: /pathtoproject/rig/0/rig_0000.png

Not using a prefix avoids that warning, however the import only works for images with proper metadata (as tested with my phone). Does this depend on the initialization of intrinsics, as images without metadata get the default option - and hence just one intrinsic for all views, no rig?

If I copy metadata to the extracted frames with exiftool, it kind of works but the little sign for rig initialized in the images view does not show up, so I guess the rig constraints will not be used in SfM?

My main objective is to precisely track the camera positions of four fixed cameras in a moving rig. Using the same image datasets without the rig structure works fine, but it would be good to have constraints on the relative camera (view) poses. Later I'd like to use CameraRigLocalization or CameraLocalization to get complete camera trajectories, as I'm currently sampling my videos at a lower rate (0.5 to 3 fps).

I tested this with the prebuilt release and with the newest AliceVision version built from source. Any suggestions are much appreciated.

[norru]

+1 I am trying to do a similar thing and some instructions would be appreciated!

[norru]

Hi @pnuehrenberg, perhaps you could help me here!

I am actually struggling to import a camera rig at all

I've got the following structure:

rig/
  0/
    00001.png
    00002.png
    ...
  1/
    00001.png
    00002.png
    ...
  ...

If I drag/drop the rig folder I get the following error:

[10:15:24.725054][info] CameraInit report:
        - # views listed: 897
           - # views with an initialized intrinsic listed: 897
           - # views without metadata (with a default intrinsic): 0
        - # intrinsics listed: 1
Program called with the following parameters:
 * allowSingleView = 1
 * defaultCameraModel = "" (default)
 * defaultFieldOfView = 45
 * defaultFocalLengthPix = -1 (default)
 * defaultIntrinsic = "" (default)
 * groupCameraFallback =  Unknown Type "20EGroupCameraFallback"
 * imageFolder = "" (default)
 * input = "" (default)
 * output = "/tmp/tmpt9xkmt_6/CameraInit/c448939571d5c70b05c9ae4ad416ada37d6273ca/cameraInit.sfm"
 * sensorDatabase = "/opt/meshroom-dist/Meshroom-2019.2.0/aliceVision/share/aliceVision/cameraSensors.db"
 * verboseLevel = "info"

[10:16:21.019394][error] Program need -i or --imageFolder option
No input images.
ERROR:root:Error while building intrinsics : Traceback (most recent call last):
  File "/opt/Meshroom/meshroom/ui/reconstruction.py", line 399, in buildIntrinsics
  File "/opt/Meshroom/meshroom/nodes/aliceVision/CameraInit.py", line 192, in buildIntrinsics
RuntimeError: CameraInit failed with error code 1. Command was: "aliceVision_cameraInit  --sensorDatabase "/opt/meshroom-dist/Meshroom-2019.2.0/aliceVision/share/aliceVision/cameraSensors.db" --defaultFieldOfView 45.0 --groupCameraFallback folder --verboseLevel info --output "/tmp/tmpt9xkmt_6/CameraInit/c448939571d5c70b05c9ae4ad416ada37d6273ca/cameraInit.sfm" --allowSingleView 1 "

If I drag/drop the folders separately as 0 1 2 etc, only images in folder 0 are imported.

What is the correct workflow? Is it documented anywhere? Couldn't find anything in the manual, or anywhere else.

Also every time I retry I have to start the project from scratch as I could not find a way to remove all imported images in bulk and removing all frames one at a time takes ages. Is there a way to "select all/remove all"?

Many thanks!

[pnuehrenberg]

If I drag/drop the folders separately as 0 1 2 etc, only images in folder 0 are imported.

This happened to me as well, except when the images have appropriate metadata of different cameras. Adding metadata is possible with other tools, such as exiftool, but I haven't looked at this since I opened the issue. Also, the rig initialized icon did not show up in the images view.

According to the wiki, having correctly named images with prefixes and so on should solve this, but then I got the invalid rig structure error.

I also did not test the 2019.2.0 release, maybe this was fixed already? Which release did you use?

The only way this actually worked for me was using different cameras, capturing images instead of video so that I get correct metadata.

[norru]

I am on 2019.2.0 "revised version" (2019.2.0 was gasp released twice)

The images are not directly coming from a camera. They're extracted video frames which have been heavily preprocessed. I don't have the luxury of an alternative here. As such, they do not have - and do not need - a prefix such as DSC_.

EXIF injection sounds painful. I can feed precalculated intrinsics to the node so I don't really see the need for it.

On another note, I have found a way to delete all images by resetting the CameraInit node.

It does look like Metashape UI was designed for users to follow a very specific and narrow workflow and we just have not been lucky enough to be part of the happy majority.

[pnuehrenberg]

As such, they do not have - and do not need - a prefix such as DSC_.

From what I could figure out, this seems needed when images do not have distinct EXIF metadata to properly map the image to rig position correspondences.

I will look into the EXIF injection at some point, but I'm not sure that this will actually solve the problem. A fix on the Meshroom side or better documentation of this feature would be very appreciated.

[oliver-batchelor]

I have the same issue, my images also don't have EXIF metadata. I get the same error message as @norru

[oliver-batchelor]

I tried injecting EXIF data in the images - Meshroom does not seem to find any of it. I added all of these attributes:

FocalLength                     : 16.0 mm
SerialNumber                    : 12345
Model                           : BLACKFLY 12
Make                            : BLACKFLY
FocalLength35efl                : 16.0 mm

I also added a prefix filename to all my images. Nothing works. It still lists all the attributes as missing.

[17:31:59.130781][warning] Image 'img_720.000000.png' focal length (in mm) metadata is missing.
Can't compute focal length (px), use default.

[17:31:59.136879][warning] Some image(s) have no serial number to identify the camera/lens device.

[oliver-batchelor]

Ok, I got this working. It is necessary for a lot of conditions to be met. 1) Use jpeg - I was trying with png and it only reads EXIF from the jpeg 2) Serial number needs to be unique for each image 3) No prefix on the images - just numbering 4) Camera Make and Model needs to be put into the "cameraSensors.db" and added to EXIF of each image 5) FocalLength EXIF needs to be set for each image

[oliver-batchelor]

If serial number is unique for each image then it seems Meshroom allows intrinsics to vary for the same camera... if serial number is not unique I drag 2x40 images in and only two images show up.

[belveder79]

I also tried that and for me the following conditions worked:

1. Rig and Camera folders as described, jpeg images
2. Numbering the related files with 1.jpg, 2.jpg and so on in the subfolders, no prefix
3. camera make and model faked in the database
4. EXIF Time stamp must be the same for all related cameras (that was crucial actually)

[MrChickenBacon]

I have tried what you guys suggest, but can't get meshroom to find the SerialNumber from my pictures. I get the message: Warning: SerialNumber metadata is missing. I used AnalogExif to set my drone SerialNumber on the pictures. The files are in JPEG format. Numbered them 1.JPEG, 2.JPEG, 3.JPEG and so on. Maybe someone can share 1 picture that Meshroom finds the serial number to, so that i can compare my EXIF info?

[natowi]

The important thing here is to have a different EXIF serial number (BodySerialNumber) for each rig-subfolder image set.

Tested with MR2019.2

https://github.com/alicevision/meshroom/wiki/Multi-Camera-Rig

[tlalexander]

Thanks. Will those details be pushed to the other docs upon next release? [1] I just encountered this issue myself. It would be nice if these extra constraints were not needed. I am also using PNG files from video, and will have to convert to JPG and inject exif data. Since the images are already in separate folders, the extra requirement of separate serial numbers seems redundant. Still I'm happy to have a workaround!

[1] https://meshroom-manual.readthedocs.io/en/latest/faq/multi-camera-rig/multi-camera-rig.html

[tlalexander]

Ah well I spoke too soon. I still can't actually get this to work.

I've tried the dataset posted in #906 (which is here) and it doesn't load in to my meshroom. I get "No input images." on the command line when trying to import that dataset by dragging the "rig" folder to the UI. This is on Ubuntu 18.04 using Meshroom 2019.2.0.

My own custom dataset doesn't work either but since the sample dataset with real EXIF data doesn't work I'd say there's a bigger problem.

I noticed that the Meshroom docs say if camera data is not found in the database it will warn the user and attempt construction with reduced quality. But that is not what I am seeing when trying to load a rig - instead it silently fails to load the images with no message in the UI.

Anyway I am adding EXIF data for my custom images using the following code. See my sample images here.

exif_dict = piexif.load(output_file_name)
exif_dict['Exif'][piexif.ExifIFD.BodySerialNumber] =  bytes('00{}'.format(rig_index), 'utf-8')
exif_dict['Exif'][piexif.ExifIFD.FocalLength] = (3,1)
exif_dict['0th'][piexif.ImageIFD.Make] = bytes('GoPro', 'utf-8')
exif_dict['0th'][piexif.ImageIFD.Model] = bytes('FUSION', 'utf-8')
exif_dict['0th'][piexif.ImageIFD.DateTime] = bytes('2020:01:01 01:00:{:02d}'.format(date_number), 'utf-8')
exif_bytes = piexif.dump(exif_dict)
piexif.insert(exif_bytes, output_file_name)

That produces images with the following EXIF data with exiftool:

ExifTool Version Number         : 10.80
File Name                       : 0001.jpg
Directory                       : /home/taylor/sphere/rig/0
File Size                       : 907 kB
File Modification Date/Time     : 2020:05:30 18:51:41-07:00
File Access Date/Time           : 2020:05:30 18:56:12-07:00
File Inode Change Date/Time     : 2020:05:30 18:51:41-07:00
File Permissions                : rw-r--r--
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Exif Byte Order                 : Big-endian (Motorola, MM)
Make                            : GoPro
Camera Model Name               : FUSION
Modify Date                     : 2020:01:01 01:00:01
Focal Length                    : 3.0 mm
Serial Number                   : 000
Comment                         : Lavc58.80.100
Image Width                     : 2000
Image Height                    : 2000
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
Image Size                      : 2000x2000
Megapixels                      : 4.0
Focal Length                    : 3.0 mm

[tlalexander]

Ah. I was able to get it to work. Another undocumented requirement seems to be that you drag each subfolder from the "rig" folder in to meshroom separately (to the "add images" drop spot). I was dragging the whole rig folder in which made sense to me, but fails. Being able to drag a rig folder would be helpful.

It seems to be working even with my custom dataset, so the exif code above may be what some of yall need. I will try a larger dataset and see what kind of results I can get.

[natowi]

I was dragging the whole rig folder in which made sense to me, but fails.

@tlalexander I added this in the wiki "When you drop the rig sub-folders into Meshroom". Maybe I need to emphasize it more.

Being able to drag a rig folder would be helpful.

Meshroom is still under development, so this and other issues will be addressed in the future.

Will those details be pushed to the other docs upon next release

Yes, merging the wiki into readthedocs is something we are currently working on. You are welcome to contribute. This task is something easy to start with, if you are interested.

[jcc10]

I have a (kinda) working ingestion tool for the Insta360 Pro2 (jcc10/Meshtools-Insta360-Pro-2-Ingestion) (can't get the script to run the commands so it prints them for you to copy/paste run them)

The tags currently altered via are: -ExifIFD:SerialNumber="Pro2Subcamera${rig}" -ExifIFD:DateTimeOriginal="${fauxDate}"

With the rig being the origin number and faux date being a number based off the session id. Additionally, I have added the camera to the sensor db (need to make a pull with that...)

[jhaggle]

1. Use jpeg - I was trying with png and it only reads EXIF from the jpeg

Converting png-files to jpg-files creates compression artefacts with a LOT of extra features/keypoints as result. That makes it hard to work with jpgs:

[dongillett]

I've got it so that it is almost working but I don't know what to put for focal length. I think that is all I'm missing to get the rig to work. I'm making a 'virtual camera rig' from an already processed equirectangular image (done in my hardware). I extract it into 4 images facing 4 different directions. The 4 images are extracted with code similar to this that does the un-distortion... https://github.com/NitishMutha/equirectangular-toolbox/blob/master/nfov.py

The 4 images are extracted using a desired center point in the full image and a destination size, but I don't know the focal length nor fov. I have just been letting it solve without it but it occasionally gets a lone camera to flip so the rig isn't locking it down. I've got the warnings saying it is an invalid rig format resolved and have all the EXIF data set by @tlalexander in the working sample mentioned above except focal length and not sure how to get it in the equirectangular decomposition case.

@tlalexander how did you determine your focal length?

if_dict['Exif'][piexif.ExifIFD.FocalLength] = (3,1)

Below are previews of my 4 images followed by the full image it was extracted from.

[tlalexander]

Hey looks great. I would suggest running a few images through as a single camera and seeing what it solves for focal length, then put that in at the start and run as multi camera. Does that make sense?