search

3DOM-FBK / deep-image-matching

Multiview matching with deep-learning and hand-crafted local features for COLMAP and other SfM software. Supports high-resolution formats and images with rotations. Both CLI and GUI are supported.
https://3dom-fbk.github.io/deep-image-matching/
BSD 3-Clause "New" or "Revised" License
357 stars 45 forks source link

Run error with custom images. Could not reconstruct any model! #59

Closed monag-ucberkeley closed 6 months ago

monag-ucberkeley commented 7 months ago

I am getting the following error when I run on a pair of images. One is a regular rgb image another is the same image in lwir format.

(deep-image-matching) deep-image-matching % python3 main.py \ --images ./images \ --pipeline superpoint+superglue PNG file does not have exif data. 2024-04-15 11:58:03 | [INFO ] No exif data available for image img01.jpg (this will probably not affect the matching). PNG file does not have exif data. 2024-04-15 11:58:03 | [INFO ] No exif data available for image img02.jpg (this will probably not affect the matching). Loaded SuperPoint model Loaded SuperGlue model ("outdoor" weights) 2024-04-15 11:58:03 | [INFO ] Running image matching with the following configuration: 2024-04-15 11:58:03 | [INFO ] Image folder: images 2024-04-15 11:58:03 | [INFO ] Output folder: results_superpoint+superglue_matching_lowres_quality_high 2024-04-15 11:58:03 | [INFO ] Number of images: 2 2024-04-15 11:58:03 | [INFO ] Matching strategy: matching_lowres 2024-04-15 11:58:03 | [INFO ] Image quality: HIGH 2024-04-15 11:58:03 | [INFO ] Tile selection: NONE 2024-04-15 11:58:03 | [INFO ] Feature extraction method: superpoint 2024-04-15 11:58:03 | [INFO ] Matching method: superglue 2024-04-15 11:58:03 | [INFO ] Geometric verification: PYDEGENSAC 2024-04-15 11:58:03 | [INFO ] CUDA available: False 2024-04-15 11:58:03 | [INFO ] Low resolution matching, generating pairs .. Loaded SuperPoint model 2024-04-15 11:58:03 | [INFO ] Extracting features from downsampled images... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.13s/it] 2024-04-15 11:58:06 | [INFO ] Matching downsampled images... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 5.19it/s] 2024-04-15 11:58:06 | [INFO ] Found 0 pairs. 2024-04-15 11:58:06 | [INFO ] Extracting features with superpoint... 2024-04-15 11:58:06 | [INFO ] superpoint configuration: {'keypoint_threshold': 0.0005, 'max_keypoints': 4096, 'name': 'superpoint', 'nms_radius': 3} 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.47s/it] 2024-04-15 11:58:09 | [INFO ] Features extracted! 2024-04-15 11:58:09 | [INFO ] Matching features with superglue... 2024-04-15 11:58:09 | [INFO ] superglue configuration: {'match_threshold': 0.3, 'name': 'superglue', 'sinkhorn_iterations': 100, 'weights': 'outdoor'} 2024-04-15 11:58:09 | [INFO ] Matching features... 2024-04-15 11:58:09 | [INFO ] 0it [00:00, ?it/s] 2024-04-15 11:58:09 | [WARNING ] Was not possible to load the first image to initialize cam0 2024-04-15 11:58:09 | [WARNING ] Was not possible to load the first image to initialize cam1 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 44.05it/s] 2024-04-15 11:58:09 | [INFO ] Using pycolmap version 0.6.1 2024-04-15 11:58:09 | [INFO ] Running 3D reconstruction... I20240415 11:58:09.452909 10708766 misc.cc:198]

Loading database

I20240415 11:58:09.455094 10708766 database_cache.cc:54] Loading cameras... I20240415 11:58:09.455137 10708766 database_cache.cc:64] 1 in 0.000s I20240415 11:58:09.455160 10708766 database_cache.cc:72] Loading matches... I20240415 11:58:09.455184 10708766 database_cache.cc:78] 0 in 0.000s I20240415 11:58:09.455200 10708766 database_cache.cc:94] Loading images... I20240415 11:58:09.455260 10708766 database_cache.cc:143] 2 in 0.000s (connected 0) I20240415 11:58:09.455279 10708766 database_cache.cc:154] Building correspondence graph... I20240415 11:58:09.455296 10708766 database_cache.cc:190] in 0.000s (ignored 0) I20240415 11:58:09.455333 10708766 timer.cc:91] Elapsed time: 0.000 [minutes] W20240415 11:58:09.455351 10708766 incremental_mapper.cc:349] No images with matches found in the database 2024-04-15 11:58:09 | [ERROR ] Could not reconstruct any model! 2024-04-15 11:58:09 | [ERROR ] Pycolmap reconstruction failed 2024-04-15 11:58:09 | [INFO ] [Timer] | [Deep Image Matching] generate_pairs=4.730, extract_features=2.933, matching=0.000, export_to_colmap=0.052, pycolmap reconstruction=0.007, Total execution=7.723

lcmrl commented 7 months ago

Maybe there is a problem with the custom pairs option, I see from the log Found 0 pairs. Could you share the text inside the txt file you passed to the CLI containing the image pairs? Which is the LWIR format (also 8 or 16bit)?

monag-ucberkeley commented 7 months ago

Thanks for the response. I did not pass any txt file to main.py.. Are you refering to config ? This is the command line call that I used - python3 main.py \
--images ./images \ --pipeline superpoint+superglue \ --force

I have tried other pipeline options with the same result. The images that I am trying to match are similar in size. The scenario is to identify tie points for 2 same images in different spectrums? For example, I have an RGB image with 4 objects in it. I have the same image in infra red, showing same 4 objects. I want to find the tie points automatically.

lcmrl commented 7 months ago

I understand, some suggestions:

  1. If you want, you can share here a screenshot of the two images, could happened that are too difficult. Let'see.
  2. Even for superpoint can be too difficult, so try with disk+lightglue, loftr or roma
  3. Please try to inspect the database with COLMAP GUI, just run COLMAP and then File>New database>select path to database produced in result folder and path to images, then inspect the database (Process>inspect database) and see if there are matches there
  4. Try to convert lwir to png or jpeg format
lcmrl commented 6 months ago

Hi, hope you solved the problem. I close for inactivity, please feel free to reopen if you find any issue with the repo

lcmrl commented 4 months ago

The thermal image is almost completely homogeneous, probably this is causing the problem. You can find here an example of matching between RGB and thermal https://x.com/3DOMFBK/status/1770489611724206223 Here is working, but because there is enough texture also in the thermal image

monag-ucberkeley commented 4 months ago

Can you please elaborate on what techniques can be used to add texture to thermal image.

lcmrl commented 4 months ago

Matching with local features requires that locally we can describe an area of the image with a descriptor and we will look for the same descriptor in the second image. But while in the second image we can locally distinguish the grass, in the thermal one is even difficult with the eye to find correspondences (a part the 4 big objects). The grass is not visible at all because of the different low resolution. To have more texture on the thermal image you could choose a moment of the day where you can really see some thermal differences on the image, choosing the right distance to the object. But it depends also by the object, probably is too difficult to catch thermal differences on the grass.

If your thermal camera and RGB camera are rigidly coupled you can think of getting the orientation of the RGB image block using only RGB images, then you get your 3D point cloud and mesh, and finally you can simply reproject the thermal images on the mesh