search

alicevision / AliceVision

Photogrammetric Computer Vision Framework
http://alicevision.org
Other
2.96k stars 822 forks source link

Depth Map Estimation fails after applying SFMTransform #1514

Closed belveder79 closed 1 month ago

belveder79 commented 1 year ago

I am working on some alternative algorithm for the alignment of a reconstruction to a global coordinate system. Basically it plugs into the existing SfmTransform call right after the StructureFromMotion call and calculates R/t/s differently. Everything remains in a local coordinate system, but R, t and s simply change the position and orientation of the reconstruction.

Anyway, during the course I was playing around also with the manual transform mode and found that the DepthMap estimation is almost 100% guaranteed to fail with "Matrix is Singular" when using an ABC file after applying an SfMTransform, whether it is my own or just a random manual transform.

Can anyone replicate this behaviour? Just changing R t and s breaks it for me...

PS: actually it also fails with the standard 2023.2.0 meshroom pipeline without doing a transform... 4 GTX 980 on latest driver for Ubuntu 22.04 and Cuda 12.2...

[2023-08-23 11:34:10.775971] [0x00007f10497d4000] [trace]   Embedded OCIO configuration file: '/media/ssd2/arth/dev/Meshroom-2023.2.0-av3.1.0-centos7-cuda11.3.1/aliceVision/share/aliceVision/config.ocio' found.
Program called with the following parameters:
 * autoAdjustSmallImage = 1
 * chooseTCamsPerTile = 1
 * colorOptimizationEnabled = 1
 * colorOptimizationNbIterations = 100
 * customPatchPatternGroupSubpartsPerLevel = 48 (default)
 * customPatchPatternSubparts =  Unknown Type "St6vectorIN11aliceVision8depthMap24CustomPatchPatternParams13SubpartParamsESaIS3_EE" (default)
 * downscale = 2
 * exportIntermediateCrossVolumes = 0
 * exportIntermediateDepthSimMaps = 0
 * exportIntermediateNormalMaps = 0
 * exportIntermediateTopographicCutVolumes = 0
 * exportIntermediateVolume9pCsv = 0
 * exportIntermediateVolumes = 0
 * exportTilePattern = 0
 * imagesFolder = "/media/ssd2/arth/dev/meshroom_data/MeshroomCache/PrepareDenseScene/88bc5c32e5260c142ab8c132d2aa274d6b8ed797"
 * input = "/media/ssd2/arth/dev/meshroom_data/MeshroomCache/StructureFromMotion/90ff1a327e320966b3cbb0529f0a9ae7fb58e1d0/sfm.abc"
 * maxCoresAvailable =  Unknown Type "j" (default)
 * maxMemoryAvailable = 18446744073709551615 (default)
 * maxTCams = 10
 * maxViewAngle = 70
 * minViewAngle = 2
 * nbGPUs = 0
 * output = "/media/ssd2/arth/dev/meshroom_data/MeshroomCache/DepthMap/9730bb64166af1b2a54943cdc5650c2e57d85dfe"
 * rangeSize = 3
 * rangeStart = 0
 * refineEnabled = 1
 * refineGammaC = 15.5
 * refineGammaP = 8
 * refineHalfNbDepths = 15
 * refineInterpolateMiddleDepth = 0
 * refineMaxTCamsPerTile = 4
 * refineScale = 1
 * refineSigma = 15
 * refineStepXY = 1
 * refineSubsampling = 10
 * refineUseConsistentScale = 0
 * refineUseCustomPatchPattern = 0
 * refineWSH = 3
 * sgmDepthListPerTile = 1
 * sgmDepthThicknessInflate = 0
 * sgmFilteringAxes = "YX"
 * sgmGammaC = 5.5
 * sgmGammaP = 8
 * sgmMaxDepths = 1500
 * sgmMaxSimilarity = 1
 * sgmMaxTCamsPerTile = 4
 * sgmP1 = 10
 * sgmP2Weighting = 100
 * sgmScale = 2
 * sgmSeedsRangeInflate = 0.2
 * sgmStepXY = 2
 * sgmStepZ = -1
 * sgmUseConsistentScale = 0
 * sgmUseCustomPatchPattern = 0
 * sgmUseSfmSeeds = 1
 * sgmWSH = 4
 * tileBufferHeight = 1024
 * tileBufferWidth = 1024
 * tilePadding = 64
 * verboseLevel = "trace"

Hardware : 
    Detected core count : 40
    OpenMP will use 40 cores
    Detected available memory : 125550 Mo

[11:34:10.781777][info] CUDA-Enabled GPU.
Device information:
    - id:                      0
    - name:                    NVIDIA GeForce GTX 980
    - compute capability:      5.2
    - clock frequency (kHz):   1215500
    - total device memory:     4037 MB 
    - device memory available: 3707 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   2048
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {65536,65536}
    - max 3D array texture:    {4096,4096,4096}
    - max 2D linear texture:   {65536,65536,1048544}
    - max 2D layered texture:  {16384,16384,2048}
    - number of SM(x)s:        16
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte
Device information:
    - id:                      1
    - name:                    NVIDIA GeForce GTX 980
    - compute capability:      5.2
    - clock frequency (kHz):   1215500
    - total device memory:     4037 MB 
    - device memory available: 3952 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   2048
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {65536,65536}
    - max 3D array texture:    {4096,4096,4096}
    - max 2D linear texture:   {65536,65536,1048544}
    - max 2D layered texture:  {16384,16384,2048}
    - number of SM(x)s:        16
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte
Device information:
    - id:                      2
    - name:                    NVIDIA GeForce GTX 980
    - compute capability:      5.2
    - clock frequency (kHz):   1215500
    - total device memory:     4037 MB 
    - device memory available: 3952 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   2048
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {65536,65536}
    - max 3D array texture:    {4096,4096,4096}
    - max 2D linear texture:   {65536,65536,1048544}
    - max 2D layered texture:  {16384,16384,2048}
    - number of SM(x)s:        16
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte
Device information:
    - id:                      3
    - name:                    NVIDIA GeForce GTX 980
    - compute capability:      5.2
    - clock frequency (kHz):   1215500
    - total device memory:     4037 MB 
    - device memory available: 3952 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   2048
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {65536,65536}
    - max 3D array texture:    {4096,4096,4096}
    - max 2D linear texture:   {65536,65536,1048544}
    - max 2D layered texture:  {16384,16384,2048}
    - number of SM(x)s:        16
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte
CUDA build version: 11.3
[11:34:11.384376][info] Supported CUDA-Enabled GPU detected.
[11:34:11.385047][debug] 0 samples found in this animated xform.
[11:34:11.385145][debug] 0 samples found in this animated xform.
[11:34:11.390390][debug] 0 samples found in this animated xform.
[11:34:11.390427][debug] 0 samples found in this animated xform.
[11:34:11.390453][debug] 0 samples found in this animated xform.
[11:34:11.552035][info] Found 1 image dimension(s): 
[11:34:11.552108][info]     - [4000x3000]
[11:34:11.560629][debug] Reading view 90093706 projection matrix from image metadata.
[11:34:11.561457][debug] Reading view 177338633 projection matrix from image metadata.
[11:34:11.562272][debug] Reading view 247999453 projection matrix from image metadata.
[11:34:11.563045][debug] Reading view 328912215 projection matrix from image metadata.
[11:34:11.563800][debug] Reading view 331824189 projection matrix from image metadata.
[11:34:11.564574][debug] Reading view 379178484 projection matrix from image metadata.
[11:34:11.565356][debug] Reading view 419277979 projection matrix from image metadata.
[11:34:11.566124][debug] Reading view 426776291 projection matrix from image metadata.
[11:34:11.566889][debug] Reading view 483435510 projection matrix from image metadata.
[11:34:11.567750][debug] Reading view 529349387 projection matrix from image metadata.
[11:34:11.568841][debug] Reading view 594186769 projection matrix from image metadata.
[11:34:11.569647][debug] Reading view 645343765 projection matrix from image metadata.
[11:34:11.570417][debug] Reading view 746643463 projection matrix from image metadata.
[11:34:11.571208][debug] Reading view 748132734 projection matrix from image metadata.
[11:34:11.571983][debug] Reading view 825258686 projection matrix from image metadata.
[11:34:11.572750][debug] Reading view 868458939 projection matrix from image metadata.
[11:34:11.573527][debug] Reading view 923701541 projection matrix from image metadata.
[11:34:11.574295][debug] Reading view 941727634 projection matrix from image metadata.
[11:34:11.575065][debug] Reading view 961092562 projection matrix from image metadata.
[11:34:11.575834][debug] Reading view 997370462 projection matrix from image metadata.
[11:34:11.576607][debug] Reading view 1125796016 projection matrix from image metadata.
[11:34:11.577420][debug] Reading view 1239382617 projection matrix from image metadata.
[11:34:11.578191][debug] Reading view 1257043265 projection matrix from image metadata.
[11:34:11.578973][debug] Reading view 1293779064 projection matrix from image metadata.
[11:34:11.579744][debug] Reading view 1306741363 projection matrix from image metadata.
[11:34:11.580514][debug] Reading view 1438913142 projection matrix from image metadata.
[11:34:11.581303][debug] Reading view 1455680296 projection matrix from image metadata.
[11:34:11.582072][debug] Reading view 1476867501 projection matrix from image metadata.
[11:34:11.582831][debug] Reading view 1499714136 projection matrix from image metadata.
[11:34:11.583601][debug] Reading view 1527977600 projection matrix from image metadata.
[11:34:11.584374][debug] Reading view 1536321766 projection matrix from image metadata.
[11:34:11.585137][debug] Reading view 1592705556 projection matrix from image metadata.
[11:34:11.586774][debug] Reading view 1831411506 projection matrix from image metadata.
[11:34:11.588409][debug] Reading view 1840656848 projection matrix from image metadata.
[11:34:11.589645][debug] Reading view 1867465813 projection matrix from image metadata.
[11:34:11.590410][debug] Reading view 1874550607 projection matrix from image metadata.
[11:34:11.591175][debug] Reading view 1876088828 projection matrix from image metadata.
[11:34:11.591949][debug] Reading view 1917171884 projection matrix from image metadata.
[11:34:11.592717][debug] Reading view 1970770218 projection matrix from image metadata.
[11:34:11.593499][debug] Reading view 2021819100 projection matrix from image metadata.
[11:34:11.593528][info] Overall maximum dimension: [4000x3000]
[11:34:11.593588][info] Tiling information: 
    - parameters: 
          - buffer width:  1024 px
          - buffer height: 1024 px
          - padding: 64 px
    - maximum downscale:  4
    - maximum image width:  2000 px
    - maximum image height: 1500 px
    - maximum effective tile width:  896 px
    - maximum effective tile height: 896 px
    - # tiles on X-side: 3
    - # tiles on Y-side: 2
    - effective tile width:  668 px
    - effective tile height: 752 px
    - tile list: 
       - tile (1/6) size: 732x816 px, roi: [x: 0-732, y: 0-816]
       - tile (2/6) size: 732x748 px, roi: [x: 0-732, y: 752-1500]
       - tile (3/6) size: 732x816 px, roi: [x: 668-1400, y: 0-816]
       - tile (4/6) size: 732x748 px, roi: [x: 668-1400, y: 752-1500]
       - tile (5/6) size: 664x816 px, roi: [x: 1336-2000, y: 0-816]
       - tile (6/6) size: 664x748 px, roi: [x: 1336-2000, y: 752-1500]
[11:34:11.593623][info] SGM parameters:
    - scale: 2
    - stepXY: 2
[11:34:11.593653][info] Refine parameters:
    - scale: 1
    - stepXY: 1
[11:34:11.594610][debug] 4 CUDA devices found:
    - Device 0: NVIDIA GeForce GTX 980
    - Device 1: NVIDIA GeForce GTX 980
    - Device 2: NVIDIA GeForce GTX 980
    - Device 3: NVIDIA GeForce GTX 980
[11:34:11.594641][info] Number of GPU devices: 4, number of CPU threads: 40
[11:34:11.595075][info] CPU thread 0 (of 4) uses CUDA device: 0
[11:34:11.595173][info] CPU thread 3 (of 4) uses CUDA device: 3
[11:34:11.595187][info] CPU thread 2 (of 4) uses CUDA device: 2
[11:34:11.595216][info] CPU thread 1 (of 4) uses CUDA device: 1
[11:34:11.599887][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.599920][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.600273][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.600298][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.600854][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.600906][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.601284][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.601320][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.601479][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.601504][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.601689][debug] GPU 3D allocation: 256x256x1500, type size=1, pitch=512
[11:34:11.601733][debug]                  : 98304000, padded=196608000, wasted=98304000, wasted ratio=100%
[11:34:11.631941][debug] GPU 3D allocation: 1024x1024x31, type size=2, pitch=2048
[11:34:11.632018][debug]                  : 65011712, padded=65011712, wasted=0, wasted ratio=0%
[11:34:11.632147][debug] GPU 3D allocation: 1024x1024x31, type size=2, pitch=2048
[11:34:11.632180][debug]                  : 65011712, padded=65011712, wasted=0, wasted ratio=0%
[11:34:11.632652][debug] GPU 3D allocation: 1024x1024x31, type size=2, pitch=2048
[11:34:11.632682][debug]                  : 65011712, padded=65011712, wasted=0, wasted ratio=0%
[11:34:11.639445][info] Device memory:
    - available: 2965.65 MB
    - requirement for the first tile: 850.092 MB
    - # computation buffers per tile: 472.437 MB (Sgm: 378.437 MB, Refine: 94 MB)
    - # input images (R + 10 Ts): 377.655 MB (single mipmap image size: 34.3323 MB)
[11:34:11.639491][debug] Theoretical device memory cost for a tile without padding: 284.936 MB (Sgm: 190.936 MB, Refine: 94 MB)
[11:34:11.639508][info] Parallelization:
    - # tiles per image: 6
    - # simultaneous depth maps computation: 1
    - # simultaneous tiles computation: 5
aliceVision_depthMapEstimation: /opt/AliceVision_git/src/aliceVision/depthMap/cuda/host/DeviceStreamManager.cpp:17: aliceVision::depthMap::DeviceStreamManager::DeviceStreamManager(int): Assertion `nbStreams > 0' failed.
belveder79 commented 1 year ago

seems to be linked to the graphics card type GTX980 - it works on single RTX3060 but also fails on multi-GPU setup with 4 NVIDIA TITAN V cards...

belveder79 commented 1 year ago

ok, now I have tested it on four setups using MeshRoom 2023.2.0...

1x 4-TITAN-V - failed 1x 4-GTX980 - failed 1x 1-GTS1060 - worked 1x 1-GTX3060 - worked

seems to be a multi-GPU issue?

aashutoshpy commented 1 year ago

I have the same issue, but it is working when rolled back to Meshroom 2023.1.0. Tested on g4dn.12xlarge

DerrangedGadgeteer commented 1 year ago

I encountered this same error using the current (Sept 2, 2023) precompiled Linux binary, with all default settings on my own photo dataset.

My system uses one RTX 2080 Super for graphics output, and 4x Tesla P100's for compute. OS: Ubuntu 22.04

I believe I can support the Multi-GPU hypothesis. If I use the system variables to hide all but one GPU from Meshroom, the computation of Depth Maps goes ahead without error.

user@computer:~$ export CUDA_VISIBLE_DEVICES="1" user@computer:~$ ./Meshroom

Note: the CUDA_VISIBLE_DEVICES variable takes either a GPU ID number, or a comma-separated list of GPU ID Numbers. So if your desired GPU is ID: 0, then you'd put ="0" or if you're trying multi-gpu, it should be something more like ="0,1,3," etc.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.