alicevision / Meshroom

3D Reconstruction Software
http://alicevision.org
Other
11.13k stars 1.08k forks source link

DepthMap step fails in 2019.2 #593

Closed MightyBOBcnc closed 3 years ago

MightyBOBcnc commented 5 years ago

Many users, including myself, ran into an issue in 2019.1 where the DepthMap step would frequently fail, but would not have this issue in 2018.1. See #409

Unfortunately this continues to happen in 2019.2. I am running a set of 52 photos that works perfectly fine in 2018.1, but failed in 2019.1 and now fail 2019.2 as well. 52/52 photos have been successfully matched but the DepthMap step immediately fails with no error message.

I'm so sad that I'm stuck on 2018.1. :( Let me know if I can provide any additional information to help.

Program called with the following parameters:
 * downscale = 1
 * exportIntermediateResults = 0
 * imagesFolder = "C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074"
 * input = "C:/Users/MightyBOBcnc/Documents/MeshroomCache/StructureFromMotion/aea25df45110183e26e96f2e7a6f670ca2d22e76/sfm.abc"
 * maxViewAngle = 70
 * minViewAngle = 2
 * nbGPUs = 0
 * output = "C:/Users/MightyBOBcnc/Documents/MeshroomCache/DepthMap/d218bfd7f81a4ecc8c8d23b59ef263f6330d26ed"
 * rangeSize = 3
 * rangeStart = 0
 * refineGammaC = 15.5
 * refineGammaP = 8
 * refineMaxTCams = 6
 * refineNDepthsToRefine = 31
 * refineNSamplesHalf = 150
 * refineNiters = 100
 * refineSigma = 15
 * refineUseTcOrRcPixSize = 0
 * refineWSH = 3
 * sgmGammaC = 5.5
 * sgmGammaP = 8
 * sgmMaxTCams = 10
 * sgmWSH = 4
 * verboseLevel = "trace"

[16:22:09.271649][info] CUDA-Enabled GPU.
Device information:
    - id:                      0
    - name:                    GeForce GTX 870M
    - compute capability:      3.0
    - total device memory:     3072 MB 
    - device memory available: 2535 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   2048
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {65536,65536}
    - max 3D array texture:    {4096,4096,4096}
    - max 2D linear texture:   {65000,65000,1048544}
    - max 2D layered texture:  {16384,16384,2048}
    - number of SM(x)s:        7
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte

[16:22:09.526676][info] Supported CUDA-Enabled GPU detected.
[16:22:09.542285][debug] 0 samples found in this animated xform.
[16:22:09.542285][debug] 0 samples found in this animated xform.
[16:22:09.547826][debug] 0 samples found in this animated xform.
[16:22:09.547826][debug] 0 samples found in this animated xform.
[16:22:09.547826][debug] 0 samples found in this animated xform.
[16:22:09.717173][info] Found 1 image dimension(s): 
[16:22:09.717173][info]     - [6000x4000]
[16:22:09.717173][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/138937560.exr
[16:22:09.737044][debug] Reading view 138937560 projection matrix from image metadata.
[16:22:09.738175][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/182304928.exr
[16:22:09.738175][debug] Reading view 182304928 projection matrix from image metadata.
[16:22:09.738175][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/185748809.exr
[16:22:09.738175][debug] Reading view 185748809 projection matrix from image metadata.
[16:22:09.738175][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/188977424.exr
[16:22:09.738175][debug] Reading view 188977424 projection matrix from image metadata.
[16:22:09.738175][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/189305965.exr
[16:22:09.738175][debug] Reading view 189305965 projection matrix from image metadata.
[16:22:09.738175][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/314926199.exr
[16:22:09.758265][debug] Reading view 314926199 projection matrix from image metadata.
[16:22:09.758265][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/324593110.exr
[16:22:09.759409][debug] Reading view 324593110 projection matrix from image metadata.
[16:22:09.759409][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/416300754.exr
[16:22:09.759409][debug] Reading view 416300754 projection matrix from image metadata.
[16:22:09.759409][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/418802622.exr
[16:22:09.759409][debug] Reading view 418802622 projection matrix from image metadata.
[16:22:09.759409][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/454723133.exr
[16:22:09.759409][debug] Reading view 454723133 projection matrix from image metadata.
[16:22:09.759409][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/472207525.exr
[16:22:09.778396][debug] Reading view 472207525 projection matrix from image metadata.
[16:22:09.779396][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/525194669.exr
[16:22:09.780522][debug] Reading view 525194669 projection matrix from image metadata.
[16:22:09.780522][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/526255660.exr
[16:22:09.780522][debug] Reading view 526255660 projection matrix from image metadata.
[16:22:09.780522][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/552946041.exr
[16:22:09.780522][debug] Reading view 552946041 projection matrix from image metadata.
[16:22:09.780522][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/568878920.exr
[16:22:09.780522][debug] Reading view 568878920 projection matrix from image metadata.
[16:22:09.780522][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/609468370.exr
[16:22:09.799531][debug] Reading view 609468370 projection matrix from image metadata.
[16:22:09.799531][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/686262615.exr
[16:22:09.801654][debug] Reading view 686262615 projection matrix from image metadata.
[16:22:09.801654][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/696597151.exr
[16:22:09.801654][debug] Reading view 696597151 projection matrix from image metadata.
[16:22:09.801654][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/725978194.exr
[16:22:09.801654][debug] Reading view 725978194 projection matrix from image metadata.
[16:22:09.801654][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/763733817.exr
[16:22:09.801654][debug] Reading view 763733817 projection matrix from image metadata.
[16:22:09.801654][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/795654320.exr
[16:22:09.818496][debug] Reading view 795654320 projection matrix from image metadata.
[16:22:09.820508][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/811391851.exr
[16:22:09.822687][debug] Reading view 811391851 projection matrix from image metadata.
[16:22:09.822687][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/847271445.exr
[16:22:09.822687][debug] Reading view 847271445 projection matrix from image metadata.
[16:22:09.822687][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/854937190.exr
[16:22:09.822687][debug] Reading view 854937190 projection matrix from image metadata.
[16:22:09.822687][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/911328903.exr
[16:22:09.822687][debug] Reading view 911328903 projection matrix from image metadata.
[16:22:09.822687][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/917510181.exr
[16:22:09.842733][debug] Reading view 917510181 projection matrix from image metadata.
[16:22:09.842733][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/970210069.exr
[16:22:09.843859][debug] Reading view 970210069 projection matrix from image metadata.
[16:22:09.843859][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/982733772.exr
[16:22:09.843859][debug] Reading view 982733772 projection matrix from image metadata.
[16:22:09.843859][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1022442193.exr
[16:22:09.843859][debug] Reading view 1022442193 projection matrix from image metadata.
[16:22:09.843859][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1047345500.exr
[16:22:09.843859][debug] Reading view 1047345500 projection matrix from image metadata.
[16:22:09.843859][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1077337592.exr
[16:22:09.863857][debug] Reading view 1077337592 projection matrix from image metadata.
[16:22:09.863857][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1116765584.exr
[16:22:09.864990][debug] Reading view 1116765584 projection matrix from image metadata.
[16:22:09.864990][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1173590034.exr
[16:22:09.864990][debug] Reading view 1173590034 projection matrix from image metadata.
[16:22:09.864990][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1208148656.exr
[16:22:09.864990][debug] Reading view 1208148656 projection matrix from image metadata.
[16:22:09.864990][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1255699229.exr
[16:22:09.864990][debug] Reading view 1255699229 projection matrix from image metadata.
[16:22:09.864990][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1530813093.exr
[16:22:09.883930][debug] Reading view 1530813093 projection matrix from image metadata.
[16:22:09.884930][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1629838864.exr
[16:22:09.886051][debug] Reading view 1629838864 projection matrix from image metadata.
[16:22:09.886051][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1662174919.exr
[16:22:09.886051][debug] Reading view 1662174919 projection matrix from image metadata.
[16:22:09.886051][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1665799199.exr
[16:22:09.886051][debug] Reading view 1665799199 projection matrix from image metadata.
[16:22:09.886051][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1736740251.exr
[16:22:09.886051][debug] Reading view 1736740251 projection matrix from image metadata.
[16:22:09.886051][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1768208214.exr
[16:22:09.904935][debug] Reading view 1768208214 projection matrix from image metadata.
[16:22:09.905927][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1808188529.exr
[16:22:09.907028][debug] Reading view 1808188529 projection matrix from image metadata.
[16:22:09.907028][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1873865586.exr
[16:22:09.907028][debug] Reading view 1873865586 projection matrix from image metadata.
[16:22:09.907028][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1907589102.exr
[16:22:09.907028][debug] Reading view 1907589102 projection matrix from image metadata.
[16:22:09.907028][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1935991157.exr
[16:22:09.907028][debug] Reading view 1935991157 projection matrix from image metadata.
[16:22:09.907028][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1947623263.exr
[16:22:09.925795][debug] Reading view 1947623263 projection matrix from image metadata.
[16:22:09.926780][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1954777901.exr
[16:22:09.927943][debug] Reading view 1954777901 projection matrix from image metadata.
[16:22:09.927943][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1985161208.exr
[16:22:09.927943][debug] Reading view 1985161208 projection matrix from image metadata.
[16:22:09.927943][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/2011580237.exr
[16:22:09.927943][debug] Reading view 2011580237 projection matrix from image metadata.
[16:22:09.927943][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/2059519298.exr
[16:22:09.927943][debug] Reading view 2059519298 projection matrix from image metadata.
[16:22:09.927943][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/2095575051.exr
[16:22:09.946857][debug] Reading view 2095575051 projection matrix from image metadata.
[16:22:09.946857][debug] [IO] Read Image Metadata: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/2108408827.exr
[16:22:09.949002][debug] Reading view 2108408827 projection matrix from image metadata.
[16:22:09.949002][info] Overall maximum dimension: [6000x4000]
[16:22:09.949002][info] Create depth maps.
number of CUDA devices: 1
   0: GeForce GTX 870M
[16:22:09.949002][info] # GPU devices: 1, # CPU threads: 8
[16:22:09.949002][info] Plane sweeping parameters:
    - scale: 2
    - step: 5
[16:22:09.949002][info] PlaneSweepingCuda:
    - _nImgsInGPUAtTime: 2
    - scales: 2
    - subPixel: Yes
    - varianceWSH: 4
CUDA device no 0 for 0
Device 0 memory - used: 536.846863, free: 2535.153076, total: 3072.000000
Device 0 memory - used: 771.346863, free: 2300.653076, total: 3072.000000
[16:22:09.991299][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:09.991299][debug] allDepths: 6045
[16:22:09.991299][debug] All correct depths: 2012
[16:22:09.991299][debug] used depths: 2011
[16:22:10.006944][debug] allDepths: 4014
[16:22:10.008256][debug] All correct depths: 1711
[16:22:10.009259][debug] used depths: 0
[16:22:10.009259][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.012404][debug] allDepths: 6001
[16:22:10.012404][debug] All correct depths: 2143
[16:22:10.012404][debug] used depths: 2144
[16:22:10.012404][debug] allDepths: 6030
[16:22:10.012404][debug] All correct depths: 2552
[16:22:10.012404][debug] used depths: 0
[16:22:10.012404][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.033631][debug] allDepths: 5530
[16:22:10.033631][debug] All correct depths: 2467
[16:22:10.033631][debug] used depths: 0
[16:22:10.033631][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.033631][debug] allDepths: 5224
[16:22:10.033631][debug] All correct depths: 1315
[16:22:10.033631][debug] used depths: 0
[16:22:10.033631][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.054943][debug] allDepths: 4002
[16:22:10.054943][debug] All correct depths: 1654
[16:22:10.054943][debug] used depths: 0
[16:22:10.054943][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.054943][debug] allDepths: 6124
[16:22:10.054943][debug] All correct depths: 1569
[16:22:10.054943][debug] used depths: 0
[16:22:10.054943][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.076101][debug] allDepths: 4176
[16:22:10.076101][debug] All correct depths: 1342
[16:22:10.076101][debug] used depths: 0
[16:22:10.076101][debug] Compute min/max/mid/nb depth for view id: 138937560
[16:22:10.076101][debug] allDepths: 6084
[16:22:10.093532][debug] All correct depths: 2582
[16:22:10.096528][debug] used depths: 2069
[16:22:10.097728][debug] rc depths: 2803
[16:22:10.223169][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1255699229.exr
[16:22:10.665553][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1255699229.exr (encoded in Linear colorspace).
[16:22:10.736900][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1255699229.exr to image cache. Elapsed time: 00 minutes 00 seconds 622 miliseconds

[16:22:10.843537][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/454723133.exr
[16:22:11.307911][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/454723133.exr (encoded in Linear colorspace).
[16:22:11.366463][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/454723133.exr to image cache. Elapsed time: 00 minutes 00 seconds 635 miliseconds

[16:22:11.472866][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/526255660.exr
[16:22:11.959990][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/526255660.exr (encoded in Linear colorspace).
[16:22:12.020422][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/526255660.exr to image cache. Elapsed time: 00 minutes 00 seconds 639 miliseconds

[16:22:12.105593][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/811391851.exr
[16:22:12.594775][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/811391851.exr (encoded in Linear colorspace).
[16:22:12.654141][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/811391851.exr to image cache. Elapsed time: 00 minutes 00 seconds 631 miliseconds

[16:22:12.740486][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/188977424.exr
[16:22:13.216770][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/188977424.exr (encoded in Linear colorspace).
[16:22:13.289776][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/188977424.exr to image cache. Elapsed time: 00 minutes 00 seconds 635 miliseconds

[16:22:13.395313][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1736740251.exr
[16:22:13.872078][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1736740251.exr (encoded in Linear colorspace).
[16:22:13.929971][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1736740251.exr to image cache. Elapsed time: 00 minutes 00 seconds 645 miliseconds

[16:22:14.037340][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/418802622.exr
[16:22:14.505516][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/418802622.exr (encoded in Linear colorspace).
[16:22:14.567334][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/418802622.exr to image cache. Elapsed time: 00 minutes 00 seconds 630 miliseconds

[16:22:14.654448][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/185748809.exr
[16:22:15.136548][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/185748809.exr (encoded in Linear colorspace).
[16:22:15.196540][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/185748809.exr to image cache. Elapsed time: 00 minutes 00 seconds 630 miliseconds

[16:22:15.287569][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1665799199.exr
[16:22:15.772024][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1665799199.exr (encoded in Linear colorspace).
[16:22:15.832269][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1665799199.exr to image cache. Elapsed time: 00 minutes 00 seconds 633 miliseconds

[16:22:15.918222][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1629838864.exr
[16:22:16.388634][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1629838864.exr (encoded in Linear colorspace).
[16:22:16.462382][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1629838864.exr to image cache. Elapsed time: 00 minutes 00 seconds 628 miliseconds

[16:22:16.462382][info] Estimate depth map, view id: 138937560
[16:22:16.462382][debug] SGM (rc: 1 / 52)
[16:22:16.586960][debug] sweepPixelsVolume:
    - scale: 2
    - step: 1
    - npixels: 240000
    - volStepXY: 5
    - volDimX: 600
    - volDimY: 400
    - volDimZ: 1500
[16:22:16.718724][debug] [IO] Read Image: C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/138937560.exr
[16:22:17.211632][trace] Read image C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/138937560.exr (encoded in Linear colorspace).
[16:22:17.257401][debug] Add C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/138937560.exr to image cache. Elapsed time: 00 minutes 00 seconds 647 miliseconds

[16:22:17.774888][debug] copy image from disk to GPU  Elapsed time: 00 minutes 01 seconds 190 miliseconds

[16:22:17.774888][debug] rc: 0
tcams: 
[16:22:17.774888][debug]    - 34
[16:22:17.795839][debug] Reuse C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/1255699229.exr from image cache. 
[16:22:18.337876][debug] copy image from disk to GPU  Elapsed time: 00 minutes 00 seconds 563 miliseconds

Device 0 memory - used: 1357.284424, free: 1714.715576, total: 3072.000000
total size of volume map in GPU memory: 585.937500
nDepths 1500, nDepthsToSearch 1500 
ps_computeSimilarityVolume elapsed time: 28.000000 ms 
[16:22:31.945219][debug]  Elapsed time: 00 minutes 15 seconds 357 miliseconds

[16:22:31.946234][debug] SemiGlobalMatchingRcTc::computeDepthSimMapVolume  Elapsed time: 00 minutes 15 seconds 480 miliseconds

[16:22:31.947243][debug] GPU memory : free: 2300.65, total: 3072, used: 771.347
[16:22:31.948249][debug] GPU memory volume: 2189.06
[16:22:31.948249][warning] Low GPU memory volume step Z: 2
[16:22:33.269939][debug] sweepPixelsVolume:
    - scale: 2
    - step: 1
    - npixels: 240000
    - volStepXY: 5
    - volDimX: 600
    - volDimY: 400
    - volDimZ: 1500
[16:22:33.269939][debug] rc: 0
tcams: 
[16:22:33.269939][debug]    - 9
[16:22:33.269939][debug] Reuse C:/Users/MightyBOBcnc/Documents/MeshroomCache/PrepareDenseScene/3a90bd517183879f06096b82fd12787df4d31074/454723133.exr from image cache. 
[16:22:33.796523][debug] copy image from disk to GPU  Elapsed time: 00 minutes 00 seconds 526 miliseconds
norru commented 5 years ago

I am evaluating Meshroom and AliceVision and stumbled upon this showstopper at the start.

Meshroom version: 2019.2 OS: Ubuntu 18.04 x86_64 Nvidia driver: 418.78 Nvidia hardware: GeForce RTX 2080 Ti CUDA version: 10.1

$ nvidia-smi 
Fri Aug 16 14:44:48 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:04:00.0  On |                  N/A |
|  0%   49C    P8    24W / 300W |    558MiB / 10981MiB |     15%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2695      G   /usr/lib/xorg/Xorg                           426MiB |
...
+-----------------------------------------------------------------------------+
Program called with the following parameters:
 * downscale = 2
 * exportIntermediateResults = 0
 * imagesFolder = "/home/norru/Projects/ivt/meshroom_test/MeshroomCache/PrepareDenseScene/83b2802cbb853d07b8c1ceafa6b660cc88394795"
 * input = "/home/norru/Projects/ivt/meshroom_test/MeshroomCache/StructureFromMotion/cb48e0481690b2bac4d0c797a55bf6831c701f59/sfm.abc"
 * maxViewAngle = 70
 * minViewAngle = 2
 * nbGPUs = 0
 * output = "/home/norru/Projects/ivt/meshroom_test/MeshroomCache/DepthMap/cd9460337ef57214ceb1104a65d6a25fa4933b2f"
 * rangeSize = 3
 * rangeStart = 0
 * refineGammaC = 15.5
 * refineGammaP = 8
 * refineMaxTCams = 6
 * refineNDepthsToRefine = 31
 * refineNSamplesHalf = 150
 * refineNiters = 100
 * refineSigma = 15
 * refineUseTcOrRcPixSize = 0
 * refineWSH = 3
 * sgmGammaC = 5.5
 * sgmGammaP = 8
 * sgmMaxTCams = 10
 * sgmWSH = 4
 * verboseLevel = "info"

[14:38:05.437844][warning] Cannot get available memory information for CUDA gpu device 0.
[14:38:04.512995][warning] CUDA-Enabled GPU.
Device information:
    - id:                      0
    - name:                    GeForce RTX 2080 Ti
    - compute capability:      7.5
    - total device memory:     10981 MB 
    - device memory available: 0 MB 
    - per-block shared memory: 49152
    - warp size:               32
    - max threads per block:   1024
    - max threads per SM(X):   1024
    - max block sizes:         {1024,1024,64}
    - max grid sizes:          {2147483647,65535,65535}
    - max 2D array texture:    {131072,65536}
    - max 3D array texture:    {16384,16384,16384}
    - max 2D linear texture:   {131072,65000,2097120}
    - max 2D layered texture:  {32768,32768,2048}
    - number of SM(x)s:        68
    - registers per SM(x):     65536
    - registers per block:     65536
    - concurrent kernels:      yes
    - mapping host memory:     yes
    - unified addressing:      yes
    - texture alignment:       512 byte
    - pitch alignment:         32 byte

[14:38:05.438540][info] Supported CUDA-Enabled GPU detected.
[14:38:05.443070][info] Found 1 image dimension(s): 
[14:38:05.443097][info]     - [5472x3648]
[14:38:05.463809][info] Overall maximum dimension: [5472x3648]
[14:38:05.463898][info] Create depth maps.

CUDAError: unknown error
  file:       /opt/AliceVision_git/src/aliceVision/depthMap/cuda/planeSweeping/plane_sweeping_cuda.cu
  function:   ps_listCUDADevices
  line:       205

[14:38:05.464366][info] # GPU devices: 1, # CPU threads: 24
[14:38:05.464401][info] Plane sweeping parameters:
    - scale: 2
    - step: 4
[14:38:05.464442][info] PlaneSweepingCuda:
    - _nImgsInGPUAtTime: 2
    - scales: 2
    - subPixel: Yes
    - varianceWSH: 4
terminate called after throwing an instance of 'std::runtime_error'
  what():  Device alloc 2D array failed: unknown error
Aborted (core dumped)
fabiencastan commented 5 years ago

@norru Please open a dedicated issue, as you pbl is specific to the 2019.2 release binaries on linux. So I will remove your comment from this PR about windows.

fabiencastan commented 5 years ago

@MightyBOBcnc : Could you set the verbose to "trace" and send the log with the 2019.2 release? Is it the same error with 2019.1 and 2019.2?

MightyBOBcnc commented 5 years ago

@fabiencastan The log I included in the issue report is already set to trace. That's the full trace log for the DepthMap step in 2019.2.

The last line in the log is: [16:22:33.796523][debug] copy image from disk to GPU Elapsed time: 00 minutes 00 seconds 526 miliseconds and then the DepthMap fails silently <5 seconds after that without writing anything else to the log.

I have taken a look in the Windows Event Viewer and I have found that aliceVision_depthMapEstimation.exe appears to be crashing. There is 1 "Error" level events and 1 "Information" level events that appear to be connected to the crash:

Faulting application name: aliceVision_depthMapEstimation.exe, version: 2.0.0.0, time stamp: 0x5d4c28ec
Faulting module name: ucrtbase.dll, version: 10.0.18362.267, time stamp: 0x080a13f7
Exception code: 0xc0000409
Fault offset: 0x000000000006d33e
Faulting process id: 0x2994
Faulting application start time: 0x01d55455fac373a5
Faulting application path: C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_depthMapEstimation.exe
Faulting module path: C:\WINDOWS\System32\ucrtbase.dll
Report Id: 647ba96f-9a0c-41dc-9d60-1d8bb34a6c55
Faulting package full name: 
Faulting package-relative application ID: 

and

Fault bucket 2130295549337383112, type 5
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: aliceVision_depthMapEstimation.exe
P2: 2.0.0.0
P3: 5d4c28ec
P4: ucrtbase.dll
P5: 10.0.18362.267
P6: 080a13f7
P7: 000000000006d33e
P8: c0000409
P9: 0000000000000007
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD183.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD2BD.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD2ED.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD2ED.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD30D.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_aliceVision_dept_5d7d516957a4f08eb65d9541212c5f4bc291bae0_58acabf4_910262ed-c0de-4365-b694-2de1727bcb6f

Analysis symbol: 
Rechecking for solution: 0
Report Id: 647ba96f-9a0c-41dc-9d60-1d8bb34a6c55
Report Status: 268435456
Hashed bucket: 70d6fe45de100cb98d905483ecd9b0c8
Cab Guid: 0

Here is the associated Windows error report:

Version=1
EventType=BEX64
EventTime=132104492504375502
ReportType=2
Consent=1
UploadTime=132104492508719711
ReportStatus=268435456
ReportIdentifier=910262ed-c0de-4365-b694-2de1727bcb6f
IntegratorReportIdentifier=647ba96f-9a0c-41dc-9d60-1d8bb34a6c55
Wow64Host=34404
NsAppName=aliceVision_depthMapEstimation.exe
AppSessionGuid=00002994-0001-001b-a573-c3fa5554d501
TargetAppId=W:0006a1becd8e4b1a401d3fe3b618506c34f200000904!0000ca899cbbfc87b878012355e3a6c0ae8102c9cc3d!aliceVision_depthMapEstimation.exe
TargetAppVer=2019//08//08:13:51:40!0!aliceVision_depthMapEstimation.exe
BootId=4294967295
ServiceSplit=3720940304
TargetAsId=253
IsFatal=1
EtwNonCollectReason=1
Response.BucketId=70d6fe45de100cb98d905483ecd9b0c8
Response.BucketTable=5
Response.LegacyBucketId=2130295549337383112
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=aliceVision_depthMapEstimation.exe
Sig[1].Name=Application Version
Sig[1].Value=2.0.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=5d4c28ec
Sig[3].Name=Fault Module Name
Sig[3].Value=ucrtbase.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=10.0.18362.267
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=080a13f7
Sig[6].Name=Exception Offset
Sig[6].Value=000000000006d33e
Sig[7].Name=Exception Code
Sig[7].Value=c0000409
Sig[8].Name=Exception Data
Sig[8].Value=0000000000000007
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.18362.2.0.0.768.101
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=7785
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=77857e0ccf5abe9e7c6d80eb88fb7ebb
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=33e6
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=33e61f5127b8023b9f017798aa17382e
UI[2]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_depthMapEstimation.exe
LoadedModule[0]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_depthMapEstimation.exe
LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\WINDOWS\System32\KERNEL32.DLL
LoadedModule[3]=C:\WINDOWS\System32\KERNELBASE.dll
LoadedModule[4]=C:\WINDOWS\System32\ucrtbase.dll
LoadedModule[5]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_mvsUtils.dll
LoadedModule[6]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_program_options-vc140-mt-x64-1_70.dll
LoadedModule[7]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_sfmDataIO.dll
LoadedModule[8]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_depthMap.dll
LoadedModule[9]=C:\WINDOWS\SYSTEM32\MSVCP140.dll
LoadedModule[10]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_system.dll
LoadedModule[11]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_gpu.dll
LoadedModule[12]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_log-vc140-mt-x64-1_70.dll
LoadedModule[13]=C:\WINDOWS\System32\ADVAPI32.dll
LoadedModule[14]=C:\WINDOWS\System32\WS2_32.dll
LoadedModule[15]=C:\WINDOWS\System32\msvcrt.dll
LoadedModule[16]=C:\WINDOWS\System32\sechost.dll
LoadedModule[17]=C:\WINDOWS\System32\RPCRT4.dll
LoadedModule[18]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\VCRUNTIME140.dll
LoadedModule[19]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_filesystem-vc140-mt-x64-1_70.dll
LoadedModule[20]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\OpenImageIO.dll
LoadedModule[21]=C:\WINDOWS\System32\SHELL32.dll
LoadedModule[22]=C:\WINDOWS\System32\cfgmgr32.dll
LoadedModule[23]=C:\WINDOWS\System32\shcore.dll
LoadedModule[24]=C:\WINDOWS\System32\combase.dll
LoadedModule[25]=C:\WINDOWS\System32\bcryptPrimitives.dll
LoadedModule[26]=C:\WINDOWS\System32\windows.storage.dll
LoadedModule[27]=C:\WINDOWS\System32\msvcp_win.dll
LoadedModule[28]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_mvsData.dll
LoadedModule[29]=C:\WINDOWS\System32\profapi.dll
LoadedModule[30]=C:\WINDOWS\System32\powrprof.dll
LoadedModule[31]=C:\WINDOWS\System32\UMPDC.dll
LoadedModule[32]=C:\WINDOWS\System32\shlwapi.dll
LoadedModule[33]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\zlib1.dll
LoadedModule[34]=C:\WINDOWS\System32\GDI32.dll
LoadedModule[35]=C:\WINDOWS\System32\win32u.dll
LoadedModule[36]=C:\WINDOWS\System32\gdi32full.dll
LoadedModule[37]=C:\WINDOWS\System32\USER32.dll
LoadedModule[38]=C:\WINDOWS\System32\kernel.appcore.dll
LoadedModule[39]=C:\WINDOWS\System32\cryptsp.dll
LoadedModule[40]=C:\WINDOWS\System32\ole32.dll
LoadedModule[41]=C:\WINDOWS\SYSTEM32\VCOMP140.DLL
LoadedModule[42]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\Alembic.dll
LoadedModule[43]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\cudart64_80.dll
LoadedModule[44]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_sfmData.dll
LoadedModule[45]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_regex-vc140-mt-x64-1_70.dll
LoadedModule[46]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_thread-vc140-mt-x64-1_70.dll
LoadedModule[47]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_multiview.dll
LoadedModule[48]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_date_time-vc140-mt-x64-1_70.dll
LoadedModule[49]=C:\WINDOWS\SYSTEM32\Secur32.dll
LoadedModule[50]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_feature.dll
LoadedModule[51]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\IlmImf-2_3.dll
LoadedModule[52]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\Imath-2_3.dll
LoadedModule[53]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_image.dll
LoadedModule[54]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\Iex-2_3.dll
LoadedModule[55]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\Half-2_3.dll
LoadedModule[56]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\libpng16.dll
LoadedModule[57]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\jpeg62.dll
LoadedModule[58]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\raw_r.dll
LoadedModule[59]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\tiff.dll
LoadedModule[60]=C:\WINDOWS\SYSTEM32\dbgeng.dll
LoadedModule[61]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\hdf5.dll
LoadedModule[62]=C:\WINDOWS\System32\bcrypt.dll
LoadedModule[63]=C:\WINDOWS\System32\OLEAUT32.dll
LoadedModule[64]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\IlmThread-2_3.dll
LoadedModule[65]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_numeric.dll
LoadedModule[66]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\glog.dll
LoadedModule[67]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\ceres.dll
LoadedModule[68]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\CCTag.dll
LoadedModule[69]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\lcms2.dll
LoadedModule[70]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\jasper.dll
LoadedModule[71]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\opencv_features2d343.dll
LoadedModule[72]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\opencv_core343.dll
LoadedModule[73]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\lzma.dll
LoadedModule[74]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\popsift.dll
LoadedModule[75]=C:\WINDOWS\SYSTEM32\dbgmodel.dll
LoadedModule[76]=C:\WINDOWS\SYSTEM32\XmlLite.dll
LoadedModule[77]=C:\WINDOWS\SYSTEM32\SSPICLI.DLL
LoadedModule[78]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\gflags.dll
LoadedModule[79]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\openblas.dll
LoadedModule[80]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\tbb.dll
LoadedModule[81]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\boost_serialization-vc140-mt-x64-1_70.dll
LoadedModule[82]=C:\WINDOWS\SYSTEM32\OPENGL32.dll
LoadedModule[83]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\CONCRT140.dll
LoadedModule[84]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\opencv_imgproc343.dll
LoadedModule[85]=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\opencv_flann343.dll
LoadedModule[86]=C:\WINDOWS\SYSTEM32\GLU32.dll
LoadedModule[87]=C:\WINDOWS\SYSTEM32\dxcore.dll
LoadedModule[88]=C:\WINDOWS\SYSTEM32\dbghelp.dll
LoadedModule[89]=C:\WINDOWS\System32\IMM32.DLL
LoadedModule[90]=C:\Windows\system32\guard64.dll
LoadedModule[91]=C:\WINDOWS\system32\version.dll
LoadedModule[92]=C:\WINDOWS\system32\fltlib.dll
LoadedModule[93]=C:\WINDOWS\system32\nvcuda.dll
LoadedModule[94]=C:\WINDOWS\System32\SETUPAPI.dll
LoadedModule[95]=C:\WINDOWS\SYSTEM32\nvfatbinaryLoader.dll
LoadedModule[96]=C:\WINDOWS\system32\uxtheme.dll
LoadedModule[97]=C:\WINDOWS\system32\nvapi64.dll
LoadedModule[98]=C:\WINDOWS\SYSTEM32\dxgi.dll
LoadedModule[99]=C:\WINDOWS\SYSTEM32\DEVOBJ.dll
LoadedModule[100]=C:\WINDOWS\System32\WINTRUST.dll
LoadedModule[101]=C:\WINDOWS\System32\MSASN1.dll
LoadedModule[102]=C:\WINDOWS\System32\CRYPT32.dll
State[0].Key=Transport.DoneStage1
State[0].Value=1
OsInfo[0].Key=vermaj
OsInfo[0].Value=10
OsInfo[1].Key=vermin
OsInfo[1].Value=0
OsInfo[2].Key=verbld
OsInfo[2].Value=18362
OsInfo[3].Key=ubr
OsInfo[3].Value=295
OsInfo[4].Key=versp
OsInfo[4].Value=0
OsInfo[5].Key=arch
OsInfo[5].Value=9
OsInfo[6].Key=lcid
OsInfo[6].Value=1033
OsInfo[7].Key=geoid
OsInfo[7].Value=244
OsInfo[8].Key=sku
OsInfo[8].Value=101
OsInfo[9].Key=domain
OsInfo[9].Value=0
OsInfo[10].Key=prodsuite
OsInfo[10].Value=768
OsInfo[11].Key=ntprodtype
OsInfo[11].Value=1
OsInfo[12].Key=platid
OsInfo[12].Value=10
OsInfo[13].Key=sr
OsInfo[13].Value=0
OsInfo[14].Key=tmsi
OsInfo[14].Value=84246
OsInfo[15].Key=osinsty
OsInfo[15].Value=3
OsInfo[16].Key=iever
OsInfo[16].Value=11.295.18362.0-11.0.140
OsInfo[17].Key=portos
OsInfo[17].Value=0
OsInfo[18].Key=ram
OsInfo[18].Value=32676
OsInfo[19].Key=svolsz
OsInfo[19].Value=117
OsInfo[20].Key=wimbt
OsInfo[20].Value=0
OsInfo[21].Key=blddt
OsInfo[21].Value=190318
OsInfo[22].Key=bldtm
OsInfo[22].Value=1202
OsInfo[23].Key=bldbrch
OsInfo[23].Value=19h1_release
OsInfo[24].Key=bldchk
OsInfo[24].Value=0
OsInfo[25].Key=wpvermaj
OsInfo[25].Value=0
OsInfo[26].Key=wpvermin
OsInfo[26].Value=0
OsInfo[27].Key=wpbuildmaj
OsInfo[27].Value=0
OsInfo[28].Key=wpbuildmin
OsInfo[28].Value=0
OsInfo[29].Key=osver
OsInfo[29].Value=10.0.18362.295.amd64fre.19h1_release.190318-1202
OsInfo[30].Key=buildflightid
OsInfo[30].Value=E61D6EF4-31F5-4EFA-8D73-A62AB2676F2F.1
OsInfo[31].Key=edition
OsInfo[31].Value=Core
OsInfo[32].Key=ring
OsInfo[32].Value=Retail
OsInfo[33].Key=expid
OsInfo[33].Value=RS:379B
OsInfo[34].Key=containerid
OsInfo[35].Key=containertype
OsInfo[36].Key=edu
OsInfo[36].Value=0
FriendlyEventName=Stopped working
ConsentKey=BEX64
AppName=aliceVision_depthMapEstimation - AliceVision 2.2.0
AppPath=C:\Portable\Meshroom-2019.2.0\aliceVision\bin\aliceVision_depthMapEstimation.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=1700A2128AB29719FDC496CB94DD0F98
MetadataHash=-173061553

The same thing happens in release 2019.1, which is to say that aliceVision_depthMapEstimation.exe appears to be crashing and in the Windows Event Viewer there is a crash that implicates ucrtbase.dll

I am searching Google for errors related to ucrtbase.dll and I have found this so far: https://superuser.com/questions/1397010/application-crash-due-to-ucrtbase-dll

MightyBOBcnc commented 5 years ago

@fabiencastan another update: I created a dump of the crash. I can send it to you if you would like. It is 38MB.

I ran the dump through the Debugging Tools for Windows (WinDbg). This is the analysis:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(2994.2aa0): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
For analysis of this file, run !analyze -v
ucrtbase!abort+0x4e:
00007ffc`8b35d33e cd29            int     29h
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify timestamp for aliceVision_depthMapEstimation.exe
*** WARNING: Unable to verify timestamp for openblas.dll
*** WARNING: Unable to verify timestamp for nvcuda.dll
*** WARNING: Unable to verify timestamp for IlmThread-2_3.dll
*** WARNING: Unable to verify timestamp for OpenImageIO.dll

KEY_VALUES_STRING: 1

    Key  : Timeline.Process.Start.DeltaSec
    Value: 19

PROCESSES_ANALYSIS: 1

SERVICE_ANALYSIS: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

Timeline: !analyze.Start
    Name: <blank>
    Time: 2019-08-16T18:21:26.155Z
    Diff: 4034155 mSec

Timeline: Dump.Current
    Name: <blank>
    Time: 2019-08-16T17:14:12.0Z
    Diff: 0 mSec

Timeline: Process.Start
    Name: <blank>
    Time: 2019-08-16T17:13:53.0Z
    Diff: 19000 mSec

DUMP_CLASS: 2

DUMP_QUALIFIER: 400

CONTEXT:  (.ecxr)
rax=0000000000000001 rbx=0000000000000000 rcx=0000000000000007
rdx=000000000000000f rsi=00007ff6646f9bdc rdi=0000000000000000
rip=00007ffc8b35d33e rsp=000000bf162fcbc0 rbp=000000bf162ff9f0
 r8=0000020d984a5690  r9=00007ffc8c061ec0 r10=0000000000000013
r11=0000020d984b5570 r12=000000bf162fcde0 r13=ffffffffffffffff
r14=0000000000000001 r15=00007ffc8c2ab790
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
ucrtbase!abort+0x4e:
00007ffc`8b35d33e cd29            int     29h
Resetting default scope

FAULTING_IP: 
ucrtbase!abort+4e
00007ffc`8b35d33e cd29            int     29h

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ffc8b35d33e (ucrtbase!abort+0x000000000000004e)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000000000007
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT

PROCESS_NAME:  aliceVision_depthMapEstimation.exe

ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE_STR:  c0000409

EXCEPTION_PARAMETER1:  0000000000000007

WATSON_BKT_PROCSTAMP:  5d39f0fb

WATSON_BKT_PROCVER:  2019.0.0.0

WATSON_BKT_MODULE:  ucrtbase.dll

WATSON_BKT_MODSTAMP:  80a13f7

WATSON_BKT_MODOFFSET:  6d33e

WATSON_BKT_MODVER:  10.0.18362.267

MODULE_VER_PRODUCT:  Microsoft® Windows® Operating System

BUILD_VERSION_STRING:  18362.1.amd64fre.19h1_release.190318-1202

MODLIST_WITH_TSCHKSUM_HASH:  cd7534d90e6c9596ba6ec4f89f143d047823dd36

MODLIST_SHA1_HASH:  4084fb95910959220d2986ffb6338539ce7e59b4

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

DUMP_FLAGS:  94

DUMP_TYPE:  1

ANALYSIS_SESSION_HOST:  ANTEC

ANALYSIS_SESSION_TIME:  08-16-2019 14:21:26.0155

ANALYSIS_VERSION: 10.0.18362.1 x86fre

THREAD_ATTRIBUTES: 
OS_LOCALE:  ENU

BUGCHECK_STR:  FAIL_FAST_FATAL_APP_EXIT

DEFAULT_BUCKET_ID:  FAIL_FAST_FATAL_APP_EXIT

PRIMARY_PROBLEM_CLASS:  FAIL_FAST

PROBLEM_CLASSES: 

    ID:     [0n282]
    Type:   [FAIL_FAST]
    Class:  Primary
    Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
            BUCKET_ID
    Name:   Add
    Data:   Omit
    PID:    [Unspecified]
    TID:    [Unspecified]
    Frame:  [0]

    ID:     [0n271]
    Type:   [FATAL_APP_EXIT]
    Class:  Addendum
    Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
            BUCKET_ID
    Name:   Add
    Data:   Omit
    PID:    [Unspecified]
    TID:    [Unspecified]
    Frame:  [0]

LAST_CONTROL_TRANSFER:  from 00007ffc8b35ca7f to 00007ffc8b35d33e

STACK_TEXT:  
000000bf`162fcbc0 00007ffc`8b35ca7f : 00000000`00000003 00000000`00000003 00000000`00000000 00007ff6`646f9000 : ucrtbase!abort+0x4e
000000bf`162fcbf0 00007ff6`646f9c13 : 00000000`00003000 00000020`00001000 00000000`01000000 00000000`00000000 : ucrtbase!terminate+0x1f
000000bf`162fcc20 00000000`00003000 : 00000020`00001000 00000000`01000000 00000000`00000000 00000000`00000000 : aliceVision_depthMapEstimation+0x19c13
000000bf`162fcc28 00000020`00001000 : 00000000`01000000 00000000`00000000 00000000`00000000 00007ffc`8c13fbca : 0x3000
000000bf`162fcc30 00000000`01000000 : 00000000`00000000 00000000`00000000 00007ffc`8c13fbca 00000000`00000000 : 0x00000020`00001000
000000bf`162fcc38 00000000`00000000 : 00000000`00000000 00007ffc`8c13fbca 00000000`00000000 00007ff6`646f9bdc : 0x1000000

THREAD_SHA1_HASH_MOD_FUNC:  041c5cbc22cc630155d0bd903e2c80ce982778c8

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  855a67a898f044d235c8525585417dfef9c5b1c3

THREAD_SHA1_HASH_MOD:  4ff6ac4a28a0b53422e8962b825743ee47a186e0

FOLLOWUP_IP: 
ucrtbase!abort+4e
00007ffc`8b35d33e cd29            int     29h

FAULT_INSTR_CODE:  15ba29cd

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ucrtbase!abort+4e

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ucrtbase

IMAGE_NAME:  ucrtbase.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  80a13f7

STACK_COMMAND:  ~0s ; .ecxr ; kb

BUCKET_ID:  FAIL_FAST_FATAL_APP_EXIT_ucrtbase!abort+4e

FAILURE_EXCEPTION_CODE:  c0000409

FAILURE_IMAGE_NAME:  ucrtbase.dll

BUCKET_ID_IMAGE_STR:  ucrtbase.dll

FAILURE_MODULE_NAME:  ucrtbase

BUCKET_ID_MODULE_STR:  ucrtbase

FAILURE_FUNCTION_NAME:  abort

BUCKET_ID_FUNCTION_STR:  abort

BUCKET_ID_OFFSET:  4e

BUCKET_ID_MODTIMEDATESTAMP:  80a13f7

BUCKET_ID_MODCHECKSUM:  ffdfb

BUCKET_ID_MODVER_STR:  10.0.18362.267

BUCKET_ID_PREFIX_STR:  FAIL_FAST_FATAL_APP_EXIT_

FAILURE_PROBLEM_CLASS:  FAIL_FAST

FAILURE_SYMBOL_NAME:  ucrtbase.dll!abort

FAILURE_BUCKET_ID:  FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/aliceVision_depthMapEstimation.exe/2019.0.0.0/5d39f0fb/ucrtbase.dll/10.0.18362.267/ 80a13f7/c0000409/0006d33e.htm?Retriage=1

TARGET_TIME:  2019-08-16T17:14:12.000Z

OSBUILD:  18362

OSSERVICEPACK:  86

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  768

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt SingleUserTS Personal

USER_LCID:  0

OSBUILD_TIMESTAMP:  unknown_date

BUILDDATESTAMP_STR:  190318-1202

BUILDLAB_STR:  19h1_release

BUILDOSVER_STR:  10.0.18362.1.amd64fre.19h1_release.190318-1202

ANALYSIS_SESSION_ELAPSED_TIME:  4f9a

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:fail_fast_fatal_app_exit_c0000409_ucrtbase.dll!abort

FAILURE_ID_HASH:  {e31753ac-c98a-8055-3663-47e707543d20}

Followup:     MachineOwner
---------

What stands out to me is this:

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application
natowi commented 5 years ago

Just a check: Did you make sure your Antivirus software does not interfere?

MightyBOBcnc commented 5 years ago

I have disabled the antivirus software (and also disabled its services in Services.msc) so I do not believe that it is interfering. The antivirus log files also do not contain any alerts for any Meshroom/Alicevision executables.

ChrisDWhitehead commented 5 years ago

Just to add more data points to the discussion...

I also have a program termination "aliceVision_depthMapEstimation - AliceVision 2.1.0 has stopped working" for ver 2019.1 , and a program termination "aliceVision_depthMapEstimation - AliceVision 2.2.0 has stopped working" along with a kernel fault in the Nvidia display driver for ver 2019.2 . I have this error on two different machines, both with Nvidia GT730 graphics cards.

I am using the MonsterTree sample files. The 2018.1 version will finish the mesh correctly.

natowi commented 5 years ago

Meshroom works fine for me, but I also have this in my system log:

P1: aliceVision_incrementalSfM.exe P2: 2.1.0.0 P3: 5c61901e P4: ucrtbase.dll P5: 10.0.17134.677 P6: 9f346d3f P7: 000000000006e14e P8: c0000409 P9: 0000000000000007 P10:

same with

aliceVision_featureMatching.exe

aliceVision_depthMapFiltering.exe

This crash info is in my system log (event viewer), pointing to the Meshroom path of my 2019.1 installation. There are app crash reports for today even tough I did not start MR...

For aliceVision_depthMapEstimation.exe I found this entry from an actual MR 2019.2 crash two days ago:

Event name: RADAR_PRE_LEAK_64

with no other useful information.

ChrisDWhitehead commented 5 years ago

I forgot to add.. I'm using Win 7 64 (all current updates), and my Nvidia driver is 431.60 (latest), DirectX 11 The first machine is FX-8350 with 8GB memory, the second is Athlon 7450 with 2GB memory.

voto1979 commented 5 years ago

I run into problems with meshroom … if i save the project in my (user) documents Folder ... but if i save the project on my desktop or on any other drive or folder it works fine.

natowi commented 5 years ago

@voto1979 Do you have special characters/non-ASCII characters in your file paths (#209)? Than this is the cause and not related to this specific issue.

voto1979 commented 5 years ago

@natowi … no I use a "mklink" for my user documents folder!

C:\Users\Name\Documents\

=

D:\Documents\

(C:\Users\Name\Documents) is linked to (D:\Documents). If I open the project over (C:\Users\Name\Documents\ it stops on the "StructureFromMotion" node and if I open it trough (D:\Documents) it works without any issues.

mklink /J "C:\Users\Name\Documents\" "D:\Documents\"

Sorry for confusing everybody!

MightyBOBcnc commented 5 years ago

This sounds specific to symbolic link junctions. Maybe reopen #599.

MightyBOBcnc commented 5 years ago

Update: The aliceVision_depthMapEstimation.exe crash occurs when downscale=1. I changed to downscale=2 and it completed the task but this is a severe hit to quality of the mesh that can be extracted in the subsequent meshing steps. By comparison, release 2018.1 does not crash with downscale=1.

Baasje85 commented 5 years ago

Can confirm the windows event log error; for some projects downscale is > 1 works, for others we had to increase it to 4. Our last attempt fails at chunk 32 of 37 on Downscale = 2, while 4 works.

 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2019-08-26T04:15:39.307564400Z" /> 
  <EventRecordID>70692</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>DESKTOP-TR6MJER</Computer> 
  <Security /> 
  </System>
 <EventData>
  <Data>aliceVision_depthMapEstimation.exe</Data> 
  <Data>2.0.0.0</Data> 
  <Data>5d4c28ec</Data> 
  <Data>ucrtbase.dll</Data> 
  <Data>10.0.17763.404</Data> 
  <Data>490b0aeb</Data> 
  <Data>c0000409</Data> 
  <Data>000000000006e91e</Data> 
  <Data>410</Data> 
  <Data>01d55bc4a031309d</Data> 
  <Data>D:\Mijn Downloads\Meshroom-2019.2.0-win64\aliceVision\bin\aliceVision_depthMapEstimation.exe</Data> 
  <Data>C:\WINDOWS\System32\ucrtbase.dll</Data> 
  <Data>63147727-4525-4377-a9ce-c4192c6823c1</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>
natowi commented 5 years ago

Can confirm the windows event log error; for some projects downscale is > 1 works, for others we had to increase it to 4.

This depends on the available memory. 8GB is the bare minimum for ~30-50 images (depending on resolution). 16gb are recommended for small scale reconstructions, 32gb+ for 100+ images. (all estimations)

If you don´t have enough memory, you need to increase the downscale factor accordingly.

MightyBOBcnc commented 5 years ago

I don't believe that system RAM is the bottleneck here. I have 32GB of RAM and aliceVision_depthMapEstimation.exe 2019.2 crashes on datasets of even only 40-50 images.

I've tested the Buddha and Monstree datasets and they both crash when downscale=1. Buddha crashes on the 19th chunk. Monstree crashes immediately on the 1st chunk.

2018.1 does not crash on any dataset that I throw at it including datasets that are 3x the size of Buddha and Monstree.

R0adRunn3r commented 5 years ago

Same problem here, same photos in 2018.1 worked fine... updated to 2019.2 depthMap crashes. I tried all the suggested fixes but the only increasing downscale seems to do something. Doesn't seem a RAM problem since my ram was always at 50%/max 70% all the time with all downscales factors; including the fact that with 2018.1 no problems occurred.

fabiencastan commented 5 years ago

I need to reproduce it to be able to fix it. But I never managed to reproduce a crash with the 2019.2 and I have tested lots of data sets. Could someone send me one or multiple data sets where you have the issue?

MightyBOBcnc commented 5 years ago

@fabiencastan You can download my data set (~912MB). However as I mentioned previously the Monstree data set and the Buddha data set also crash.

Curiously, though, the Buddha set does not crash until chunk 19 of the DepthMap stage and I'm struggling to understand why that would be the case. All of the images in the Buddha set have the exact same dimensions so I don't understand why it may crash on any individual image. If all of the images before chunk 19 work I would intuitively think that all the images should work all the way through. But I don't know how the underlying process works so all I can do is make wild non-programmer guesses (I'm just a nerdy artist) like if something is going out of bounds with the memory allocation or the garbage collector isn't deleting something.

I'm trying to uncover a pattern in the crashes but so far I don't see one. At first I thought it might be crashing if the images are larger than a certain number of megapixels but as we can see below the Buddha set crashes despite having a smaller number of megapixels than when my own data set has downscale=2 enabled, which does not crash.

My data set:          52 images | 6000x4000 | 24,000,000 pixels per image  | Instant crash
Monstree set:         41 images | 3024x4032 | 12,192,768 pixels per image  | Instant crash
My set downscale=2:   52 images | 3000x2000 | 6,000,000 pixels per image   | No crash
Buddha set:           67 images | 2736x1540 | 4,213,440 pixels per image   | Crash on chunk 19
Monstree downscale=2: 41 images | 1512x2016 | 3,048,192 pixels per image   | No crash

I did some more googling about the specific errors involved (0xc0000409 STATUS_STACK_BUFFER_OVERRUN and 0x7 FAST_FAIL_FATAL_APP_EXIT) and came across these articles here and here. I don't know if that's helpful in any way.

From the above list, the interesting one for me is the FAST_FAIL_FATAL_APP_EXIT code. This code is used when the C runtime function abort() gets called. And abort() is called by std::terminate(). And std::terminate() is called automatically for things like throwing an exception out of a noexcept function, or if a thrown exception goes unhandled. It’s also used by some modules when an internal assertion failure is hit, or when an error occurs and the program simply wants to give up rather than try to recover. These are not necessarily security issues. It’s just the program saying, “Um, I’m in trouble. I think I’ll just stop right here.”

I also have the crashdump .dmp files available to share if those would be of any help.

MightyBOBcnc commented 5 years ago

Small update: I installed Linux Mint 19.2 on the same laptop where I was having the DepthMap crashes and did not experience a crash when using the linux version of Meshroom 2019.2.0. So it looks like the crash is likely exclusive to the Windows binaries.

fabiencastan commented 5 years ago

@MightyBOBcnc Thanks for the feedback. BTW, for people who have the problem on Windows, it could make sense to try to update your drivers.

pierrepilou commented 4 years ago

same issue for me , windows 10, crash at depthmap with low downscale 1 or 2 (needs to set at downscale 4 to avoid crash) with 2019 relases. 2018 version works ok with downscale1... Is the only solution is to install Inux?? thanks

GitFrankHub commented 4 years ago

same issue for me , windows 10 (32 Gb), K3100 (4Gb) crash at depthmap with low downscale 1 or 2 (needs to set at downscale 4 to avoid crash) with 2019 relases. Factory machine works well Xenon W-2123 (16 Gb), Driver 388.26, Quadro P620 (2 Gb)

gameanimation3d commented 4 years ago

Same issue for me aswell Very inconsiting DepthMap Crashes. At only one machine the other more less same spec don't has this Issue. Consistent Crashes at Downscale 1. Inconsitent on Downscale 2

Problem Machine (Laptop): Windows 10 64bit GTX 960M 4GB VRAM 16GB RAM Driver: 441.66 (newest)

Working Machine: Windows 10 64bit GTX 970 3,5/4GB VRAM 16GB RAM Driver: 441.08

No Hints for an out of memory case. RAM and VRAM don't go over 60%

ingmaniac commented 4 years ago

Same Issue for me here Intel i7-5820k 64GB Ram Nvidia GeForce GTX 1060 3GB Nvidia GeForce GTX 750 Ti

Maybe 2018.1 used cpu to calculate depthmap so 64GB Ram was used? Now 2019.2 uses GPU fpr depthmap so it uses only 3GB from GTX 1060??

Can we set 2019.2 to use CPU instead of GPU aswell for testing purpose?

MightyBOBcnc commented 4 years ago

GPU has always been used to calculate depth maps, including in 2018.1. Something in the behavior changed between 2018.1 and 2019.1. Maybe related to AliceVision commits 671, 616, 642.

djoerg commented 4 years ago

Some comments here sound very simmilar to a problem i had with the DepthMapNode (and the low performance of my Quadro K3000M). If you experience random crashes after around 2 seconds of freeze, it's very likely related to a Windows feature called Timeout Detection and Recovery (TDR). This feature completely resets the GPU after a 2 second GPU-Freeze causing Meshroom to crash with the message "... stopped working". I posted more details on issue #741 recently.

I highly recommend to raise the "TdrDelay" value to several seconds (maybe even 1 or 2 minutes) while trying to find the root cause of this bug to eliminate the possible interference with the TDR feature!

MightyBOBcnc commented 4 years ago

Oh gosh I had never considered that the Windows TDR may be interfering with Meshroom. I've heard of TDR before because Substance Painter explicitly calls it out as part of their recommended program setup and documentation. They have a good tutorial for changing it if anyone wants to give it a try: https://docs.substance3d.com/spdoc/gpu-drivers-crash-with-long-computations-128745489.html

djoerg commented 4 years ago

I never heard of the TDR feature before i stumbled across bug report for Battlefield 5 lest week. I had problems with the DepthMap node since i started using MeshRoom in 2018. I couldn't go lower than downscale 4 and even that crashed sometimes. I blamed my poor K3000M and decided to wait using Meshroom for serious work until a buy a beefier machine. After raising the TdrDelay to 20sec a had no problems with downscale 2.

Maybe this should be mentioned in the MeshRoom FAQ?

davezap commented 4 years ago

Came here to figure out same issue. I did a side by side test of Monstree dataset in 2019.2.0 with all defaults I could not get past chunk 0 of DepthMap with Downscale=2, worked ok with Downscale=4. Under same conditions, 2018.1.0 would work with Downscale=2.

Interestingly I also have a GeForce GT 730 like another poster above and 16GB system memory Core i7-7700. Windows 10

Set registry values for TdrDelay and 2019.2 passes the first chunk now. Giving the registry settings here for others. Just copy paste into reg file and merge. Reboot required.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers]
"TdrDelay"=dword:0000003c
"TdrDdiDelay"=dword:0000003c

Last few lines of log with is was 'crashing', it really just stops.

[19:20:06.455351][info] PlaneSweepingCuda:
    - _nImgsInGPUAtTime: 2
    - scales: 2
    - subPixel: Yes
    - varianceWSH: 4
CUDA device no 0 for 0
Device 0 memory - used: 361.649994, free: 1686.349976, total: 2048.000000
Device 0 memory - used: 479.774994, free: 1568.224976, total: 2048.000000
[19:20:17.295567][info] Estimate depth map, view id: 12251753
Theaudorant commented 4 years ago

What is this ? I have a laptop with 8GB RAM. It's happen in the Depthmap. image

Theaudorant commented 4 years ago

and @davezap I don't understand where we have to do this.

natowi commented 4 years ago

@Theaudorant Your error log suggest that you do not have a nvidia GPU or your drivers are not up do date. The solution by davezap is for the case that depthmap fails during computation.

Theaudorant commented 4 years ago

And if I don't a GPU, can I use Meshroom ?

natowi commented 4 years ago

@Theaudorant Read https://github.com/alicevision/meshroom/wiki/Draft-Meshing

Theaudorant commented 4 years ago

ok thank you

Theaudorant commented 4 years ago

but there is nothing image

natowi commented 4 years ago

@Theaudorant The link works. Please navigate to https://github.com/alicevision/meshroom/wiki You can navigate to the page in the right hand side menu.

Theaudorant commented 4 years ago

ok thank you I found

Nathipha commented 4 years ago

I've never worked with Meshroom before and just downloaded version 2019.2.0 from the official website. I was watching this tutorial while letting it run with 22 jpg files and when it got to about 15% of "DepthMap", it just stopped. Pressing "Start" again doesn't work, it just stops again. There's no error message in the log, it just stops:

CUDA device no 0 for 1 Device 1 memory - used: 202.800003, free: 821.200012, total: 1024.000000 Device 0 memory - used: 884.200012, free: 3211.800049, total: 4096.000000 [16:13:05.854836][info] Found only 7/10 nearest cameras for view id: 135287188 Device 1 memory - used: 355.799988, free: 668.200012, total: 1024.000000 [16:13:05.871827][info] Found only 8/10 nearest cameras for view id: 279736322 [16:13:33.674459][info] Estimate depth map, view id: 135287188 Device 0 memory - used: 987.700012, free: 3108.300049, total: 4096.000000 total size of volume map in GPU memory: 103.464844 nDepths 981, nDepthsToSearch 981 ps_computeSimilarityVolume elapsed time: 89.000000 ms [16:13:39.028682][info] Estimate depth map, view id: 279736322

My PC: Win 10, Xeon E3-1220V2, GTX 710/GTX 1050Ti (drivers from November 2019), 8gb of RAM. The ".mg" file and the images are in "C:\misc\3dstuff\scan". 10 of the images have a "Camera: Not constructed" symbol in the upper right corner but "StructureFromMotion" finished successfully, so not sure if I have to do aynthing about that.

Edit: I'm currently testing the same images in version 2018.1 and, even though now 12 images are marked as "not reconstructed", "DepthMap" is now already at 30% (working on chunk 2).

stickseloni commented 4 years ago

For me updating GPU drivers & increasing tdr delay solved the issue

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers] "TdrDelay"=dword:0000003c "TdrDdiDelay"=dword:0000003c

MarkScT commented 4 years ago

Same problem, DepthMap step fails. Win10 64bit 8GB RAM GeForce GT470M

Workaround #1 set downscale factor to 4 - confirmed to work. Workaround #2 modify registry - confirmed to work even if downscale set to 2 or 1 (also confirmed reboot is required)

hoegge commented 4 years ago

Raising TDR values prevents the process from crashing, but the calculations never really finishes? Also, no calculations should make the GPU hang totally, should they?

remmel commented 4 years ago

Same issue with my NVIDIA GeForce GT 630m on Windows 10. Works "fine" on Meshroom 2018.1 (but the mesh is totally a mess) but; it stops on Estimate depth map, view id when running Meshroom 2019.2 GUI or CLI meshroom_photogrammetry.exe (2019.2) Maybe Meshroom 2019.2 is using CUDA toolkit > 8 which doesn't handle anymore CUDA 2.1 (GT 630 is CUDA 2.1) log-error.txt

Not the same cuda dll are used :

Also when running aliceVision_depthMapEstimation via command line; it goes further :

CUDA device no 0 for 0
Device 0 memory - used: 368.071869, free: 1679.928101, total: 2048.000000
Device 0 memory - used: 456.759369, free: 1591.240601, total: 2048.000000
[13:24:25.260260][info] Estimate depth map, view id: 105925454 (Remy: it normally stops here)
Device 0 memory - used: 495.009369, free: 1552.990601, total: 2048.000000
total size of volume map in GPU memory: 38.242188
nDepths 445, nDepthsToSearch 445
ps_computeSimilarityVolume elapsed time: 16.000000 ms

[EDIT] [SOLUTION] : Increase the TDR delay - it fixed the problem : works fine and texture mesh is good (tip found in Meshroom github ) - but seems slow as the DeshMap took 55min for that sample https://rcdata.capturingreality.com/testset-ga.zip

AlVal7 commented 4 years ago

I confirm this solution for Nvidia GT 710 and Windows 7. Explicit setting of the TDR delay parameter helped to solve the problem (in my case, 8 seconds was enough). Everything is great now!

hoegge commented 4 years ago

On my Lenovo W530 with a Nvidia Quadro K1000M card, everything stalls if I change the TDR setting to 60. The machine becomes unresponsive several seconds at the time, and then I can move the mouse for 2 seconds and it stalls again as long as the reconstruction runs - and after hours it has still not finished. If I run the same on a new Laptop with a Quadro RTX4000 it finishes in a few minutes. If the same reconstructions runs fine on the 2018 release, there must be something wrong going on in the new versions.

mph070770 commented 3 years ago

The registry edits worked for me, thanks, and my frustration with meshroom is replaced with enjoyment in getting some textured meshes! I think the developers should add a note in the FAQs about this - as it was far from obvious what was happening until I found this post.

Manuel2021-lab commented 3 years ago

Hey together. I had the same problem. So i testet it out. First Test: 4000x3000px Photos - The problem came up. Sec. Test: 1920x1080 px Photos. Result: No Problem.

Try it , maybe it works 4U 2 . Edit: I used only 12 Pics. With more i doesnt work. :-(