[bug] CUDAError: the launch timed out and was terminated

[jiye33]

I tried to reconstruct mesh with Monstree Image Dataset. However, It didn't work. Is there anyone who know how to fix CUDAError? GPU: GeForce mx130 OS: windows 11

[2023-04-25 14:50:16.236261] [0x00002618] [trace] Embedded OCIO configuration file: 'C:\Meshroom-2023.1.0\aliceVision/share/aliceVision/config.ocio' found. [2023-04-25 14:50:32.051684] [0x00004794] [trace] Embedded OCIO configuration file: 'C:\Meshroom-2023.1.0\aliceVision/share/aliceVision/config.ocio' found. Program called with the following parameters:

• allowSingleView = 1
• allowedCameraModels = "pinhole,radial1,radial3,brown,fisheye4,fisheye1,3deanamorphic4,3deradial4,3declassicld"
• colorProfileDatabase = "" (default)
• defaultCameraModel = "" (default)
• defaultFieldOfView = 45
• defaultFocalLength = -1 (default)
• defaultFocalRatio = 1 (default)
• defaultOffsetX = 0 (default)
• defaultOffsetY = 0 (default)
• errorOnMissingColorProfile = 1 (default)
• groupCameraFallback = Unknown Type "enum EGroupCameraFallback"
• imageFolder = "" (default)
• input = "C:\Users\Lenovo\AppData\Local\Temp\tmpl_15r78a/CameraInit/ee5aca9875d6756077ac43f2ac3f4b6023cde933/viewpoints.sfm"
• lensCorrectionProfileInfo = "${ALICEVISION_LENS_PROFILE_INFO}"
• lensCorrectionProfileSearchIgnoreCameraModel = 1
• maxCoresAvailable = Unknown Type "unsigned int" (default)
• maxMemoryAvailable = 18446744073709551615 (default)
• output = "C:/Users/Lenovo/AppData/Local/Temp/tmpl_15r78a/CameraInit/ee5aca9875d6756077ac43f2ac3f4b6023cde933/cameraInit.sfm"
• rawColorInterpretation = Unknown Type "enum aliceVision::image::ERawColorInterpretation"
• sensorDatabase = "C:\Meshroom-2023.1.0\aliceVision\share\aliceVision\cameraSensors.db"
• verboseLevel = "info"
• viewIdMethod = Unknown Type "enum aliceVision::sfmDataIO::EViewIdMethod"
• viewIdRegex = ".*?(\d+)" (default)

Hardware : Detected core count : 8 OpenMP will use 8 cores Detected available memory : 2786 Mo

[14:50:32.200296][info] 0 profile(s) stored in the LCP database. [14:50:32.211264][warning] Some image(s) have no serial number to identify the camera/lens device. This makes it impossible to correctly group the images by device if you have used multiple identical (same model) camera devices. The reconstruction will assume that only one device has been used, so if 2 images share the same focal length approximation they will share the same internal camera parameters. 3 image(s) are concerned. [14:50:32.222473][info] CameraInit report:

• Views: 3

     - # with focal length initialization (from metadata): 3
     - # without metadata: 0
     - # with DCP color calibration (raw images only): 0
     - # with LCP lens distortion initialization: 0
     - # with LCP vignetting calibration: 0

• Cameras Intrinsics: 1

  WARNING:root:Downgrade status on node "DepthMap_1(0)" from Status.ERROR to Status.SUBMITTED

  • commandLine: aliceVision_depthMapEstimation --input "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/StructureFromMotion/b62e2ada7e73d6fec87d0bc78794398bf3c71234/sfm.abc" --imagesFolder "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/PrepareDenseScene/55222867eebf8b6f90d59776cf8eb16f75822d6e" --downscale 2 --minViewAngle 2.0 --maxViewAngle 70.0 --tileBufferWidth 1024 --tileBufferHeight 1024 --tilePadding 64 --autoAdjustSmallImage True --chooseTCamsPerTile True --maxTCams 10 --sgmScale 2 --sgmStepXY 2 --sgmStepZ -1 --sgmMaxTCamsPerTile 4 --sgmWSH 4 --sgmUseSfmSeeds True --sgmSeedsRangeInflate 0.2 --sgmGammaC 5.5 --sgmGammaP 8.0 --sgmP1 10.0 --sgmP2Weighting 100.0 --sgmMaxDepths 1500 --sgmFilteringAxes "YX" --sgmDepthListPerTile True --refineEnabled True --refineScale 1 --refineStepXY 1 --refineMaxTCamsPerTile 4 --refineSubsampling 10 --refineHalfNbDepths 15 --refineWSH 3 --refineSigma 15.0 --refineGammaC 15.5 --refineGammaP 8.0 --colorOptimizationEnabled True --colorOptimizationNbIterations 100 --exportIntermediateDepthSimMaps False --exportIntermediateVolumes False --exportIntermediateCrossVolumes False --exportIntermediateVolume9pCsv False --exportTilePattern False --nbGPUs 0 --verboseLevel info --output "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/DepthMap/743027f842ea713475880a7e53cb1d67c1db0557" --rangeStart 0 --rangeSize 3
  • logFile: C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/DepthMap/743027f842ea713475880a7e53cb1d67c1db0557/0.log ERROR:root:Error on node computation: Error on node "DepthMap_1(0)": Log: [2023-04-25 14:50:35.831671] [0x00000a4c] [trace] Embedded OCIO configuration file: 'C:\Meshroom-2023.1.0\aliceVision/share/aliceVision/config.ocio' found. Program called with the following parameters:
  • autoAdjustSmallImage = 1
  • chooseTCamsPerTile = 1
  • colorOptimizationEnabled = 1
  • colorOptimizationNbIterations = 100
  • downscale = 2
  • exportIntermediateCrossVolumes = 0
  • exportIntermediateDepthSimMaps = 0
  • exportIntermediateVolume9pCsv = 0
  • exportIntermediateVolumes = 0
  • exportTilePattern = 0
  • imagesFolder = "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/PrepareDenseScene/55222867eebf8b6f90d59776cf8eb16f75822d6e"
  • input = "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/StructureFromMotion/b62e2ada7e73d6fec87d0bc78794398bf3c71234/sfm.abc"
  • maxCoresAvailable = Unknown Type "unsigned int" (default)
  • maxMemoryAvailable = 18446744073709551615 (default)
  • maxTCams = 10
  • maxViewAngle = 70
  • minViewAngle = 2
  • nbGPUs = 0
  • output = "C:/Users/Lenovo/AppData/Local/Temp/MeshroomCache/DepthMap/743027f842ea713475880a7e53cb1d67c1db0557"
  • rangeSize = 3
  • rangeStart = 0
  • refineEnabled = 1
  • refineGammaC = 15.5
  • refineGammaP = 8
  • refineHalfNbDepths = 15
  • refineMaxTCamsPerTile = 4
  • refineScale = 1
  • refineSigma = 15
  • refineStepXY = 1
  • refineSubsampling = 10
  • refineWSH = 3
  • sgmDepthListPerTile = 1
  • sgmFilteringAxes = "YX"
  • sgmGammaC = 5.5
  • sgmGammaP = 8
  • sgmMaxDepths = 1500
  • sgmMaxTCamsPerTile = 4
  • sgmP1 = 10
  • sgmP2Weighting = 100
  • sgmScale = 2
  • sgmSeedsRangeInflate = 0.2
  • sgmStepXY = 2
  • sgmStepZ = -1
  • sgmUseSfmSeeds = 1
  • sgmWSH = 4
  • tileBufferHeight = 1024
  • tileBufferWidth = 1024
  • tilePadding = 64
  • verboseLevel = "info"

Hardware : Detected core count : 8 OpenMP will use 8 cores Detected available memory : 2571 Mo

[14:50:35.845694][info] CUDA-Enabled GPU. Device information:

• id: 0
• name: NVIDIA GeForce MX130
• compute capability: 5.0
• clock frequency (kHz): 1189000
• total device memory: 2047 MB
• device memory available: 1687 MB
• per-block shared memory: 49152
• warp size: 32
• max threads per block: 1024
• max threads per SM(X): 2048
• max block sizes: {1024,1024,64}
• max grid sizes: {2147483647,65535,65535}
• max 2D array texture: {65536,65536}
• max 3D array texture: {4096,4096,4096}
• max 2D linear texture: {65536,65536,1048544}
• max 2D layered texture: {16384,16384,2048}
• number of SM(x)s: 3
• registers per SM(x): 65536
• registers per block: 65536
• concurrent kernels: yes
• mapping host memory: yes
• unified addressing: yes
• texture alignment: 512 byte
• pitch alignment: 32 byte CUDA build version: 11.6 [14:50:36.005074][info] Supported CUDA-Enabled GPU detected. [14:50:36.047005][info] Found 1 image dimension(s): [14:50:36.048007][info] - [4032x3024] [14:50:36.072648][info] Overall maximum dimension: [4032x3024] [14:50:36.072648][info] Create depth maps. [14:50:36.073645][info] Number of GPU devices: 1, number of CPU threads: 8 [14:50:36.073645][info] Tiling information:
• parameters:
  • buffer width: 1024 px
  • buffer height: 1024 px
  • padding: 64 px
• maximum downscale: 4
• maximum image width: 2016 px
• maximum image height: 1512 px
• maximum effective tile width: 896 px
• maximum effective tile height: 896 px

• tiles on X-side: 3

• tiles on Y-side: 2

• effective tile width: 672 px
• effective tile height: 756 px
• tile list:
  • tile (1/6) size: 736x820 px, roi: [x: 0-736, y: 0-820]
  • tile (2/6) size: 736x756 px, roi: [x: 0-736, y: 756-1512]
  • tile (3/6) size: 736x820 px, roi: [x: 672-1408, y: 0-820]
  • tile (4/6) size: 736x756 px, roi: [x: 672-1408, y: 756-1512]
  • tile (5/6) size: 672x820 px, roi: [x: 1344-2016, y: 0-820]
  • tile (6/6) size: 672x756 px, roi: [x: 1344-2016, y: 756-1512] [14:50:36.073645][info] SGM parameters:
• scale: 2
• stepXY: 2 [14:50:36.073645][info] Refine parameters:
• scale: 1
• stepXY: 1 [14:50:36.127916][info] Device memory:
• available: 1349.69 MB
• requirement for the first tile: 856.158 MB

• computation buffers per tile: 472.437 MB (Sgm: 378.437 MB, Refine: 94 MB)

• input images (R + 10 Ts): 383.722 MB (single multi-res image size: 34.8838 MB)

  [14:50:36.127916][info] Parallelization:

• tiles per image: 6

• simultaneous depth maps computation: 1

• streams: 2

  [14:50:36.138483][info] Found only 2/10 nearest cameras for view id: 187293821 [14:50:36.167945][info] Found only 2/10 nearest cameras for view id: 344328980 [14:50:36.201633][info] Found only 2/10 nearest cameras for view id: 1292747793 [14:50:36.263587][info] Device memory (device id: 0):

• used: 1306.76 MB
• available: 741.116 MB
• total: 2047.88 MB [14:50:45.170547][info] (rc: 0, tile: 1/6) Camera / Depth information:
• R camera:
  • id: 0
  • view id: 187293821
  • depth planes: 146
  • depths range: [1.9372-2.91781]
• T cameras:
  • T camera (1/2):
    • id: 2
    • view id: 1292747793
    • depth planes: 146
    • depths range: [1.9372-2.91781]
    • depth indexes range: [0-146]
  • T camera (2/2):
    • id: 1
    • view id: 344328980
    • depth planes: 146
    • depths range: [1.9372-2.91781]
    • depth indexes range: [0-146] [14:50:45.170547][info] (rc: 0, tile: 1/6) SGM depth/sim map of view id: 187293821, rc: 0 (1 / 3). [14:50:45.170547][info] (rc: 0, tile: 1/6) SGM Compute similarity volume. [14:50:45.170547][info] (rc: 0, tile: 1/6) SGM Compute similarity volume done. [14:50:45.170547][info] (rc: 0, tile: 1/6) SGM Optimizing volume (filtering axes: YX). [14:50:45.194354][info] (rc: 0, tile: 1/6) SGM Optimizing volume done. [14:50:45.194354][info] (rc: 0, tile: 1/6) SGM Retrieve best depth in volume. [14:50:45.194354][info] (rc: 0, tile: 1/6) SGM Retrieve best depth in volume done. [14:50:45.194354][info] (rc: 0, tile: 1/6) SGM depth/sim map done. [14:50:45.194354][info] (rc: 0, tile: 1/6) Refine depth/sim map of view id: 187293821, rc: 0 (1 / 3). [14:50:45.195353][info] (rc: 0, tile: 1/6) Refine and fuse depth/sim map volume. [14:50:45.195353][info] (rc: 0, tile: 1/6) Refine and fuse depth/sim map volume done. [14:50:45.195353][info] (rc: 0, tile: 1/6) Color optimize depth/sim map. [14:50:45.197347][info] (rc: 0, tile: 1/6) Color optimize depth/sim map done. [14:50:45.197347][info] (rc: 0, tile: 1/6) Refine depth/sim map done. [14:50:45.201857][info] (rc: 0, tile: 2/6) Camera / Depth information:
• R camera:
  • id: 0
  • view id: 187293821
  • depth planes: 89
  • depths range: [1.80577-2.42327]
• T cameras:
  • T camera (1/2):
    • id: 2
    • view id: 1292747793
    • depth planes: 89
    • depths range: [1.80577-2.42327]
    • depth indexes range: [0-89]
  • T camera (2/2):
    • id: 1
    • view id: 344328980
    • depth planes: 89
    • depths range: [1.80577-2.42327]
    • depth indexes range: [0-89] [14:50:45.201857][info] (rc: 0, tile: 2/6) SGM depth/sim map of view id: 187293821, rc: 0 (1 / 3). [14:50:45.201857][info] (rc: 0, tile: 2/6) SGM Compute similarity volume. [14:50:45.201857][info] (rc: 0, tile: 2/6) SGM Compute similarity volume done. [14:50:45.201857][info] (rc: 0, tile: 2/6) SGM Optimizing volume (filtering axes: YX). [14:50:45.939601][info] (rc: 0, tile: 2/6) SGM Optimizing volume done. [14:50:45.939601][info] (rc: 0, tile: 2/6) SGM Retrieve best depth in volume. [14:50:45.939601][info] (rc: 0, tile: 2/6) SGM Retrieve best depth in volume done. [14:50:45.939601][info] (rc: 0, tile: 2/6) SGM depth/sim map done. [14:50:45.939601][info] (rc: 0, tile: 2/6) Refine depth/sim map of view id: 187293821, rc: 0 (1 / 3). [14:50:45.939601][info] (rc: 0, tile: 2/6) Refine and fuse depth/sim map volume. [14:50:45.940603][info] (rc: 0, tile: 2/6) Refine and fuse depth/sim map volume done. [14:50:45.940603][info] (rc: 0, tile: 2/6) Color optimize depth/sim map. [14:50:45.955728][info] (rc: 0, tile: 2/6) Color optimize depth/sim map done. [14:50:45.955728][info] (rc: 0, tile: 2/6) Refine depth/sim map done. [14:50:45.958728][info] (rc: 0, tile: 3/6) Camera / Depth information:
• R camera:
  • id: 0
  • view id: 187293821
  • depth planes: 940
  • depths range: [0.644274-8.68923]
• T cameras:
  • T camera (1/2):
    • id: 2
    • view id: 1292747793
    • depth planes: 940
    • depths range: [0.644274-8.68923]
    • depth indexes range: [0-940]
  • T camera (2/2):
    • id: 1
    • view id: 344328980
    • depth planes: 758
    • depths range: [0.861219-6.66463]
    • depth indexes range: [168-926] [14:50:45.958728][info] (rc: 0, tile: 3/6) SGM depth/sim map of view id: 187293821, rc: 0 (1 / 3). [14:50:45.958728][info] (rc: 0, tile: 3/6) SGM Compute similarity volume. [14:50:45.958728][info] (rc: 0, tile: 3/6) SGM Compute similarity volume done. [14:50:45.958728][info] (rc: 0, tile: 3/6) SGM Optimizing volume (filtering axes: YX).

CUDAError: the launch timed out and was terminated file: C:\Release\sources\AliceVision\src\aliceVision\depthMap\cuda\planeSweeping\deviceSimilarityVolume.cu function: aliceVision::depthMap::cuda_volumeAggregatePath line: 282

WARNING:root:Downgrade status on node "DepthMapFilter_1(0)" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "Meshing_1" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "Texturing_1" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "MeshFiltering_1" from Status.SUBMITTED to Status.NONE

[PluggPreagar]

Hi, it seems related https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/2144 - so I added Batch file with SET CUDA_LAUNCH_BLOCKING=1 before starting msehroom.exe . This worked for me.

[okin89]

Hi, it seems related AUTOMATIC1111/stable-diffusion-webui#2144 - so I added Batch file with SET CUDA_LAUNCH_BLOCKING=1 before starting msehroom.exe . This worked for me.

Mind telling me where exactly i can find the .bat file to add thoses commands on windows? I can't find the file at all, even searched within all the folders in the meshroom. Thanks

[jiye33]

Did you solve this problem? Using meshroom 2018 works for me without changing anything, but meshroom 2023 still have same problem. I also use windows. If you find how to fix it, please let me know!

[alsper78]

I had this problem on Windows 10 with RAM 8 GB and NVIDIA GeForce MX130. In my case it seems to be related to the available memory. I changed these parameters in DepthMap:

• Buffer Width from 1024 to 256,
• Buffer Height from 1024 to 256,
• Padding from 64 to 32,

and now it works.