search

naver / mast3r

Grounding Image Matching in 3D with MASt3R
Other
1.3k stars 98 forks source link

CUDA error in custom dataset #17

Closed hwanhuh closed 3 months ago

hwanhuh commented 3 months ago

Description:

Firstly, I would like to express my appreciation for the outstanding work on this project. The features and optimizations provided have been valuable and demonstrate exceptional skill and dedication.

I encountered a RuntimeError related to CUDA when running the condense_data function on certain datasets. The error is as follows:

RuntimeError: CUDA error: device-side assert triggered

Details:

The error occurs during the subsampling of preds_21:

# Subsample preds_21
subsamp_preds_21 = {}
for imk, imv in preds_21.items():
    subsamp_preds_21[imk] = {}
    for im2k, (pred, conf) in preds_21[imk].items():
        idxs = img_anchors[imgs.index(im2k)][1]
        subsamp_preds_21[imk][im2k] = (pred[idxs], conf[idxs])

Upon investigation, I observed that for some datasets, pred.shape is torch.Size([1248, 3]), while its typical shape is 2304. The idxs values exceed the bounds of pred, leading to the device-side assertion error due to the invalid index.

hwanhuh commented 3 months ago

Sorry... It appears that the issue was caused by my oversight of not clearing the cache directory before running the program.