search

prs-eth / Marigold

[CVPR 2024 - Oral, Best Paper Award Candidate] Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
https://marigoldmonodepth.github.io
Apache License 2.0
2.39k stars 132 forks source link

Wandb logging is always interrupted unexpectedly #115

Closed water221 closed 2 months ago

water221 commented 2 months ago

When I was training the model, I found that 'wandb' always interrupted abnormally, attached my log file below, can someone help me answer it
The first figure is the visual display of 'wandb', the second figure is the running log of 'wandb', and the third figure is the training log of the project。 This training in the figure was abnormally interrupted at step 932, and the visualized loss was no longer updated.
image image image

markkua commented 2 months ago

Hi, I haven't experienced this before. I would upgrade the packages and check the network. Otherwise, I suggest asking in wandb github.