search

cpnr / computing

0 stars 0 forks source link

lapras GPU 상태 이상 #57

Open jhgoh opened 1 month ago

jhgoh commented 1 month ago

lapras 사용중 GPU상태 이상징후가 있음.

  File ".....pyg24/lib/python3.11/site-packages/torch/optim/adam.py", line 551, in _multi_tensor_adam
    bias_correction1 = [1 - beta1  _get_value(step) for step in device_state_steps]
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "....pyg24/lib/python3.11/site-packages/torch/optim/adam.py", line 551, in <listcomp>
    bias_correction1 = [1 - beta1  _get_value(step) for step in device_state_steps]
                                     ^^^^^^^^^^^^^^^^
  File ".....pyg24/lib/python3.11/site-packages/torch/optim/optimizer.py", line 89, in _get_value
    return x.item()
           ^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

@ChangHyunYoo92