使用多gpu启动worker，对话时报错

[kimi360]

启动命令

CUDA_VISIBLE_DEVICES=0,1 python -m mgm.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path work_dirs/MGM/MGM-7B

进行对话时报错

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`

cuda版本12.3 完整日志： model_worker_eee903.log

有人遇到过这样的问题吗？

[kimi360]

执行 export CUDA_LAUNCH_BLOCKING=1后报错变了

2024-05-30 10:19:00 | ERROR | stderr | RuntimeError: CUDA error: device-side assert triggered
2024-05-30 10:19:00 | ERROR | stderr | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

完整日志： model_worker_8dffd9.log