Error when running comfyUI using the official demo workflow

[luochenxi]

Run the official example error

Environment

• nvidia/cuda:12.2.2-cudnn8-devel-ubuntu20.04
• python3.10.13
• NVIDIA GeForce RTX 3090

OneDiff git commit id:5677af571fdd7e601ea84b0ee9700d54147a83ba

OneFlow version info

libibverbs not available, ibv_fork_init skipped
path: ['/app/venv/lib/python3.10/site-packages/oneflow']
version: 0.9.1.dev20240612+cu121
git_commit: ec7b682
cmake_build_type: Release
rdma: True
mlir: True
enterprise: False

How To Reproduce

1. Setup Community Edition Install OneFlow Community(CUDA 12.x)

   pip install --pre oneflow -f https://oneflow-pro.oss-cn-beijing.aliyuncs.com/branch/community/cu121

   Install OneDiff

   git clone https://github.com/siliconflow/onediff.git
   cd onediff && pip install -e .

   Install onediff_comfy_nodes for ComfyUI

   cp -r onediff_comfy_nodes path/to/ComfyUI/custom_nodes

2. Drag and drop the Sample picture into comfyui and run it.

The complete error message

model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
Requested to load BaseModel
Loading 1 new model
  0%|          | 0/20 [00:00<?, ?it/s]F20240613 18:08:20.498903   336 cublas_fused_mlp_util.cuh:297] Check failed: cublasLtMatmulAlgoGetHeuristic( cuda_stream->cublas_lt_handle(), matmul_grad_cache->operation_desc, matmul_grad_cache->cublas_a_desc, matmul_grad_cache->cublas_b_desc, matmul_grad_cache->cublas_c_desc, matmul_grad_cache->cublas_c_desc, matmul_grad_cache->cublas_preference, 1, &algo_heuristic_result, &algo_returned_result) : CUBLAS_STATUS_INVALID_VALUE (7)
*** Check failure stack trace: ***
    @     0x7fd423b6296a  google::LogMessage::Fail()
    @     0x7fd423b658a1  google::LogMessage::SendToLog()
    @     0x7fd423b62499  google::LogMessage::Flush()
    @     0x7fd423b66189  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fd41d267393  oneflow::(anonymous namespace)::SetCublasAttr()
    @     0x7fd41d26971f  oneflow::(anonymous namespace)::FusedMatmulBiasKernel::Compute()
    @     0x7fd41f30ec75  oneflow::one::StatefulOpKernel::Compute()
    @     0x7fd41bd3c69a  _ZZN7oneflow2vm21OpCallInstructionUtil7ComputeEPNS0_23OpCallInstructionPolicyEPNS0_6StreamEbbENKUlvE_clEv
    @     0x7fd41bd3dfc8  oneflow::vm::OpCallInstructionUtil::Compute()
    @     0x7fd41bd3abbd  _ZZN7oneflow2vm23OpCallInstructionPolicy7ComputeEPNS0_11InstructionEENKUlPKcE_clES5_.constprop.0
    @     0x7fd41bd3b419  oneflow::vm::OpCallInstructionPolicy::Compute()
    @     0x7fd41bd34038  oneflow::vm::Instruction::Compute()
    @     0x7fd41bd30d35  oneflow::vm::EpStreamPolicyBase::Run()
    @     0x7fd41bd8e599  oneflow::vm::ThreadCtx::TryReceiveAndRun()
    @     0x7fd41bd92d1d  oneflow::(anonymous namespace)::WorkerLoop()
    @     0x7fd41bd934a8  _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvPN7oneflow2vm9ThreadCtxERKSt8functionIFvS6_EEES6_ZNS3_14VirtualMachine15CreateThreadCtxENS3_6SymbolINS3_6DeviceEEENS3_10StreamTypeEmEUlS6_E3_EEEEE6_M_runEv
    @     0x7fd7a9357df4  (unknown)
    @     0x7fd7d4ae9609  start_thread
    @     0x7fd7d48b4133  clone
Stack trace (most recent call last) in thread 336:
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd934a7, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd92d1c, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd8e598, in vm::ThreadCtx::TryReceiveAndRun()
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd30d34, in vm::EpStreamPolicyBase::Run(vm::Instruction*) const
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd34037, in vm::Instruction::Compute()
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd3b418, in vm::OpCallInstructionPolicy::Compute(vm::Instruction*)
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd3abbc, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd3dfc7, in vm::OpCallInstructionUtil::Compute(vm::OpCallInstructionPolicy*, vm::Stream*, bool, bool)
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41bd3c699, in vm::OpCallInstructionUtil::Compute(vm::OpCallInstructionPolicy*, vm::Stream*, bool, bool)::{lambda()#1}::operator()() const
watf? exit
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41f30ec74, in StatefulOpKernel::Compute(eager::CallContext*, ep::Stream*, user_op::OpKernel const*, user_op::OpKernelState*, user_op::OpKernelCache const*) const
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41d26971e, in (anonymous namespace)::FusedMatmulBiasKernel::Compute(user_op::KernelComputeContext*, user_op::OpKernelState*, user_op::OpKernelCache const*) const
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd41d267392, in (anonymous namespace)::SetCublasAttr(ep::CudaStream*, (anonymous namespace)::CublasFusedMLPKernelCache const*, cublasComputeType_t, cudaDataType_t, bool, ep::primitive::BlasTransposeType, ep::primitive::BlasTransposeType, cublasLtEpilogue_t, void const*, void const*, unsigned long, unsigned long, unsigned long, long, long, long)
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd423b66188, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd423b62498, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd423b658a0, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd423b62969, in
   Object "/app/venv/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-6ebf2b05.so", at 0x7fd413d7b44e, in

Aborted (Signal sent by tkill() 1 0)