ArthurBrussee / brush

3D Reconstruction for all
Apache License 2.0
1.04k stars 26 forks source link

[Ubuntu 24.04 Crash] Cmma on inputs Float(F16) and outputs Float(F32) with shape m=16, n=16, k=16 not supported. #53

Open pablovela5620 opened 1 day ago

pablovela5620 commented 1 day ago

I have a 3060 I'm using, getting the following error when I try to train

RUST_BACKTRACE=1 cargo run --release
    Finished `release` profile [optimized] target(s) in 0.70s
     Running `target/release/brush_bin`
thread 'tokio-runtime-worker' panicked at /home/pablo/.cargo/git/checkouts/cubecl-aa41a28b39b598f9/a4e2b77/crates/cubecl-linalg/src/matmul/base.rs:51:14:
Accelerated strategy should be available on your device: Unable to launch matmul because a required feature is unavailable: Cmma on inputs Float(F16) and outputs Float(F32) with shape m=16, n=16, k=16 not supported.

stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: cubecl_linalg::matmul::base::launch_ref
   4: <burn_jit::kernel::matmul::tune::base::MatmulAccelerated<R,E> as cubecl_runtime::tune::operation::AutotuneOperation>::execute
   5: futures_lite::future::block_on
   6: cubecl_runtime::tune::local::LocalTuner<AK,ID>::execute
   7: burn_jit::kernel::matmul::tune::base::matmul_autotune
   8: burn_jit::ops::float_ops::<impl burn_tensor::tensor::ops::tensor::FloatTensorOps<burn_jit::backend::JitBackend<R,F,I,BT>> for burn_jit::backend::JitBackend<R,F,I,BT>>::float_matmul
   9: <burn_fusion::ops::float::<impl burn_tensor::tensor::ops::tensor::FloatTensorOps<burn_fusion::backend::Fusion<B>> for burn_fusion::backend::Fusion<B>>::float_matmul::MatmulOps<B> as burn_fusion::stream::execution::base::Operation<<B as burn_fusion::backend::FusionBackend>::FusionRuntime>>::execute
  10: burn_fusion::stream::execution::base::<impl burn_fusion::stream::base::OperationQueue<R>>::execute
  11: burn_fusion::stream::execution::processor::Processor<O>::process
  12: burn_fusion::stream::multi::MultiStream<R>::register
  13: burn_fusion::ops::float::<impl burn_tensor::tensor::ops::tensor::FloatTensorOps<burn_fusion::backend::Fusion<B>> for burn_fusion::backend::Fusion<B>>::float_matmul
  14: burn_autodiff::ops::tensor::<impl burn_tensor::tensor::ops::tensor::FloatTensorOps<burn_autodiff::backend::Autodiff<B,C>> for burn_autodiff::backend::Autodiff<B,C>>::float_matmul
  15: burn_tensor::tensor::api::float::<impl burn_tensor::tensor::api::base::Tensor<B,_>>::matmul
  16: brush_train::ssim::Ssim<B>::new
  17: brush_viewer::train_loop::train_loop::{{closure}}::{{closure}}
  18: <async_fn_stream::TryFnStream<T,E,Fut> as futures_core::stream::Stream>::poll_next
  19: <core::pin::Pin<P> as core::future::future::Future>::poll
  20: <async_fn_stream::TryFnStream<T,E,Fut> as futures_core::stream::Stream>::poll_next
  21: brush_viewer::viewer::ViewerContext::start_data_load::{{closure}}
  22: tokio::runtime::task::raw::poll
  23: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  24: tokio::runtime::scheduler::multi_thread::worker::Context::run
  25: tokio::runtime::context::scoped::Scoped<T>::set
  26: tokio::runtime::context::runtime::enter_runtime
  27: tokio::runtime::scheduler::multi_thread::worker::run
  28: tokio::runtime::task::core::Core<T,S>::poll
  29: tokio::runtime::task::harness::Harness<T,S>::poll
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aborted (core dumped)
ArthurBrussee commented 1 day ago

Sounds like something is wrong in the Burn upgrade, trying to see what's wrong.