I cannot run the binary in the build directory.

[liuhao230]

I used the following command to build the ICICLE library using nvcc.

nvcc -o build/bn254 ./icicle/curves/index.cu -lib -arch=native

I cannot run the binary without Nsight Compute.

I want to profile the binary file bn254 by ncu(NVIDIA Nsight Compute).

ncu build/bn254

But I get an error as follows:

==ERROR== The application returned an error code (1).
==WARNING== No kernels were profiled.
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option.

Does anyone know what might be happening? Any advice would be appreciated.

[ImmanuelSegol]

Not sure why your getting ==ERROR== The application returned an error code (1). ill run this locally to check.

the warning says that ncu wasnt able to find any CUDA kernels running, Does ncu build/bn254 --target-processes work (this also could happen due to the error) ?

What version of nvcc are you running (nvcc --version) ?

[vhnatyk]

Hi, output of the command you mentioned is a library, not an executable. You would need to launch some executable(that loads and calls the functions) to profile the library. Try to profile output of cargo build --example --release - then it will output the binary to path like icicle/target/release/icicle-utils. Then you can profile it with ncu.

[liuhao230]

Hi, output of the command you mentioned is a library, not an executable. You would need to launch some executable(that loads and calls the functions) to profile the library. Try to profile output of cargo build --example --release - then it will output the binary to path like icicle/target/release/icicle-utils. Then you can profile it with ncu.

Thanks vhnatyk. As you suggested, I have run cargo build --example ntt --release-then it outputs the executable ntt to path icicle/target/release/examples/. Then I run the executable ntt successfully. But when I want to profile it with ncu (sudo /usr/local/cuda-11.1/bin/ncu target/release/examples/ntt), I get the following error:

NTT of size 2^20 in batch 1
==PROF== Connected to process 81768 (/home/liu/icicle/target/release/examples/ntt)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidContext', src/test_bls12_381.rs:859:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'Failed to deallocate CUDA Device memory.: InvalidValue', /home/liu/.cargo/registry/src/github.com-1ecc6299db9ec823/rustacuda-0.1.3/src/memory/device/device_buffer.rs:258:32
stack backtrace:
   0:     0x55ad60f9e5cf - std::backtrace_rs::backtrace::libunwind::trace::hf4179216f2af7039
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55ad60f9e5cf - std::backtrace_rs::backtrace::trace_unsynchronized::hfa14eb12e4b8ce4f
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55ad60f9e5cf - std::sys_common::backtrace::_print_fmt::hc81cbee2043ec868
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55ad60f9e5cf - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7b2be2009a6bcba2
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55ad60fb72fe - core::fmt::write::hccee7882499b620f
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/core/src/fmt/mod.rs:1209:17
   5:     0x55ad60f88125 - std::io::Write::write_fmt::h69e1f37bb24a9f49
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/io/mod.rs:1682:15
   6:     0x55ad60f9e385 - std::sys_common::backtrace::_print::h86df92192f156e4c
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x55ad60f9e385 - std::sys_common::backtrace::print::h3b76d5113eea7ae3
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x55ad60f9041f - std::panicking::default_hook::{{closure}}::h634bc187e8e58de8
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:267:22
   9:     0x55ad60f900d8 - std::panicking::default_hook::h262aef5e4abfc826
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:286:9
  10:     0x55ad60f90a2b - std::panicking::rust_panic_with_hook::hedb8a8e4067bf251
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:688:13
  11:     0x55ad60f9e927 - std::panicking::begin_panic_handler::{{closure}}::hf501de419c9dcde3
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:579:13
  12:     0x55ad60f9e71c - std::sys_common::backtrace::__rust_end_short_backtrace::h8542e72ad556a307
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/sys_common/backtrace.rs:137:18
  13:     0x55ad60f905d2 - rust_begin_unwind
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:575:5
  14:     0x55ad60f66f93 - core::panicking::panic_fmt::h1688e859668ba1d0
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/core/src/panicking.rs:65:14
  15:     0x55ad60f67213 - core::result::unwrap_failed::h9f204f08b8ff9323
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/core/src/result.rs:1791:5
  16:     0x55ad60f6a287 - core::ptr::drop_in_place<rustacuda::memory::device::device_buffer::DeviceBuffer<icicle::curves::bls12_381::Point_BLS12_381>>::h1340780c1bbea58a
  17:     0x55ad60f6afc4 - icicle::test_bls12_381::set_up_scalars_bls12_381::ha2797d02a9cd03ac
  18:     0x55ad60f699f5 - ntt::bench_ntt_template::haf5fda7ee6f36c15
  19:     0x55ad60f69ec5 - ntt::main::h1c1398bbe51db8c3
  20:     0x55ad60f68cd3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf58c10acbde8798a
  21:     0x55ad60f68ca9 - std::rt::lang_start::{{closure}}::h1e39a0fd1de5e0b3
  22:     0x55ad60f873a4 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h37feed43940ccfab
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/core/src/ops/function.rs:286:13
  23:     0x55ad60f873a4 - std::panicking::try::do_call::h8fe1ed7bdb58d0f5
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:483:40
  24:     0x55ad60f873a4 - std::panicking::try::h87c3f91ef362d0ca
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:447:19
  25:     0x55ad60f873a4 - std::panic::catch_unwind::hd6a68686ac8f53ed
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panic.rs:137:14
  26:     0x55ad60f873a4 - std::rt::lang_start_internal::{{closure}}::hdc22961783632ef0
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/rt.rs:148:48
  27:     0x55ad60f873a4 - std::panicking::try::do_call::h92fd045c6d8aed83
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:483:40
  28:     0x55ad60f873a4 - std::panicking::try::haa1526bc83feb2c9
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panicking.rs:447:19
  29:     0x55ad60f873a4 - std::panic::catch_unwind::hb3d0e1586c57c804
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/panic.rs:137:14
  30:     0x55ad60f873a4 - std::rt::lang_start_internal::hed890805d128cbd3
                               at /build/rustc-v6rcRM/rustc-1.66.1+dfsg0ubuntu1~llvm/library/std/src/rt.rs:148:20
  31:     0x55ad60f6a195 - main
  32:     0x7f8715f36083 - __libc_start_main
                               at /build/glibc-BHL3KM/glibc-2.31/csu/../csu/libc-start.c:308:16
  33:     0x55ad60f68bbe - _start
  34:                0x0 - <unknown>
thread panicked while panicking. aborting.
==ERROR== The application returned an error code (6).
==WARNING== No kernels were profiled.
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option.

sudo /usr/local/cuda-11.1/bin/ncu target/release/examples/ntt --target-processes gets the same error.

The output of nvcc --version is

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

The reason why I use ncu under directory /usr/local/cuda-11.1/bin/ is that my GPU is 1080ti and Pascal support was dropped from Nsight Compute after Nsight Compute 2019.5.1.

[vhnatyk]

umm, the error you got - it's a known issue with Rust ntt_inplace wrapper on some systems, anyway - you can come up with your own example calling some kernel of your interest and profile the executable binary then with ncu 👍🏻

[liuhao230]

Thanks, vhnatyk. I was able to successfully profile the executable on a different machine using a 3090 graphics card.

[ImmanuelSegol]

@sss1h I hope everything is working now, I will be closing this issue as it seems it has been resolved.