Closed MaisenbacherD closed 2 years ago
Abhinandan-Purkait
commented
2 years ago Hi @MaisenbacherD there were some naming changes on this front but the docs were probably not updated. Can you please try with nix develop -f . io-engine.
You can also do nix-shell in the root, and that should just bring up the environment for you.
MaisenbacherD
commented
2 years ago Hi @Abhinandan-Purkait :)
Thank you, nix develop -f . io-engine helped!
I had to manually do export SRCDIR=$(pwd) before I was able to successfully run cargo build.
Not sure how nix works, but the shell.nix should have done that?
Furthermore, I had to build spdk on my own in order to run cargo test -- --test-threads 1. Is that correct? I thought that spdk would be provided by nix as long as the norust argument is not set to true when accessing the nix-shell.
For completeness this is how I build spdk (in my regular shell):
#pwd = mayastor root directory
cd spdk-rs
git clone https://github.com/spdk/spdk.git
cd spdk
git checkout v22.01
git submodule update --init --recursive
sudo ./scripts/pkgdep.sh
cd ..
#The next step required me to point `--with-fio` to the location of the from source build fio such that `config-host.h` would be found.
./build_spdk.sh
#spdk-rs needs to be build from its own directory (and not through `cargo test -- --test-threads 1` in the parent directory) because otherwise `get_target_dir` from https://github.com/mayadata-io/spdk-rs/blob/develop/build.rs does not set the correct path to search the `libspdk-bundle.so` file.
cargo build
cd ..Now I was able to run cargo test -- --test-threads 1, which fails for now on the add_child test case:
running 1 test
test add_child ... [2022-07-14T15:15:28.302775016+02:00 INFO io_engine::subsys::config:mod.rs:216] Applying Mayastor configuration settings
[2022-07-14T15:15:28.303063857+02:00 DEBUG io_engine::subsys::config::opts:opts.rs:261] spdk_bdev_nvme_opts { action_on_timeout: 4, timeout_us: 5000000, timeout_admin_us: 5000000, keep_alive_timeout_ms: 1000, transport_retry_count: 0, arbitration_burst: 0, low_priority_weight: 0, medium_priority_weight: 0, high_priority_weight: 0, nvme_adminq_poll_period_us: 1000, nvme_ioq_poll_period_us: 0, io_queue_requests: 0, delay_cmd_submit: true, bdev_retry_count: 0 }
[2022-07-14T15:15:28.303168082+02:00 DEBUG io_engine::subsys::config:mod.rs:220] Config {
source: None,
nvmf_tcp_tgt_conf: NvmfTgtConfig {
name: "mayastor_target",
max_namespaces: 2048,
opts: NvmfTcpTransportOpts {
max_queue_depth: 32,
max_qpairs_per_ctrl: 32,
in_capsule_data_size: 4096,
max_io_size: 131072,
io_unit_size: 131072,
max_aq_depth: 32,
num_shared_buf: 2048,
buf_cache_size: 64,
dif_insert_or_strip: false,
abort_timeout_sec: 1,
acceptor_poll_rate: 10000,
zcopy: true,
},
},
nvme_bdev_opts: NvmeBdevOpts {
action_on_timeout: 4,
timeout_us: 5000000,
timeout_admin_us: 5000000,
keep_alive_timeout_ms: 1000,
transport_retry_count: 0,
arbitration_burst: 0,
low_priority_weight: 0,
medium_priority_weight: 0,
high_priority_weight: 0,
nvme_adminq_poll_period_us: 1000,
nvme_ioq_poll_period_us: 0,
io_queue_requests: 0,
delay_cmd_submit: true,
bdev_retry_count: 0,
async_mode: false,
},
bdev_opts: BdevOpts {
bdev_io_pool_size: 65535,
bdev_io_cache_size: 512,
small_buf_pool_size: 8191,
large_buf_pool_size: 1023,
},
nexus_opts: NexusOpts {
nvmf_enable: true,
nvmf_discovery_enable: true,
nvmf_nexus_port: 4421,
nvmf_replica_port: 8420,
},
}
[2022-07-14T15:15:28.303440273+02:00 DEBUG io_engine::core::env:env.rs:547] EAL arguments ["mayastor", "--no-shconf", "-m 0", "--no-pci", "--base-virtaddr=0x200000000000", "--file-prefix=mayastor_pid4122777", "--huge-unlink", "--log-level=lib.eal:6", "--log-level=lib.cryptodev:5", "--log-level=user1:6", "--match-allocations", "-c 0x1"]
EAL: No available 1048576 kB hugepages reported
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
FAILED
failures:
---- add_child stdout ----
thread 'mayastor_master' panicked at 'Failed to init EAL', io-engine/src/core/env.rs:556:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', io-engine/tests/common/compose.rs:83:33
failures:
add_child
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
error: test failed, to rerun pass '-p io-engine --test add_child'As suggested I configured 1024 2MiB huge pages (in two different ways):
# Option 1:
echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# Option 2:
cd spdk-rs/spdk/dpdk/usertools
sudo ./dpdk-hugepages.py -p 2048K --setup 2097152K Is it correct that also 1048576 kB huge pages are required or am I doing something wrong?
Abhinandan-Purkait
commented
2 years ago You would not need to build spdk. You should get it built on entering nix-shell. You would just need to run nix-shell from the root dir and that should just do everything for you. The option 1 for setting HugePages is right. The number of HugePages needed depends on the number of io-engine containers you want to run, each needing 1024.
MaisenbacherD
commented
2 years ago When I run nix-shell spdk seems to be build. However, when I run cargo build I am getting the following error:
Compiling err-derive v0.2.4
error: `err-derive` depends on `proc-macro-error`, which requires rustc >= 1.34
--> /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/err-derive-0.2.4/build.rs:2:1
|
2 | compile_error!("`err-derive` depends on `proc-macro-error`, which requires rustc >= 1.34");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/dennis/src/mayastor/target/debug/deps/librustversion-f83f8ace0cac2a45.so)
--> /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/err-derive-0.2.4/build.rs:1:3
|
1 | #[rustversion::before(1.34)]
| ^^^^^^^^^^^
error: cannot determine resolution for the attribute macro `rustversion::before`
--> /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/err-derive-0.2.4/build.rs:1:3
|
1 | #[rustversion::before(1.34)]
| ^^^^^^^^^^^^^^^^^^^
|
= note: import resolution is stuck, try simplifying macro imports
error: could not compile `err-derive` due to 3 previous errors
warning: build failed, waiting for other jobs to finish...It looks like GLIBC_2.32 is not installed within the nix-shell
Abhinandan-Purkait
commented
2 years ago Can you check whether your rust version matches the one being used.
https://github.com/openebs/mayastor/blob/be4c196f24f696d502ed281fb605290e0743e6ba/nix/lib/rust.nix#L7
You can check that by rustup toolchain list
Install that version by rustup toolchain install nightly-2021-11-30
Set it as default by rustup default nightly-2021-11-30-x86_64-unknown-linux-gnu
After that probably you can exit and enter nix-shell and do a cargo clean.
MaisenbacherD
commented
2 years ago The nightly rust toolchain was indeed not enabled within the nix shell. I followed your instructions and got the following output when executing rustup toolchain list:
stable-x86_64-unknown-linux-gnu
nightly-2021-11-30-x86_64-unknown-linux-gnu (default)For rustc --version:
rustc 1.59.0-nightly (6db0a0e9a 2021-11-29)After exiting and entering the nix-shell and running cargo clean followed by cargo build the error persists:
error: `err-derive` depends on `proc-macro-error`, which requires rustc >= 1.34
--> /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/err-derive-0.2.4/build.rs:2:1
|
2 | compile_error!("`err-derive` depends on `proc-macro-error`, which requires rustc >= 1.34");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/dennis/src/mayastor/target/debug/deps/librustversion-e67a322b869b9b50.so)
--> /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/err-derive-0.2.4/build.rs:1:3
|
1 | #[rustversion::before(1.34)]
| ^^^^^^^^^^^
error: could not compile `err-derive` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed
tiagolobocastro
commented
2 years ago Have you tried "nix-shell ci.nix"?
MaisenbacherD
commented
2 years ago When I enter the nix shell with nix-shell ci.nix I can cargo build without any error.
However, if I am running cargo test -- --test-threads 1 I am getting the following compile error:
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.227zxd363hp6uelq.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.2kvr9klybxfa46no.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.3f1homfasr5oo61s.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.3puuuoziaen8bala.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.4stpvupew178h4q0.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.562gv39nqdx3beyw.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.5brjhirzzrecyei4.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.5g8fzc4w73fkiw1q.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.bnfcv25i3rfx0xd.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.xatxbpwv51ic558.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.xkal4l8w7fguft5.rcgu.o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915.6ggqp02vkbkgdmw.rcgu.o" "-Wl,--as-needed" "-L" "/home/dennis/src/mayastor/target/debug/deps" "-L" "/home/dennis/src/mayastor/target/debug/build/spdk-rs-d69df47bd3068b02/out" "-L" "/nix/store/dmd8i924mcm17nclazfmplyfrl8r67bv-rust-default-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-lspdk-bundle" "-Wl,-Bstatic" "-Wl,--whole-archive" "-lhelpers" "-Wl,--no-whole-archive" "/home/dennis/src/mayastor/target/debug/deps/libuuid-0b0ef25926be8e52.rlib" "/home/dennis/src/mayastor/target/debug/deps/libgetrandom-d7a43268b84fee5d.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-7ddc9ce4941507e1.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-964c30d87a838365.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunicode_width-915c2cbb4b7aa7c9.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_std-1bdb232450d24449.rlib" "/home/dennis/src/mayastor/target/debug/deps/libsnafu-4648e562ae6d7d9d.rlib" "/home/dennis/src/mayastor/target/debug/deps/libdoc_comment-5fc477848092951f.rlib" "/home/dennis/src/mayastor/target/debug/deps/libnix-6db103b0a8e41cb6.rlib" "/home/dennis/src/mayastor/target/debug/deps/libbitflags-69211c01e1d2aa85.rlib" "/home/dennis/src/mayastor/target/debug/deps/libmemoffset-74319917765de537.rlib" "/home/dennis/src/mayastor/target/debug/deps/liblibc-681c1f60245940b3.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures-580f1152896027f3.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_executor-221b3029a970ecf5.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_util-e9c711ec6baff525.rlib" "/home/dennis/src/mayastor/target/debug/deps/libmemchr-2b8f7268d78fa7bd.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_io-cbabb44616baa9a6.rlib" "/home/dennis/src/mayastor/target/debug/deps/libslab-14f7fadda46aa470.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_channel-e69d4ac0e84ae734.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_sink-a40df182792851fb.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_task-e4f868f8b9072efd.rlib" "/home/dennis/src/mayastor/target/debug/deps/libpin_utils-a33c58baef8f8b55.rlib" "/home/dennis/src/mayastor/target/debug/deps/libfutures_core-e9cfa78b4873651e.rlib" "/home/dennis/src/mayastor/target/debug/deps/libserde_json-09263a3775d0a616.rlib" "/home/dennis/src/mayastor/target/debug/deps/libryu-25660ecbe64d8c1e.rlib" "/home/dennis/src/mayastor/target/debug/deps/libitoa-21dbbd85f89181ca.rlib" "/home/dennis/src/mayastor/target/debug/deps/libserde-48edbcfab4c32050.rlib" "/home/dennis/src/mayastor/target/debug/deps/libtracing-92db53aea8eff841.rlib" "/home/dennis/src/mayastor/target/debug/deps/libpin_project_lite-0108d58e3f46fd51.rlib" "/home/dennis/src/mayastor/target/debug/deps/libtracing_core-9c9130c715fa5e8b.rlib" "/home/dennis/src/mayastor/target/debug/deps/liblazy_static-972302aead5ba927.rlib" "/home/dennis/src/mayastor/target/debug/deps/liblog-a9441c04002a1c67.rlib" "/home/dennis/src/mayastor/target/debug/deps/libcfg_if-0c646b04edbea3dc.rlib" "-Wl,--start-group" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-65a28bf1738424c0.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-ba99c63b775f734c.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-2c795be2d1f43514.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-4225fa5bc39268de.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-a6e9f64eae6e45dd.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-b1c963add6e8b853.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-541ca82647a7fb7c.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-b93187d0f71da706.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-12352e7b2d1a318e.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-3787a903f1bee791.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-fdd9bd7e7b7760b0.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-4274f5e0121314fe.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-496757222b2a1fa7.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e68feee69f5a82d8.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-5d6d15fab36665ef.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-10916d3506f78066.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-f6aebf5fbe06d09a.rlib" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-4e1b16fc9c111ada.rlib" "-Wl,--end-group" "/nix/store/nd5skgjdwy0jznjhlaifl5rpgxgbjl3z-rust-std-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-02203e01b7df4fdd.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/nix/store/dmd8i924mcm17nclazfmplyfrl8r67bv-rust-default-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/dennis/src/mayastor/target/debug/deps/spdk_rs-3434e6399ffbb915" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
= note: /nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: cannot find -lspdk-bundle
collect2: error: ld returned 1 exit status
The following warnings were emitted during compilation:
warning: /nix/store/a4yw1svqqk4d8lhwinn9xp847zz9gfma-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf-8)
warning: /nix/store/a4yw1svqqk4d8lhwinn9xp847zz9gfma-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf-8)
warning: /nix/store/a4yw1svqqk4d8lhwinn9xp847zz9gfma-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf-8)
error: could not compile `spdk-rs` due to previous error
warning: build failed, waiting for other jobs to finish...
error: build failedI can link the libspdk-bundle.so by hand to the library directory of the cc call that fails:
sudo ln -s /home/dennis/src/mayastor/target/debug/libspdk-bundle.so /nix/store/dmd8i924mcm17nclazfmplyfrl8r67bv-rust-default-1.56.1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libspdk-bundle.soNow cargo test -- --test-threads 1 starts running the tests which fail in add_child (that also failed when I compiled spdk on my own):
Running tests/add_child.rs (target/debug/deps/add_child-3a8be718b7019c26)
running 1 test
test add_child ... [2022-07-18T11:24:22.179761508+02:00 INFO io_engine::subsys::config:mod.rs:216] Applying Mayastor configuration settings
[2022-07-18T11:24:22.180042154+02:00 DEBUG io_engine::subsys::config::opts:opts.rs:261] spdk_bdev_nvme_opts { action_on_timeout: 4, timeout_us: 5000000, timeout_admin_us: 5000000, keep_
alive_timeout_ms: 1000, transport_retry_count: 0, arbitration_burst: 0, low_priority_weight: 0, medium_priority_weight: 0, high_priority_weight: 0, nvme_adminq_poll_period_us: 1000, nvm
e_ioq_poll_period_us: 0, io_queue_requests: 0, delay_cmd_submit: true, bdev_retry_count: 0 }
[2022-07-18T11:24:22.180132935+02:00 DEBUG io_engine::subsys::config:mod.rs:220] Config {
source: None,
nvmf_tcp_tgt_conf: NvmfTgtConfig {
name: "mayastor_target",
max_namespaces: 2048,
opts: NvmfTcpTransportOpts {
max_queue_depth: 32,
max_qpairs_per_ctrl: 32,
in_capsule_data_size: 4096,
max_io_size: 131072,
io_unit_size: 131072,
max_aq_depth: 32,
num_shared_buf: 2048,
buf_cache_size: 64,
dif_insert_or_strip: false,
abort_timeout_sec: 1,
acceptor_poll_rate: 10000,
zcopy: true,
},
},
nvme_bdev_opts: NvmeBdevOpts {
action_on_timeout: 4,
timeout_us: 5000000,
timeout_admin_us: 5000000,
keep_alive_timeout_ms: 1000,
transport_retry_count: 0,
arbitration_burst: 0,
low_priority_weight: 0,
medium_priority_weight: 0,
high_priority_weight: 0,
nvme_adminq_poll_period_us: 1000,
nvme_ioq_poll_period_us: 0,
io_queue_requests: 0,
delay_cmd_submit: true,
bdev_retry_count: 0,
async_mode: false,
},
bdev_opts: BdevOpts {
bdev_io_pool_size: 65535,
bdev_io_cache_size: 512,
small_buf_pool_size: 8191,
large_buf_pool_size: 1023,
},
nexus_opts: NexusOpts {
nvmf_enable: true,
nvmf_discovery_enable: true,
nvmf_nexus_port: 4421,
nvmf_replica_port: 8420,
},
}
[2022-07-18T11:24:22.180386149+02:00 DEBUG io_engine::core::env:env.rs:547] EAL arguments ["mayastor", "--no-shconf", "-m 0", "--no-pci", "--base-virtaddr=0x200000000000", "--file-prefi
x=mayastor_pid1750993", "--huge-unlink", "--log-level=lib.eal:6", "--log-level=lib.cryptodev:5", "--log-level=user1:6", "--match-allocations", "-c 0x1"]
EAL: No available 1048576 kB hugepages reported
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
FAILED
failures:
---- add_child stdout ----
thread 'mayastor_master' panicked at 'Failed to init EAL', io-engine/src/core/env.rs:556:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', io-engine/tests/common/compose.rs:83:33
failures:
add_child
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
error: test failed, to rerun pass '-p io-engine --test add_child'I have setup 1024 2MiB huge pages - grep Huge /proc/meminfo:
AnonHugePages: 1644544 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 1024
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 2097152 kBDo I need to do something else in order to start spdk correctly?
tiagolobocastro
commented
2 years ago Could you try cd into io-engine and run the tests from there?
MaisenbacherD
commented
2 years ago Thank you @tiagolobocastro, executing the test within the io-engine directory solved the spdk issue!
Now RUST_BACKTRACE=1 cargo test -- --test-threads 1 fails with the persistence test cases:
Running tests/persistence.rs (/home/dennis/src/mayastor/target/debug/deps/persistence-ad35c24c81526093)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
running 4 tests
test persist_clean_shutdown ... FAILED
test persist_io_failure ... FAILED
test persist_unexpected_restart ... FAILED
test persistent_store_connection ... FAILED
failures:
---- persist_clean_shutdown stdout ----
thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }', composer/src/lib.rs:182:37
stack backtrace:
0: rust_begin_unwind
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
1: core::panicking::panic_fmt
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
2: core::result::unwrap_failed
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1617:5
3: core::result::Result<T,E>::expect
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1259:23
4: composer::Binary::new
at /home/dennis/src/mayastor/composer/src/lib.rs:182:19
5: composer::Binary::from_nix
at /home/dennis/src/mayastor/composer/src/lib.rs:131:9
6: persistence::start_infrastructure::{{closure}}
at ./tests/persistence.rs:345:17
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
8: persistence::persist_clean_shutdown::{{closure}}
at ./tests/persistence.rs:93:16
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
10: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/future.rs:119:9
11: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:48
12: tokio::coop::with_budget::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
13: std::thread::local::LocalKey<T>::try_with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
14: std::thread::local::LocalKey<T>::with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
15: tokio::coop::with_budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
16: tokio::coop::budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:25
18: tokio::runtime::basic_scheduler::Context::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
19: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:497:36
20: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
21: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
22: tokio::runtime::basic_scheduler::CoreGuard::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
23: tokio::runtime::basic_scheduler::CoreGuard::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
24: tokio::runtime::basic_scheduler::BasicScheduler::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
25: tokio::runtime::Runtime::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
26: persistence::persist_clean_shutdown
at ./tests/persistence.rs:143:5
27: persistence::persist_clean_shutdown::{{closure}}
at ./tests/persistence.rs:92:7
28: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
29: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
---- persist_io_failure stdout ----
thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }', composer/src/lib.rs:182:37
stack backtrace:
0: rust_begin_unwind
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
1: core::panicking::panic_fmt
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
2: core::result::unwrap_failed
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1617:5
3: core::result::Result<T,E>::expect
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1259:23
4: composer::Binary::new
at /home/dennis/src/mayastor/composer/src/lib.rs:182:19
5: composer::Binary::from_nix
at /home/dennis/src/mayastor/composer/src/lib.rs:131:9
6: persistence::start_infrastructure::{{closure}}
at ./tests/persistence.rs:345:17
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
8: persistence::persist_io_failure::{{closure}}
at ./tests/persistence.rs:150:16
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
10: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/future.rs:119:9
11: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:48
12: tokio::coop::with_budget::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
13: std::thread::local::LocalKey<T>::try_with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
14: std::thread::local::LocalKey<T>::with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
15: tokio::coop::with_budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
16: tokio::coop::budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:25
18: tokio::runtime::basic_scheduler::Context::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
19: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:497:36
20: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
21: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
22: tokio::runtime::basic_scheduler::CoreGuard::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
23: tokio::runtime::basic_scheduler::CoreGuard::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
24: tokio::runtime::basic_scheduler::BasicScheduler::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
25: tokio::runtime::Runtime::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
26: persistence::persist_io_failure
at ./tests/persistence.rs:292:5
27: persistence::persist_io_failure::{{closure}}
at ./tests/persistence.rs:149:7
28: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
29: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
---- persist_unexpected_restart stdout ----
thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }', composer/src/lib.rs:182:37
stack backtrace:
0: rust_begin_unwind
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
1: core::panicking::panic_fmt
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
2: core::result::unwrap_failed
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1617:5
3: core::result::Result<T,E>::expect
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1259:23
4: composer::Binary::new
at /home/dennis/src/mayastor/composer/src/lib.rs:182:19
5: composer::Binary::from_nix
at /home/dennis/src/mayastor/composer/src/lib.rs:131:9
6: persistence::start_infrastructure::{{closure}}
at ./tests/persistence.rs:345:17
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
8: persistence::persist_unexpected_restart::{{closure}}
at ./tests/persistence.rs:39:16
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
10: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/future.rs:119:9
11: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:48
12: tokio::coop::with_budget::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
13: std::thread::local::LocalKey<T>::try_with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
14: std::thread::local::LocalKey<T>::with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
15: tokio::coop::with_budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
16: tokio::coop::budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:25
18: tokio::runtime::basic_scheduler::Context::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
19: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:497:36
20: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
21: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
22: tokio::runtime::basic_scheduler::CoreGuard::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
23: tokio::runtime::basic_scheduler::CoreGuard::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
24: tokio::runtime::basic_scheduler::BasicScheduler::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
25: tokio::runtime::Runtime::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
26: persistence::persist_unexpected_restart
at ./tests/persistence.rs:86:5
27: persistence::persist_unexpected_restart::{{closure}}
at ./tests/persistence.rs:38:7
28: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
29: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
---- persistent_store_connection stdout ----
thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }', composer/src/lib.rs:182:37
stack backtrace:
0: rust_begin_unwind
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
1: core::panicking::panic_fmt
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
2: core::result::unwrap_failed
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1617:5
3: core::result::Result<T,E>::expect
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/result.rs:1259:23
4: composer::Binary::new
at /home/dennis/src/mayastor/composer/src/lib.rs:182:19
5: composer::Binary::from_nix
at /home/dennis/src/mayastor/composer/src/lib.rs:131:9
6: persistence::start_infrastructure::{{closure}}
at ./tests/persistence.rs:345:17
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
8: persistence::persistent_store_connection::{{closure}}
at ./tests/persistence.rs:299:16
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
10: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/future.rs:119:9
11: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:48
12: tokio::coop::with_budget::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
13: std::thread::local::LocalKey<T>::try_with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
14: std::thread::local::LocalKey<T>::with
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
15: tokio::coop::with_budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
16: tokio::coop::budget
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:498:25
18: tokio::runtime::basic_scheduler::Context::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:356:19
19: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:497:36
20: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:57
21: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
22: tokio::runtime::basic_scheduler::CoreGuard::enter
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:555:27
23: tokio::runtime::basic_scheduler::CoreGuard::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:488:9
24: tokio::runtime::basic_scheduler::BasicScheduler::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/basic_scheduler.rs:168:24
25: tokio::runtime::Runtime::block_on
at /home/dennis/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/mod.rs:475:46
26: persistence::persistent_store_connection
at ./tests/persistence.rs:334:5
27: persistence::persistent_store_connection::{{closure}}
at ./tests/persistence.rs:298:7
28: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
29: core::ops::function::FnOnce::call_once
at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
failures:
persist_clean_shutdown
persist_io_failure
persist_unexpected_restart
persistent_store_connection
test result: FAILED. 0 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
error: test failed, to rerun pass '--test persistence'It looks like etcd can not be found in the PATH because these tests are executed with the root user.
Output for which etcd in the nix shell with my regular user:
/nix/store/r2av08h9shmgqkzs87j7dhhaxxbpnqkd-etcd-3.3.25/bin/etcdroot user? When you enter the nix-shell it uses your current user, I'm confused, how are you running the tests? The etcd client is definitely provided within the nix-shell: https://github.com/openebs/mayastor/blob/be4c196f24f696d502ed281fb605290e0743e6ba/ci.nix#L30 EDIT: ah maybe the root is when we elevate so we can grab hugepages and other things IIRC
MaisenbacherD
commented
2 years ago EDIT: ah maybe the root is when we elevate so we can grab hugepages and other things IIRC
This might be the case. I was temporarily adding a line to print std::process::Command::new("whoami").output()? in this function, which printed root.
To recap what I did to run the tests from the start:
sudo modprobe nbd
sudo modprobe nvmet
sudo modprobe nvmet-rdma
sudo modprobe nvme-fabrics
sudo modprobe nvme-tcp
sudo modprobe nvme-rdma
sudo modprobe nvme-loop
echo 4096 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
git clone https://github.com/openebs/mayastor.git
cd mayastor
git submodule update --init --recursive
nix-shell ci.nixIn the nix shell:
cd io-engine
cargo test -- --test-threads 1Once you enter nix-shell ci.nix, can you run etcd --version?
MaisenbacherD
commented
2 years ago Sure :)
Here the output of etcd --version:
etcd Version: 3.3.25
Git SHA: GitNotFound
Go Version: go1.16.4
Go OS/Arch: linux/amd64So it is there, then I don't understand this:
thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }'
MaisenbacherD
commented
2 years ago etcd is accessible from my regular user because the PATH env is set accordingly in the nix-shell.
Thus etcd --version results in:
etcd Version: 3.3.25
Git SHA: GitNotFound
Go Version: go1.16.4
Go OS/Arch: linux/amd64However, when running sudo etcd --version the output is:
sudo: etcd: command not foundbecause the correct PATH is not preserved when running as root.
The cargo tests are run with the root user and therefore the etcd binary can not be found, which results in the panic thread 'main' panicked at 'Binary path should exist!: Custom { kind: NotFound, error: "etcd" }'.
I am not really familiar with nix, but is there the possibility to fix the PATH env for the root user within the nix shell?
tiagolobocastro
commented
2 years ago Ah you might need to add /nix to the sudoers secure path.
MaisenbacherD
commented
2 years ago That works! But only if I prepend the whole path to etcd (/nix/store/r2av08h9shmgqkzs87j7dhhaxxbpnqkd-etcd-3.3.25) in the secure_path. I unsuccessfully tied to specify wild card paths in the secure_path. Feels like a hack to me (with which I can live for now :) ) that could cause problems in the future. Maybe there is a cleaner solution with some nix/sudoers configuration.
Tests with cargo test -- --test-threads 1 pass now :)
Thanks @tiagolobocastro and @Abhinandan-Purkait for the help!
tiagolobocastro
commented
2 years ago @MaisenbacherD, could you please remove your secure_path changes and try this branch?
MaisenbacherD
commented
2 years ago Thank you @tiagolobocastro. That fix looks nice and works without hacking Debian's default secure_path :)
Describe the bug Hello :) I am trying to setup a Mayastor development environment where I am able to run the test suite with
cargo test -- --test-threads 1.To do so I am following the instructions of https://github.com/openebs/mayastor/blob/develop/doc/build.md and later on https://github.com/openebs/mayastor/blob/develop/doc/test.md
However, the
nix developstep fails when using the nix-shell with this error message:When trying to use the docker nix-shell with
docker run --name mayastor-nix-prefetch -it -v $(pwd):/scratch:rw --privileged --workdir /scratch nixos/nix nix-shell --run "exit 0"I am getting the following error:To Reproduce Installing Nix as recommended on https://nixos.org/download.html :
Following the build instructions:
After that I tried to get the nix-shell in a docker container:
Expected behavior I am expecting to be dropped into a nix-shell where the prerequisites are installed, such that I am able to execute
cargo build.Screenshots Output of
nix-shell -I nixpkgs=channel:nixpkgs-unstable -p nixUnstable --command "nix --experimental-features 'nix-command flakes' develop -f . mayastor":Output of
docker run --name mayastor-nix-prefetch -it -v $(pwd):/scratch:rw --privileged --workdir /scratch nixos/nix nix-shell --run "exit 0":OS info (please complete the following information):
Additional context x