Cannot start the Docker container

Jack12816 commented 11 months ago

Version & Environment

Redpanda version: (use rpk version): v23.2.13 (rev 07caacd64)

Operating System (e.g. from /etc/os-release): Arch Linux
Any Kafka client libraries in use (e.g. rdkafka, franz-go): No
Docker (use docker info): 24.0.5
Kubernetes (use kubectl version): No

What went wrong?

Redpanda boot up fails with unhelpful error (error system:22, Invalid argument / main - application.cc:389 - Failure during startup: std::__1::system_error (error system:22, Invalid argument))

What should have happened instead?

Redpanda boots up as expected or raise errors which are helpful.

How to reproduce the issue?

$ docker run --pull=always redpandadata/redpanda redpanda start \
  --smp 1 \
  --mode dev-container \
  --verbose \
  --default-log-level=trace

Additional information

Command outputs

``` latest: Pulling from redpandadata/redpanda Digest: sha256:2e99453fcef907e80a841796daa4b335d66ea695e074588df04370a912af8b75 Status: Image is up to date for redpandadata/redpanda:latest + '[' '' = true ']' + exec /usr/bin/rpk redpanda start --smp 1 --mode dev-container --verbose --default-log-level=trace WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly. 05:40:04.528 DEBUG Looking for redpanda install directory 05:40:04.528 DEBUG Checking if path '/opt/redpanda/bin/rpk' exists 05:40:04.528 DEBUG Checking if path '/opt/redpanda/bin/redpanda' exists 05:40:04.528 DEBUG Redpanda is installed in '/opt/redpanda' We'd love to hear about your experience with Redpanda: https://redpanda.com/feedback Starting redpanda... Running: /opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --default-log-level=trace --unsafe-bypass-fsync=true --reserve-memory=0M --overprovisioned --lock-memory=false --smp=1 Welcome to the Redpanda community! Documentation: https://docs.redpanda.com - Product documentation site GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase Support: https://support.redpanda.com - Contact the support team privately Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation! Twitter: https://twitter.com/redpandadata - All the latest Redpanda news! DEBUG 2023-10-27 05:40:04,748 seastar - smp::count: 1 DEBUG 2023-10-27 05:40:04,748 seastar - latency_goal: 0.00075 DEBUG 2023-10-27 05:40:04,758 seastar - Assign CPU0 to NUMA0 DEBUG 2023-10-27 05:40:04,758 seastar - Auto-configure 1 IO groups INFO 2023-10-27 05:40:04,891 seastar - Reactor backend: linux-aio TRACE 2023-10-27 05:40:04,891 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x6e29d55 0x6e2851c 0x6e6e88c 0x6e741df 0x6d55b91 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 TRACE 2023-10-27 05:40:04,891 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2df2f 0x6e29f84 0x6e2851c 0x6e6e88c 0x6e741df 0x6d55b91 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 WARN 2023-10-27 05:40:04,891 [shard 0] seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted) TRACE 2023-10-27 05:40:04,891 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x6e2be44 0x6eebec7 0x6e28562 0x6e6e88c 0x6e741df 0x6d55b91 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 WARN 2023-10-27 05:40:04,891 [shard 0] cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted DEBUG 2023-10-27 05:40:04,891 [shard 0] seastar - generate_config dev_id: 0 INFO 2023-10-27 05:40:04,891 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 INFO 2023-10-27 05:40:04,891 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 INFO 2023-10-27 05:40:04,891 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 DEBUG 2023-10-27 05:40:04,891 [shard 0] seastar - allocate 0 IO group INFO 2023-10-27 05:40:04,891 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 DEBUG 2023-10-27 05:40:04,891 [shard 0] seastar - attached 0 queue to 0 IO group INFO 2023-10-27 05:40:04,892 [shard 0] main - application.cc:349 - Redpanda v23.2.13 - 07caacd64fb90c07eaf2d0b40f5085ef8af5e502 INFO 2023-10-27 05:40:04,892 [shard 0] main - application.cc:357 - kernel=6.4.10-arch1-1, nodename=d59fc31bbfd4, machine=x86_64 INFO 2023-10-27 05:40:04,892 [shard 0] main - application.cc:285 - System resources: { cpus: 1, available memory: 64.000GiB, reserved memory: 0.000bytes} INFO 2023-10-27 05:40:04,892 [shard 0] main - application.cc:293 - File handle limit: 1048576/1048576 TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66ca6 0x6d34931 0x6d3f870 0x6c709e7 0x6c727e2 0x6e5393f 0x6e56fe1 0x6e541a6 0x6d56730 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 -------- seastar::continuation>, auto seastar::with_file(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file), seastar::future> seastar::future::then_impl_nrvo(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file), seastar::future>>(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&)::'lambda'(seastar::internal::promise_base_with_type>&&, auto seastar::with_file(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)&, seastar::future_state&&), seastar::file> -------- seastar::(anonymous namespace)::thread_wake_task -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'()&, seastar::future_state&&), void> -------- seastar::continuation, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>, seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>>(seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>, seastar::futurize>::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>>(seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int), seastar::future seastar::future::then_impl_nrvo ()>&&)::$_0::operator()()::'lambda'(int), seastar::future>(seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::or_terminate()::'lambda'(auto&&), seastar::futurize::type seastar::future::then_wrapped_nrvo::or_terminate()::'lambda'(auto&&)>(seastar::future::or_terminate()::'lambda'(auto&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::or_terminate()::'lambda'(auto&&)&, seastar::future_state&&), void> TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x6d349ae 0x6d3f870 0x6c709e7 0x6c727e2 0x6e5393f 0x6e56fe1 0x6e541a6 0x6d56730 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 -------- seastar::continuation>, auto seastar::with_file(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file), seastar::future> seastar::future::then_impl_nrvo(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file), seastar::future>>(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&)::'lambda'(seastar::internal::promise_base_with_type>&&, auto seastar::with_file(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)&, seastar::future_state&&), seastar::file> -------- seastar::(anonymous namespace)::thread_wake_task -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'()&, seastar::future_state&&), void> -------- seastar::continuation, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>, seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>>(seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>, seastar::futurize>::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>>(seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int), seastar::future seastar::future::then_impl_nrvo ()>&&)::$_0::operator()()::'lambda'(int), seastar::future>(seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::or_terminate()::'lambda'(auto&&), seastar::futurize::type seastar::future::then_wrapped_nrvo::or_terminate()::'lambda'(auto&&)>(seastar::future::or_terminate()::'lambda'(auto&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::or_terminate()::'lambda'(auto&&)&, seastar::future_state&&), void> TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x6d44657 0x6e5393f 0x6e56fe1 0x6e541a6 0x6d56730 0x6d54b28 0x219b415 0x71b602b /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2194fe4 -------- seastar::continuation, seastar::posix_file_impl::close()::$_0, seastar::future seastar::future>::then_impl_nrvo>(seastar::posix_file_impl::close()::$_0&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::posix_file_impl::close()::$_0&, seastar::future_state>&&), seastar::syscall_result> -------- seastar::continuation, seastar::future seastar::future::handle_exception(seastar::file::close()::$_0&&)::'lambda'(seastar::file::close()::$_0&&), seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future seastar::future::handle_exception(seastar::file::close()::$_0&&)::'lambda'(seastar::file::close()::$_0&&)>(seastar::future seastar::future::handle_exception(seastar::file::close()::$_0&&)::'lambda'(seastar::file::close()::$_0&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future seastar::future::handle_exception(seastar::file::close()::$_0&&)::'lambda'(seastar::file::close()::$_0&&)&, seastar::future_state&&), void> -------- seastar::continuation>, seastar::future>::finally_body(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)::operator()(seastar::file)::'lambda'(seastar::file&)::operator()(seastar::file&)::'lambda'(), true>::operator()(seastar::future>&&)::'lambda'(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&), seastar::futurize::type seastar::future::then_wrapped_nrvo>, seastar::future>::finally_body(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)::operator()(seastar::file)::'lambda'(seastar::file&)::operator()(seastar::file&)::'lambda'(), true>::operator()(seastar::future>&&)::'lambda'(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&)>(seastar::future>::finally_body(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)::operator()(seastar::file)::'lambda'(seastar::file&)::operator()(seastar::file&)::'lambda'(), true>::operator()(seastar::future>&&)::'lambda'(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&)&&)::'lambda'(seastar::internal::promise_base_with_type>&&, seastar::future>::finally_body(seastar::future, read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0)::'lambda'(seastar::file)::operator()(seastar::file)::'lambda'(seastar::file&)::operator()(seastar::file&)::'lambda'(), true>::operator()(seastar::future>&&)::'lambda'(read_fully_tmpbuf(std::__1::__fs::filesystem::path const&)::$_0&&)&, seastar::future_state&&), void> -------- seastar::internal::do_with_state, seastar::future>> -------- seastar::(anonymous namespace)::thread_wake_task -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'()&, seastar::future_state&&), void> -------- seastar::continuation, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>, seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>>(seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>, seastar::futurize>::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>>(seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int), seastar::future seastar::future::then_impl_nrvo ()>&&)::$_0::operator()()::'lambda'(int), seastar::future>(seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::or_terminate()::'lambda'(auto&&), seastar::futurize::type seastar::future::then_wrapped_nrvo::or_terminate()::'lambda'(auto&&)>(seastar::future::or_terminate()::'lambda'(auto&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::or_terminate()::'lambda'(auto&&)&, seastar::future_state&&), void> ERROR 2023-10-27 05:40:04,892 [shard 0] seastar - close() syscall failed: std::__1::system_error (error system:22, Invalid argument) TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2e0b0 /opt/redpanda/lib/libc++.so.1+0x51ea8 0x6d79070 0x21a5bf9 0x21eedfc 0x6f4a36b -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'()&, seastar::future_state&&), void> -------- seastar::continuation, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>, seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>>(seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>, seastar::futurize>::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>>(seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int), seastar::future seastar::future::then_impl_nrvo ()>&&)::$_0::operator()()::'lambda'(int), seastar::future>(seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::or_terminate()::'lambda'(auto&&), seastar::futurize::type seastar::future::then_wrapped_nrvo::or_terminate()::'lambda'(auto&&)>(seastar::future::or_terminate()::'lambda'(auto&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::or_terminate()::'lambda'(auto&&)&, seastar::future_state&&), void> INFO 2023-10-27 05:40:04,892 [shard 0] main - application.cc:369 - Shutdown complete. TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x22c79fa 0x21eff9d 0x6f4a36b -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'()&, seastar::future_state&&), void> -------- seastar::continuation, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>, seastar::futurize::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>>(seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda1'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>, seastar::futurize>::type seastar::future::then_wrapped_nrvo, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>>(seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::finally_body ()>&&)::$_0::operator()()::'lambda0'(), false>&, seastar::future_state&&), int> -------- seastar::continuation, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int), seastar::future seastar::future::then_impl_nrvo ()>&&)::$_0::operator()()::'lambda'(int), seastar::future>(seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::app_template::run(int, char**, std::__1::function ()>&&)::$_0::operator()()::'lambda'(int)&, seastar::future_state&&), int> -------- seastar::continuation, seastar::future::or_terminate()::'lambda'(auto&&), seastar::futurize::type seastar::future::then_wrapped_nrvo::or_terminate()::'lambda'(auto&&)>(seastar::future::or_terminate()::'lambda'(auto&&)&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::future::or_terminate()::'lambda'(auto&&)&, seastar::future_state&&), void> TRACE 2023-10-27 05:40:04,892 [shard 0] exception - Throw exception at: 0x710cd83 0x6d66fed /opt/redpanda/lib/libc++abi.so.1+0x2dd16 0x6f6e481 0x21f017b 0x6f4a36b -------- seastar::continuation, seastar::futurize::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future seastar::future::then_impl_nrvo::type>::type seastar::async(seastar::thread_attributes, application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda0'(), seastar::future>(application::run(int, char**)::$_0::operator()() const::'lambda'()&&)::'lambda'(seastar::internal::promise_base_with_type&&, seastar::futurize::type>::type seastar::async

JIRA Link: CORE-1534

Jack12816 commented 11 months ago

I have also upgraded my system now to the latest Arch state (Linux 6.5.9) and the error remains the same. I've tried the very same Docker run command on a temporary Arch Linux virtual machine (Vagrant/VBox) and there is no issue. I'm quite confused about this issue, maybe someone has an idea what I can check?

The seastar/redpanda traces are not helpful to me. I cannot see what causes seastar - close() syscall failed beside the "invalid argument".

Jack12816 commented 11 months ago

I tried to install Redpanda on my host system (https://github.com/redpanda-data/redpanda#on-other-linux) at /opt/redpanda and I was able to start it as root and as regular user (uid 1000). So it seams to be something within the container I guess.

Update: I installed Redpanda on a fresh Docker containers as described at the readme. I tried ubuntu:latest, debian:latest, redhat/ubi8-minimal:latest, redhat/ubi9-minimal, and fedora:38 - everywhere the same error. So I guess this issue is caused by Redpanda itself.

Jack12816 commented 11 months ago

I stumbled over this issue: https://github.com/scylladb/seastar/issues/545 and it seams to reason the main difference between my host and the virtual machine. My host uses 4k disk sectors while the vm uses 512 byte disk sectors.

fdisk details

### Host ``` $ fdisk /dev/md127 [..] Command (m for help): p Disk /dev/md127: 2.35 TiB, 2586771980288 bytes, 631536128 sectors Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 524288 bytes / 1048576 bytes ``` ### VM ``` $ fdisk /dev/sda3 [..] Command (m for help): p Disk /dev/sda3: 19.7 GiB, 21157117952 bytes, 41322496 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ```

Maybe it's a regression to https://github.com/redpanda-data/redpanda/commit/35a8ecc01094f737ca7039befa0136a9cb24a69d?

IsaacBell commented 10 months ago

I have the same issue. Any new workarounds?

I'll post my setup.

docker-compose.yml

services:
    influxdb:
        image: influxdb:2.0.7
        environment:
            DOCKER_INFLUXDB_INIT_MODE: setup
            DOCKER_INFLUXDB_INIT_USERNAME: ${INFLUXDB_USERNAME}
            DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_PASSWORD}
            DOCKER_INFLUXDB_INIT_ORG: iot
            DOCKER_INFLUXDB_INIT_BUCKET: users_business_events
            DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${INFLUXDB_TOKEN}
        ports:
        - "8086:8086"
    postgresql:
        image: postgres:15.1
        hostname: postgresql
        container_name: postgresDB
        volumes:
        - pg_data:/var/lib/postgresql/data
        environment:
            PGDATA: "/var/lib/postgresql/data"
            POSTGRES_DB: "core_dev"
            POSTGRES_USER: "core"
            POSTGRES_PASSWORD: "pass1234"
            POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
        ports:
        - "5432:5432"
    conduktor-platform:
        image: conduktor/conduktor-platform:1.19.2
        depends_on:
            - postgresql
            - redpanda-0
        ports:
            - "8080:8080"
        volumes:
            - conduktor_data:/var/conduktor
        environment:
            CDK_DATABASE_URL: "postgresql://conduktor:change_me@postgresql:5432/conduktor-platform"
            CDK_CLUSTERS_0_ID: "local-kafka"
            CDK_CLUSTERS_0_NAME: "local-kafka"
            CDK_CLUSTERS_0_BOOTSTRAPSERVERS: "redpanda-0:9092"
            CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: "http://redpanda-0:18081"
            CDK_CLUSTERS_0_COLOR: "#6A57C8"
            CDK_CLUSTERS_0_ICON: "kafka"
            CDK_CLUSTERS_1_ID: "cdk-gateway"
            CDK_CLUSTERS_1_NAME: "cdk-gateway"
            CDK_CLUSTERS_1_BOOTSTRAPSERVERS: "conduktor-gateway:6969"
            CDK_CLUSTERS_1_SCHEMAREGISTRY_URL: "http://redpanda-0:18081"
            CDK_CLUSTERS_1_KAFKAFLAVOR_URL: "http://conduktor-gateway:8888"
            CDK_CLUSTERS_1_KAFKAFLAVOR_USER: "admin"
            CDK_CLUSTERS_1_KAFKAFLAVOR_PASSWORD: "conduktor"
            CDK_CLUSTERS_1_KAFKAFLAVOR_VIRTUALCLUSTER: "passthrough"
            CDK_CLUSTERS_1_KAFKAFLAVOR_TYPE: "Gateway"
            CDK_CLUSTERS_1_COLOR: "#6A57C8"
            CDK_CLUSTERS_1_ICON: "dog"
            CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
            CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/
            CDK_MONITORING_CALLBACK-URL: http://conduktor-platform:8080/monitoring/api/
            CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080

    conduktor-monitoring:
        image: conduktor/conduktor-platform-cortex:1.19.2
        environment:
            CDK_CONSOLE-URL: "http://conduktor-platform:8080"

    redpanda-0:
        command:
        - redpanda
        - start
        - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
        # Address the broker advertises to clients that connect to the Kafka API.
        # Use the internal addresses to connect to the Redpanda brokers'
        # from inside the same Docker network.
        # Use the external addresses to connect to the Redpanda brokers'
        # from outside the Docker network.
        - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
        - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
        # Address the broker advertises to clients that connect to the HTTP Proxy.
        - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
        - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
        # Redpanda brokers use the RPC API to communicate with eachother internally.
        - --rpc-addr redpanda-0:33145
        - --advertise-rpc-addr redpanda-0:33145
        # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
        - --smp 1
        # The amount of memory to make available to Redpanda.
        - --memory 1G
        # Mode dev-container uses well-known configuration properties for development in containers.
        - --mode dev-container
        # enable logs for debugging.
        - --default-log-level=debug
        image: docker.redpanda.com/redpandadata/redpanda:v23.2.17
        container_name: redpanda-0
        healthcheck:
            test: ["CMD-SHELL", "rpk cluster health | grep -E 'Healthy:.+true' || exit 1"]
            interval: 15s
            timeout: 3s
            retries: 5
            start_period: 5s

        volumes:
        - redpanda-0:/var/lib/redpanda/data
        networks:
        - redpanda_network
        ports:
        - 18081:18081
        - 18082:18082
        - 19092:19092
        - 19644:9644

    console:
        container_name: redpanda-console
        image: docker.redpanda.com/redpandadata/console:v2.3.8
        networks:
        - redpanda_network
        entrypoint: /bin/sh
        command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
        environment:
            CONFIG_FILEPATH: /tmp/config.yml
            CONSOLE_CONFIG_FILE: |
                kafka:
                    brokers: ["redpanda-0:9092"]
                    schemaRegistry:
                        enabled: true
                        urls: ["http://redpanda-0:8081"]
                redpanda:
                    adminApi:
                        enabled: true
                        urls: ["http://redpanda-0:9644"]
        ports:
        - 8088:8088
        depends_on:
        - redpanda-0

    conduktor-gateway:
        image: conduktor/conduktor-gateway:2.3.0
        hostname: conduktor-gateway
        container_name: conduktor-gateway
        environment:
            KAFKA_BOOTSTRAP_SERVERS: redpanda-0:9092
        ports:
            - "8888:8888"
        healthcheck:
            test: curl localhost:8888/health
            interval: 5s
            retries: 25
        depends_on:
            redpanda-0:
                condition: service_healthy

    conduktor-data-generator:
        image: conduktor/conduktor-data-generator:0.1
        container_name: conduktor-data-generator
        platform: linux/arm64/v8
        environment:
            KAFKA_BOOTSTRAP_SERVERS: conduktor-gateway:6969
            KAFKA_SCHEMA_REGISTRY_URL: http://redpanda-0:8081
            GATEWAY_ADMIN_API: http://conduktor-gateway:8888
            restart: on-failure
        depends_on:
            redpanda-0:
                condition: service_healthy
            conduktor-gateway:
                condition: service_healthy
networks:
  redpanda_network:
    driver: bridge
volumes:
  pg_data: {}
  conduktor_data: {}
  redpanda-0: null

And my Docker output:

$ docker logs redpanda-0
+ '[' '' = true ']'
+ exec /usr/bin/rpk redpanda start --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 --rpc-addr redpanda-0:33145 --advertise-rpc-addr redpanda-0:33145 --smp 1 --memory 1G --mode dev-container --default-log-level=debug
WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.
We'd love to hear about your experience with Redpanda:
https://redpanda.com/feedback
Starting redpanda...
Running:
/opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --default-log-level=debug --lock-memory=false --unsafe-bypass-fsync=true --overprovisioned --smp=1 --memory=1G --reserve-memory=0M

Welcome to the Redpanda community!

Documentation: https://docs.redpanda.com - Product documentation site
GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions
GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase
Support: https://support.redpanda.com - Contact the support team privately
Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience
Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation!
Twitter: https://twitter.com/redpandadata - All the latest Redpanda news!

DEBUG 2023-12-10 18:37:50,317 seastar - smp::count: 1
DEBUG 2023-12-10 18:37:50,318 seastar - latency_goal: 0.00075
DEBUG 2023-12-10 18:37:50,358 seastar - Assign CPU0 to NUMA0
DEBUG 2023-12-10 18:37:50,358 seastar - Auto-configure 1 IO groups
INFO  2023-12-10 18:37:50,360 seastar - Reactor backend: linux-aio
WARN  2023-12-10 18:37:50,361 [shard 0] seastar - Creation of perf_event based stall detector failed: falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
WARN  2023-12-10 18:37:50,361 [shard 0] cpu_profiler - Creation of perf_event based cpu profiler failed: falling back to posix timer: perf_event_open() failed: Operation not permitted
DEBUG 2023-12-10 18:37:50,362 [shard 0] seastar - generate_config dev_id: 0
INFO  2023-12-10 18:37:50,362 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
INFO  2023-12-10 18:37:50,362 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0
INFO  2023-12-10 18:37:50,362 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
DEBUG 2023-12-10 18:37:50,362 [shard 0] seastar - allocate 0 IO group
INFO  2023-12-10 18:37:50,362 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000
DEBUG 2023-12-10 18:37:50,362 [shard 0] seastar - attached 0 queue to 0 IO group
INFO  2023-12-10 18:37:50,362 [shard 0] main - application.cc:349 - Redpanda v23.2.17 - ae9a71b9139178fa15cc0dccdaf49ae1ac0c8d59
INFO  2023-12-10 18:37:50,362 [shard 0] main - application.cc:357 - kernel=6.1.57-060157-generic, nodename=cf5fc135788b, machine=x86_64
INFO  2023-12-10 18:37:50,362 [shard 0] main - application.cc:285 - System resources: { cpus: 1, available memory: 1024.000MiB, reserved memory: 0.000bytes}
INFO  2023-12-10 18:37:50,362 [shard 0] main - application.cc:293 - File handle limit: 1048576/1048576
INFO  2023-12-10 18:37:50,364 [shard 0] main - application.cc:369 - Shutdown complete.
ERROR 2023-12-10 18:37:50,364 [shard 0] main - application.cc:389 - Failure during startup: std::__1::system_error (error system:22, Invalid argument)
DEBUG 2023-12-10 18:37:50,364 [shard 0] seastar - reactor::drain

rockwotj commented 5 months ago

FWIW I got this when I was mounting Redpanda on an esoteric filesystem, so it's certainly something around non-standard disk/filesystems

Jack12816 commented 5 months ago

I've retested it today and it works now. :rocket:

$ docker run --pull=always redpandadata/redpanda redpanda start \
  --smp 1 \
  --mode dev-container \
  --verbose \
  --default-log-level=trace

INFO  2024-05-03 06:23:46,339 [shard 0:main] main - application.cc:424 - Redpanda v24.1.1 - b5ade3f40320514e855bacfad3a75d76e25a7326
INFO  2024-05-03 06:23:46,339 [shard 0:main] main - application.cc:432 - kernel=6.8.8-arch1-1, nodename=3347b66a98c4, machine=x86_64
INFO  2024-05-03 06:23:46,339 [shard 0:main] main - application.cc:360 - System resources: { cpus: 1, available memory: 64.000GiB, reserved memory: 0.000bytes}
INFO  2024-05-03 06:23:46,339 [shard 0:main] main - application.cc:368 - File handle limit: 1048576/1048576

@rockwotj: [..], so it's certainly something around non-standard disk/filesystems

My disk/filesystem/os did not changed since back then, so it was fixed upstream by the application or the framework. I'm using XFS for the root on top of a mdraid level 0 on top of some PCIe NVME SSDs. Just for the sake of completeness:

$ mount

/dev/md127 on / type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=2048,noquota)

$ xfs_info /

meta-data=/dev/md127             isize=512    agcount=32, agsize=19735424 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0 nrext64=0
data     =                       bsize=4096   blocks=631533568, imaxpct=5
         =                       sunit=128    swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=308366, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ cat /proc/mdstat

Personalities : [raid0]
md127 : active raid0 nvme0n1p2[0] nvme1n1p2[1]
      2526144512 blocks super 1.2 512k chunks

unused devices: <none>

$ lspci | grep -i nvme

1b:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)
8d:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)

I'm closing the issue now, as it works for me.

redpanda-data / redpanda