search

openbmc / bmcweb

A do everything Redfish, KVM, GUI, and DBus webserver for OpenBMC
Apache License 2.0
154 stars 131 forks source link

Hangs and consuming all CPU cycles in CRYPTO_THREAD_run_once #218

Closed leiyu-bytedance closed 3 years ago

leiyu-bytedance commented 3 years ago

Describe the bug bmcweb hangs and consuming all CPU cycles.

Environment The bmcweb revision is 19a8815adc2d58d69a15f492266ae0e921816102

To Reproduce We could not find stable reproduce steps, the BMC is being used for a few weeks, various operations are done via WebUI. When the issue occurs, bmcweb does not accept new requests, and it consumes all CPU cycles.

Manually trigger a coredump, and the stack backtrace shows below:

#0  __GI___pthread_once (once_control=once_control@entry=0x4bf89230 <ex_data_init>, init_routine=0x4becc290 <do_ex_data_init_ossl_>) at pthread_once.c:140
#1  0x4bf0b284 in CRYPTO_THREAD_run_once (once=once@entry=0x4bf89230 <ex_data_init>, init=<optimized out>) at ../openssl-1.1.1h/crypto/threads_pthread.c:118
#2  0x4becc224 in get_and_lock (class_index=class_index@entry=12) at ../openssl-1.1.1h/crypto/ex_data.c:59
#3  0x4becc99c in CRYPTO_free_ex_data (class_index=class_index@entry=12, obj=obj@entry=0xa45400, ad=ad@entry=0xa45448) at ../openssl-1.1.1h/crypto/ex_data.c:329
#4  0x4be2f314 in BIO_free (a=0xa45400) at ../openssl-1.1.1h/crypto/bio/bio_lib.c:134
#5  BIO_free (a=0xa45400) at ../openssl-1.1.1h/crypto/bio/bio_lib.c:110
#6  0x00255ecc in boost::asio::ssl::detail::engine::~engine() [clone .isra.0] (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/detail/impl/engine.ipp:78
#7  0x0019b5c8 in boost::asio::ssl::detail::stream_core::~stream_core (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/detail/stream_core.hpp:90
#8  boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >::~stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/ssl/stream.hpp:135
#9  boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::~flat_stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/beast/core/flat_stream.hpp:125
#10 std::default_delete<boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > > >::operator()(boost::beast::flat_stream<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >*) const [clone .part.0] [clone .lto_priv.0] (__ptr=0x9c8fe0, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/unique_ptr.h:85
#11 0x00180344 in boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >::~ssl_stream (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/beast/ssl/ssl_stream.hpp:73
#12 crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>::~Connection (this=<optimized out>, this=<optimized out>)
    at ../git/http/http_connection.hpp:273
#13 __gnu_cxx::new_allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >::destroy<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > (__p=0xa5cc48, this=0xa5cc48)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/ext/new_allocator.h:156
#14 std::allocator_traits<std::allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > >::destroy<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> > (__p=0xa5cc48, __a=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/alloc_traits.h:531
#15 std::_Sp_counted_ptr_inplace<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>, std::allocator<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0xa5cc38)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:560
#16 0x000ac170 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xa5cc38)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:158
#17 0x000a22d8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:733
#18 0x0013f644 in std::__shared_ptr<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr_base.h:1183
#19 std::shared_ptr<crow::Connection<boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> >, crow::App> >::~shared_ptr (this=<optimized out>,
    this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/c++/10.2.0/bits/shared_ptr.h:121
#20 crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}::~error_code()
    (this=<optimized out>, this=<optimized out>) at ../git/http/http_server.hpp:192
#21 boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>::~binder1() (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/bind_handler.hpp:33
#22 boost::asio::detail::executor_function::complete<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, std::allocator<void> >(boost::asio::detail::executor_function::impl_base*, bool) (base=<optimized out>,
    call=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/executor_function.hpp:109
#23 0x00179df8 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::executor_function, boost::asio::detail::executor_function> (context=..., function=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_invoke_helpers.hpp:51
#24 boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>::dispatch<boost::asio::detail::executor_function, std::allocator<void> > (a=..., f=..., this=0x74ef14)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/io_context.hpp:350
--Type <RET> for more, q to quit, c to continue without paging--
#25 boost::asio::executor::impl<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u>, std::allocator<void> >::dispatch (this=this@entry=0x74ef08, f=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/executor.hpp:94
#26 0x00123768 in boost::asio::executor::dispatch<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, std::allocator<void> >(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&&, std::allocator<void> const&) const (
    this=0x7ebbf584, this=0x7ebbf584, a=..., f=...)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/executor.hpp:263
#27 boost::asio::detail::handler_work_base<boost::asio::executor, void, boost::asio::io_context, boost::asio::executor, void>::dispatch<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}>(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}&) (handler=...,
    function=..., this=0x7ebbf584)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_work.hpp:275
#28 boost::asio::detail::handler_work<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::asio::executor, void>::complete<boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code> >(boost::asio::detail::binder1<crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::system::error_code>&, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}&) (handler=..., function=..., this=0x7ebbf584)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/handler_work.hpp:429
#29 boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::executor>, boost::asio::ip::tcp, crow::Server<crow::App, boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::executor> > >::doAccept()::{lambda(boost::system::error_code)#1}, boost::asio::executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned int) (owner=0x7ebbf5cc, base=0x7ebbf580)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/reactive_socket_accept_op.hpp:148
#30 0x00020320 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x6b7678, this=0x8dcd98)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/scheduler_operation.hpp:40
#31 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x6b7678)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/impl/scheduler.ipp:481
#32 boost::asio::detail::scheduler::run (ec=..., this=0x6b7678)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/detail/impl/scheduler.ipp:204
#33 boost::asio::io_context::run (this=<optimized out>, this=<optimized out>)
    at /extra/leiyu/openbmc//build/tmp/work/arm1176jzs-openbmc-linux-gnueabi/bmcweb/1.0+gitAUTOINC+19a8815adc-r0/recipe-sysroot/usr/include/boost/asio/impl/io_context.ipp:63
#34 main () at ../git/src/webserver_main.cpp:134
edtanous commented 3 years ago

Please answer the bug template as it's written. Namely, you omit what system you're on.

leiyu-bytedance commented 3 years ago

Please answer the bug template as it's written. Namely, you omit what system you're on.

It's a downstream system so I did not put the name. I think the bmcweb revision and the stack backtrace already gives enough information, is it?

edtanous commented 3 years ago

Closing the bug given that this isn't an upstream system. In open source we don't know what code or configuration changes you've made in your downstream, so we can't support you in this bug. If this reproduces on an upstream system, or you submit patches to upstream your system, feel free to reopen.

leiyu-bytedance commented 3 years ago

I do not think I could not open a bug based on a downstream system, because:

edtanous commented 3 years ago

bmcweb doesn't run in isolation, and is part of openbmc, which you don't include a revision for (presumably because you're operating on your fork). Considering your crash looks like it occurred somewhere in OpenSSL, it's quite possible and likely it's a system issue. If this reproduces on an upstream machine that we can inspect the configuration of, let me know, but it's too much to debug issues on everyones forks.

If you're interested in resubmitting this, please upstream your system, then if it still reproduces, we can reopen this.