search

sysown / proxysql

High-performance MySQL proxy with a GPL license.
http://www.proxysql.com
GNU General Public License v3.0
5.95k stars 970 forks source link

Memory leak in 2.3.2-percona-1.1 #3906

Open Wouter0100 opened 2 years ago

Wouter0100 commented 2 years ago

We're running ProxySQL 2.3.2-percona-1.1, packed/distributed by Percona (due to proxysql-admin and some other additional nice-to-haves). Regardless, we're having memory leak issues overal all our ~34 ProxySQL instances and I'm unable to figure out why.

One test server grew from:

Admin> select Variable_Name,Variable_Value/1024/1024 as Mem_usage_in_MB from stats_memory_metrics;
+------------------------------+-----------------+
| Variable_Name                | Mem_usage_in_MB |
+------------------------------+-----------------+
| SQLite3_memory_bytes         | 17              |
| jemalloc_resident            | 45              |
| jemalloc_active              | 36              |
| jemalloc_allocated           | 34              |
| jemalloc_mapped              | 64              |
| jemalloc_metadata            | 7               |
| jemalloc_retained            | 400             |
| Auth_memory                  | 0               |
| query_digest_memory          | 0               |
| mysql_query_rules_memory     | 0               |
| mysql_firewall_users_table   | 0               |
| mysql_firewall_users_config  | 0               |
| mysql_firewall_rules_table   | 0               |
| mysql_firewall_rules_config  | 0               |
| stack_memory_mysql_threads   | 32              |
| stack_memory_admin_threads   | 8               |
| stack_memory_cluster_threads | 0               |
+------------------------------+-----------------+

to the following in about a week:

Admin> select Variable_Name,Variable_Value/1024/1024 as Mem_usage_in_MB from stats_memory_metrics;
+------------------------------+-----------------+
| Variable_Name                | Mem_usage_in_MB |
+------------------------------+-----------------+
| SQLite3_memory_bytes         | 267             |
| jemalloc_resident            | 340             |
| jemalloc_active              | 319             |
| jemalloc_allocated           | 315             |
| jemalloc_mapped              | 358             |
| jemalloc_metadata            | 16              |
| jemalloc_retained            | 682             |
| Auth_memory                  | 0               |
| query_digest_memory          | 0               |
| mysql_query_rules_memory     | 0               |
| mysql_firewall_users_table   | 0               |
| mysql_firewall_users_config  | 0               |
| mysql_firewall_rules_table   | 0               |
| mysql_firewall_rules_config  | 0               |
| stack_memory_mysql_threads   | 32              |
| stack_memory_admin_threads   | 8               |
| stack_memory_cluster_threads | 0               |
+------------------------------+-----------------+

Hereby a jeprof.tar.gz. In this case it grew to around 600MB of memory usage over a period of ~2 weeks. On all our servers the growth is seems constant until an OOM or we manually restart it. Some servers have way more RAM then just the 4GB on this test machine, but its never being enough. image

The above jeprof results in the following line output by jeprof with the binary that we're using:

jeprof output ``` Total: 108419.5 MB 107541.3 99.2% 99.2% 107541.3 99.2% malloc_usable_size ??:? 548.1 0.5% 99.7% 548.1 0.5% std::_Deque_base::_M_initialize_map ??:? 230.0 0.2% 99.9% 755.2 0.7% MySQL_Connection::reset ??:? 28.0 0.0% 99.9% 2985.8 2.8% MySQL_Thread::create_new_session_and_client_data_stream ??:? 28.0 0.0% 100.0% 49.0 0.0% MySQL_Session::MySQL_Session ??:? 16.0 0.0% 100.0% 39.0 0.0% MySQL_Connection::MySQL_Connection ??:? 12.0 0.0% 100.0% 12.0 0.0% __gnu_cxx::new_allocator::allocate ??:? 5.0 0.0% 100.0% 8.0 0.0% MySQL_Session::init ??:? 2.0 0.0% 100.0% 2.0 0.0% std::__cxx11::basic_string::_M_mutate ??:? 2.0 0.0% 100.0% 2.0 0.0% re2::Regexp::ParseState::PushRepeatOp ??:? 1.0 0.0% 100.0% 1.0 0.0% monitor_galera_thread ??:? 1.0 0.0% 100.0% 1.0 0.0% re2::Compiler::Compiler ??:? 1.0 0.0% 100.0% 3.0 0.0% re2::Regexp::ConcatOrAlternate ??:? 1.0 0.0% 100.0% 1.0 0.0% re2::Regexp::LiteralString ??:? 1.0 0.0% 100.0% 1.0 0.0% re2::Regexp::NewCharClass ??:? 1.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ParseState::DoLeftParen ??:? 1.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ParseState::PushRegexp ??:? 0.0 0.0% 100.0% 2.0 0.0% ConsumerThread::run ??:? 0.0 0.0% 100.0% 13051.4 12.0% MySQL_Connection::async_query ??:? 0.0 0.0% 100.0% 43993.5 40.6% MySQL_Connection::handler ??:? 0.0 0.0% 100.0% 2840.9 2.6% MySQL_Data_Stream::MySQL_Data_Stream ??:? 0.0 0.0% 100.0% 45.1 0.0% MySQL_Data_Stream::buffer2array ??:? 0.0 0.0% 100.0% 6.0 0.0% MySQL_Data_Stream::init@318bb0 ??:? 0.0 0.0% 100.0% 6.0 0.0% MySQL_Data_Stream::init@318d80 ??:? 0.0 0.0% 100.0% 45.1 0.0% MySQL_Data_Stream::read_pkts ??:? 0.0 0.0% 100.0% 754.2 0.7% MySQL_HostGroups_Manager::push_MyConn_to_pool_array ??:? 0.0 0.0% 100.0% 3425.6 3.2% MySQL_Protocol::generate_pkt_row3 ??:? 0.0 0.0% 100.0% 7.0 0.0% MySQL_Protocol::process_pkt_handshake_response ??:? 0.0 0.0% 100.0% 9619.7 8.9% MySQL_ResultSet::add_eof ??:? 0.0 0.0% 100.0% 3425.6 3.2% MySQL_ResultSet::add_row ??:? 0.0 0.0% 100.0% 13045.4 12.0% MySQL_ResultSet::buffer_to_PSarrayOut ??:? 0.0 0.0% 100.0% 25.2 0.0% MySQL_ResultSet::get_resultset ??:? 0.0 0.0% 100.0% 21.1 0.0% MySQL_ResultSet::init ??:? 0.0 0.0% 100.0% 548.1 0.5% MySQL_STMTs_local_v14::MySQL_STMTs_local_v14 ??:? 0.0 0.0% 100.0% 8.0 0.0% MySQL_Session::MySQL_Result_to_MySQL_wire ??:? 0.0 0.0% 100.0% 447.1 0.4% MySQL_Session::RequestEnd ??:? 0.0 0.0% 100.0% 57.2 0.1% MySQL_Session::create_backend ??:? 0.0 0.0% 100.0% 6014.6 5.5% MySQL_Session::get_pkts_from_client ??:? 0.0 0.0% 100.0% 19539.3 18.0% MySQL_Session::handler ??:? 0.0 0.0% 100.0% 7.0 0.0% MySQL_Session::handler___status_CONNECTING_CLIENT___STATE_SERVER_HANDSHAKE ??:? 0.0 0.0% 100.0% 5903.5 5.4% MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysql ??:? 0.0 0.0% 100.0% 22.0 0.0% MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_qpo ??:? 0.0 0.0% 100.0% 1.0 0.0% MySQL_Session::~MySQL_Session ??:? 0.0 0.0% 100.0% 33980.0 31.3% MySQL_Thread::ProcessAllMyDS_AfterPoll ??:? 0.0 0.0% 100.0% 2992.8 2.8% MySQL_Thread::listener_handle_new_connection ??:? 0.0 0.0% 100.0% 13636.8 12.6% MySQL_Thread::process_all_sessions ??:? 0.0 0.0% 100.0% 30987.2 28.6% MySQL_Thread::process_data_on_data_stream ??:? 0.0 0.0% 100.0% 47646.0 43.9% MySQL_Thread::run ??:? 0.0 0.0% 100.0% 29.2 0.0% MySQL_Thread::run___get_multiple_idle_connections ??:? 0.0 0.0% 100.0% 2.0 0.0% MySQL_Variables::client_set_value ??:? 0.0 0.0% 100.0% 1.0 0.0% MySQL_Variables::server_set_hash_and_value ??:? 0.0 0.0% 100.0% 19945.6 18.4% ProxySQL_Admin::FlushDigestTableToDisk ??:? 0.0 0.0% 100.0% 5903.5 5.4% ProxySQL_Admin::save_mysql_users_runtime_to_database ??:? 0.0 0.0% 100.0% 10933.7 10.1% ProxySQL_Statistics::MySQL_Threads_Handler_sets ??:? 0.0 0.0% 100.0% 10933.7 10.1% ProxySQL_Statistics::MySQL_Threads_Handler_sets_v1 ??:? 0.0 0.0% 100.0% 9012.0 8.3% ProxySQL_Statistics::system_cpu_sets ??:? 0.0 0.0% 100.0% 25.2 0.0% PtrSizeArray::copy_add ??:? 0.0 0.0% 100.0% 807.3 0.7% PtrSizeArray::expand ??:? 0.0 0.0% 100.0% 447.1 0.4% QP_query_digest_stats::QP_query_digest_stats ??:? 0.0 0.0% 100.0% 41.1 0.0% Query_Info::begin ??:? 0.0 0.0% 100.0% 447.1 0.4% Query_Info::end ??:? 0.0 0.0% 100.0% 447.1 0.4% Query_Info::query_parser_update_counters ??:? 0.0 0.0% 100.0% 41.1 0.0% Query_Processor::query_parser_init ??:? 0.0 0.0% 100.0% 447.1 0.4% Query_Processor::query_parser_update_counters ??:? 0.0 0.0% 100.0% 447.1 0.4% Query_Processor::update_query_digest ??:? 0.0 0.0% 100.0% 19945.6 18.4% SQLite3DB::execute ??:? 0.0 0.0% 100.0% 5903.5 5.4% SQLite3DB::prepare_v2 ??:? 0.0 0.0% 100.0% 22.0 0.0% SetParser::parse1[abi:cxx11] ??:? 0.0 0.0% 100.0% 49.0 0.0% __strdup ??:? 0.0 0.0% 100.0% 5903.5 5.4% admin_handler_command_load_or_save ??:? 0.0 0.0% 100.0% 5903.5 5.4% admin_session_handler ??:? 0.0 0.0% 100.0% 22718.5 21.0% calloc ??:? 0.0 0.0% 100.0% 5903.5 5.4% child_mysql ??:? 0.0 0.0% 100.0% 74251.3 68.5% clone ??:? 0.0 0.0% 100.0% 11283.9 10.4% hash_element ??:? 0.0 0.0% 100.0% 112.7 0.1% ma_alloc_root ??:? 0.0 0.0% 100.0% 12.0 0.0% ma_multi_malloc ??:? 0.0 0.0% 100.0% 11283.9 10.4% ma_net_read ??:? 0.0 0.0% 100.0% 11283.9 10.4% ma_net_safe_read ??:? 0.0 0.0% 100.0% 22708.5 20.9% ma_pvio_init ??:? 0.0 0.0% 100.0% 22.0 0.0% ma_read_ok_packet ??:? 0.0 0.0% 100.0% 67631.3 62.4% memalign ??:? 0.0 0.0% 100.0% 1.0 0.0% monitor_ping_thread ??:? 0.0 0.0% 100.0% 112.7 0.1% mthd_my_read_metadata_ex ??:? 0.0 0.0% 100.0% 11283.9 10.4% mthd_my_read_one_row ??:? 0.0 0.0% 100.0% 112.7 0.1% mthd_my_read_query_result ??:? 0.0 0.0% 100.0% 22708.5 20.9% mthd_my_real_connect ??:? 0.0 0.0% 100.0% 30942.1 28.5% my_context_init ??:? 0.0 0.0% 100.0% 34168.2 31.5% my_context_spawn ??:? 0.0 0.0% 100.0% 63.0 0.1% mysql_change_user ??:? 0.0 0.0% 100.0% 11283.9 10.4% mysql_fetch_row ??:? 0.0 0.0% 100.0% 30942.1 28.5% mysql_optionsv ??:? 0.0 0.0% 100.0% 41.1 0.0% mysql_query_digest_and_first_comment ??:? 0.0 0.0% 100.0% 22708.5 20.9% mysql_real_connect ??:? 0.0 0.0% 100.0% 6.0 0.0% mysql_use_result ??:? 0.0 0.0% 100.0% 47646.0 43.9% mysql_worker_thread_func ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::AddRuneRange ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::AddRuneRangeUTF8 ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::AllocInst ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::ByteRange ??:? 0.0 0.0% 100.0% 12.0 0.0% re2::Compiler::Compile ??:? 0.0 0.0% 100.0% 9.0 0.0% re2::Compiler::Finish ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::PostVisit ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::UncachedRuneByteSuffix ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::CachedState ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::FastSearchLoop ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::InlinedSearchLoop ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::RunStateOnByte ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::RunStateOnByteUnlocked ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::Search ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::SearchFFT ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::DFA::WorkqToCachedState ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::FactorAlternationImpl::Round1 ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::FactorAlternationImpl::Round3 ??:? 0.0 0.0% 100.0% 10.0 0.0% re2::PODArray::PODArray ??:? 0.0 0.0% 100.0% 9.0 0.0% re2::Prog::Flatten ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::Prog::MarkSuccessors ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::Prog::SearchDFA ??:? 0.0 0.0% 100.0% 21.0 0.0% re2::RE2::Init ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::RE2::Match ??:? 0.0 0.0% 100.0% 21.0 0.0% re2::RE2::RE2 ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::RE2::Replace ??:? 0.0 0.0% 100.0% 12.0 0.0% re2::Regexp::CompileToProg ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Regexp::FactorAlternation ??:? 0.0 0.0% 100.0% 7.0 0.0% re2::Regexp::Parse ??:? 0.0 0.0% 100.0% 3.0 0.0% re2::Regexp::ParseState::DoAlternation ??:? 0.0 0.0% 100.0% 3.0 0.0% re2::Regexp::ParseState::DoCollapse ??:? 0.0 0.0% 100.0% 3.0 0.0% re2::Regexp::ParseState::DoRightParen ??:? 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ParseState::PushLiteral ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Regexp::Walker::WalkExponential ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::Regexp::Walker::WalkInternal ??:? 0.0 0.0% 100.0% 5.0 0.0% re2::SparseArray::SparseArray ??:? 0.0 0.0% 100.0% 2.0 0.0% re2::SparseSetT::SparseSetT ??:? 0.0 0.0% 100.0% 17191.5 15.9% realloc ??:? 0.0 0.0% 100.0% 22.0 0.0% run_plugin_auth ??:? 0.0 0.0% 100.0% 4.0 0.0% sha1_pass_hex ??:? 0.0 0.0% 100.0% 5903.5 5.4% sqlite3_backup_init ??:? 0.0 0.0% 100.0% 25849.1 23.8% sqlite3_db_config ??:? 0.0 0.0% 100.0% 25850.1 23.8% sqlite3_errmsg ??:? 0.0 0.0% 100.0% 25849.1 23.8% sqlite3_exec ??:? 0.0 0.0% 100.0% 5903.5 5.4% sqlite3_prepare_v2 ??:? 0.0 0.0% 100.0% 19946.6 18.4% sqlite3_randomness ??:? 0.0 0.0% 100.0% 5903.5 5.4% sqlite3_realloc64 ??:? 0.0 0.0% 100.0% 19946.6 18.4% sqlite3_release_memory ??:? 0.0 0.0% 100.0% 19946.6 18.4% sqlite3_reset ??:? 0.0 0.0% 100.0% 5903.5 5.4% sqlite3_result_value ??:? 0.0 0.0% 100.0% 19946.6 18.4% sqlite3_step ??:? 0.0 0.0% 100.0% 74251.3 68.5% start_thread ??:? 0.0 0.0% 100.0% 1.0 0.0% std::_Vector_base::_M_allocate ??:? 0.0 0.0% 100.0% 2.0 0.0% std::__cxx11::basic_string::_M_replace ??:? 0.0 0.0% 100.0% 1.0 0.0% std::allocator_traits::allocate ??:? 0.0 0.0% 100.0% 754.2 0.7% std::error_code::default_error_condition ??:? 0.0 0.0% 100.0% 1.0 0.0% std::vector::_M_realloc_insert ??:? 0.0 0.0% 100.0% 1.0 0.0% std::vector::emplace_back ??:? 0.0 0.0% 100.0% 447.1 0.4% strndup ??:? 0.0 0.0% 100.0% 1.0 0.0% verify_set_names ??:? 0.0 0.0% 100.0% 34168.2 31.5% zError ??:? ```

I'm not sure how to read the above output though.

proxysql.cnf ``` datadir="/var/lib/proxysql" admin_variables= { admin_credentials="admin:x" mysql_ifaces="127.0.0.1:6032" stats_credentials="stats:x" } mysql_variables= { interfaces="[::]:3306;0.0.0.0:3306" monitor_username="proxysql" monitor_password="x" monitor_galera_healthcheck_interval=2000 # Should preferably be changed to the new value that may be introduced in sysown/proxysql#2597 set_query_lock_on_hostgroup=0 # On MySQL we deploy a timeout of 8 hours, so we make sure ProxySQL terminates first with a 6 hour timeout. wait_timeout=21600000 } mysql_galera_hostgroups = ( { writer_hostgroup=10 backup_writer_hostgroup=20 reader_hostgroup=30 offline_hostgroup=9999 max_writers=1 writer_is_also_reader=0 max_transactions_behind=30 active=1 } ) mysql_servers = ( { address="10.10.2.153" port=13306 hostgroup=10 } ) mysql_query_rules = ( { rule_id=300 active=1 match_pattern=".*" destination_hostgroup=10 apply=1 multiplex=0 } ) ```

Thanks.

balbaev commented 2 years ago

I’m experiencing the same issue but with official package ProxySQL version 2.3.2-10-g8cd66cf Proxysql without any activity grows to this in a week:

+------------------------------+-----------------+
| Variable_Name                | Mem_usage_in_MB |
+------------------------------+-----------------+
| SQLite3_memory_bytes         | 5               |
| jemalloc_resident            | 1901            |
| jemalloc_active              | 1863            |
| jemalloc_allocated           | 1852            |
| jemalloc_mapped              | 1940            |
| jemalloc_metadata            | 38              |
| jemalloc_retained            | 636             |
| Auth_memory                  | 0               |
| query_digest_memory          | 1739            |
| mysql_query_rules_memory     | 0               |
| mysql_firewall_users_table   | 0               |
| mysql_firewall_users_config  | 0               |
| mysql_firewall_rules_table   | 0               |
| mysql_firewall_rules_config  | 0               |
| stack_memory_mysql_threads   | 64              |
| stack_memory_admin_threads   | 32              |
| stack_memory_cluster_threads | 24              |
+------------------------------+-----------------+
renecannao commented 2 years ago

@Wouter0100 : which exact binary of proxysql are you using? For which distro? I need this information to make sure we run jeprof against the right binary. Also, Percona strips symbols from the binaries (in fact our binaries are around 10x bigger), making debugging very difficult. I recommend to use our binary (you can simply replace the binary without the need to uninstall and reinstall and package) and collect new metrics. In this way jeprof can tell us even at what exact line of code the memory is being allocated.

@balbaev : there is no evidence you have a similar issue, and in fact it seems completely unrelated. It also seems you do have activity, and perhaps a lot too. The in-memory query digest hash table is 1.7GB . Run TRUNCATE TABLE stats_mysql_query_digest.

Thanks

ssgoburdhun commented 2 years ago

We have been attempting to find out if we had same memory leak issue. Current proxysql version in use: "2.3.2". Uname: The repository "https://repo.proxysql.com/ProxySQL/proxysql-2.3.x/buster ./ " has been used for installation. Linux lab1 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

ProxySQL> SELECT * FROM stats.stats_memory_metrics;
+------------------------------+----------------+
| Variable_Name                | Variable_Value |
+------------------------------+----------------+
| SQLite3_memory_bytes         | 3493840        |
| jemalloc_resident            | 77922856960    |
| jemalloc_active              | 76263776256    |
| jemalloc_allocated           | 76242061080    |
| jemalloc_mapped              | 78113628160    |
| jemalloc_metadata            | 1624599576     |
| jemalloc_retained            | 11602010112    |
| Auth_memory                  | 1054           |
| query_digest_memory          | 59160          |
| mysql_query_rules_memory     | 5796           |
| mysql_firewall_users_table   | 0              |
| mysql_firewall_users_config  | 0              |
| mysql_firewall_rules_table   | 0              |
| mysql_firewall_rules_config  | 329            |
| stack_memory_mysql_threads   | 335544320      |
| stack_memory_admin_threads   | 8388608        |
| stack_memory_cluster_threads | 0              |
+------------------------------+----------------+
17 rows in set (0.003 sec)

We used a load balancer (h2load) to simulate proxysql activity and performance. Performance is satisfactory. The jemalloc_allocated shows 76G consumed by the app. Can we determine if jemalloc is intelligently keeping the used memory pages for future connectivity? Also, mysql-threads is set to 20. Maybe its the number of backend connections opened and kept alived that are not getting released for inactivity?

Wouter0100 commented 2 years ago

@renecannao thanks for your response. Attached the exact binary. proxysql.zip

While replacing the ProxySQL binary with one provided by you, I see this server grew again to 429M.

Admin> select Variable_Name,Variable_Value/1024/1024 as Mem_usage_in_MB from stats_memory_metrics;
+------------------------------+-----------------+
| Variable_Name                | Mem_usage_in_MB |
+------------------------------+-----------------+
| SQLite3_memory_bytes         | 258             |
| jemalloc_resident            | 422             |
| jemalloc_active              | 375             |
| jemalloc_allocated           | 372             |
| jemalloc_mapped              | 442             |
| jemalloc_metadata            | 16              |
| jemalloc_retained            | 673             |
| Auth_memory                  | 0               |
| query_digest_memory          | 0               |
| mysql_query_rules_memory     | 0               |
| mysql_firewall_users_table   | 0               |
| mysql_firewall_users_config  | 0               |
| mysql_firewall_rules_table   | 0               |
| mysql_firewall_rules_config  | 0               |
| stack_memory_mysql_threads   | 32              |
| stack_memory_admin_threads   | 8               |
| stack_memory_cluster_threads | 0               |
+------------------------------+-----------------+
17 rows in set (0.008 sec)

I replaced the binary with the binary from this deb package, as we're running Ubuntu 20.04.3. I enabled memory profiling and will get back to you beginning next week.

Wouter0100 commented 2 years ago

Okay, I got another profile with the official ProxySQL binary that has been running for some time. A significant amount of the memory should represent the leak.

image

Systemd reported 242M memory usage:

root@db1:~# systemctl status proxysql
● proxysql.service - High Performance Advanced Proxy for MySQL
     Loaded: loaded (/lib/systemd/system/proxysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-07-05 16:39:57 CEST; 6 days ago
   Main PID: 3367891 (proxysql)
      Tasks: 20 (limit: 4677)
     Memory: 241.9M
     CGroup: /system.slice/proxysql.service
             ├─3367891 /usr/bin/proxysql -c /etc/proxysql.cnf
             └─3367892 /usr/bin/proxysql -c /etc/proxysql.cnf

ProxySQL reported the following:

Admin> select Variable_Name,Variable_Value/1024/1024 as Mem_usage_in_MB from stats_memory_metrics;
+------------------------------+-----------------+
| Variable_Name                | Mem_usage_in_MB |
+------------------------------+-----------------+
| SQLite3_memory_bytes         | 145             |
| jemalloc_resident            | 231             |
| jemalloc_active              | 217             |
| jemalloc_allocated           | 214             |
| jemalloc_mapped              | 244             |
| jemalloc_metadata            | 12              |
| jemalloc_retained            | 728             |
| Auth_memory                  | 0               |
| query_digest_memory          | 0               |
| mysql_query_rules_memory     | 0               |
| mysql_firewall_users_table   | 0               |
| mysql_firewall_users_config  | 0               |
| mysql_firewall_rules_table   | 0               |
| mysql_firewall_rules_config  | 0               |
| stack_memory_mysql_threads   | 32              |
| stack_memory_admin_threads   | 8               |
| stack_memory_cluster_threads | 0               |
+------------------------------+-----------------+
17 rows in set (0.005 sec)

And.. jeprof output:

jeprof output ``` Total: 60490.4 MB 60245.4 99.6% 99.6% 60245.4 99.6% je_prof_backtrace /opt/proxysql/deps/jemalloc/jemalloc/src/prof.c:636 (discriminator 4) 154.0 0.3% 99.8% 154.0 0.3% __gnu_cxx::new_allocator::allocate (inline) /usr/include/c++/9/ext/new_allocator.h:114 74.0 0.1% 100.0% 209.0 0.3% MySQL_Connection::reset /opt/proxysql/lib/mysql_connection.cpp:2593 11.0 0.0% 100.0% 21.0 0.0% MySQL_Thread::create_new_session_and_client_data_stream /opt/proxysql/lib/MySQL_Thread.cpp:2803 3.0 0.0% 100.0% 10.0 0.0% MySQL_Connection::MySQL_Connection /opt/proxysql/lib/mysql_connection.cpp:436 1.0 0.0% 100.0% 1.0 0.0% re2::Compiler::Compiler /opt/proxysql/deps/re2/re2/re2/compile.cc:222 1.0 0.0% 100.0% 1.0 0.0% MySQL_Session::init /opt/proxysql/lib/MySQL_Session.cpp:514 1.0 0.0% 100.0% 1.0 0.0% re2::CharClass::New /opt/proxysql/deps/re2/re2/re2/regexp.cc:918 0.0 0.0% 100.0% 1.0 0.0% HGCU_thread_run /opt/proxysql/lib/MySQL_HostGroups_Manager.cpp:676 0.0 0.0% 100.0% 208.0 0.3% HGCU_thread_run /opt/proxysql/lib/MySQL_HostGroups_Manager.cpp:743 0.0 0.0% 100.0% 16617.3 27.5% MySQL_Connection::async_query /opt/proxysql/lib/mysql_connection.cpp:1855 0.0 0.0% 100.0% 18769.5 31.0% MySQL_Connection::handler /opt/proxysql/lib/mysql_connection.cpp:1037 0.0 0.0% 100.0% 17.1 0.0% MySQL_Connection::handler /opt/proxysql/lib/mysql_connection.cpp:1423 0.0 0.0% 100.0% 12886.3 21.3% MySQL_Connection::handler /opt/proxysql/lib/mysql_connection.cpp:1465 0.0 0.0% 100.0% 3713.9 6.1% MySQL_Connection::handler /opt/proxysql/lib/mysql_connection.cpp:1491 0.0 0.0% 100.0% 1.0 0.0% MySQL_Connection_userinfo::set_schemaname /opt/proxysql/lib/mysql_connection.cpp:370 0.0 0.0% 100.0% 631.8 1.0% MySQL_Data_Stream::MySQL_Data_Stream /opt/proxysql/lib/mysql_data_stream.cpp:272 0.0 0.0% 100.0% 637.9 1.1% MySQL_Data_Stream::MySQL_Data_Stream /opt/proxysql/lib/mysql_data_stream.cpp:273 0.0 0.0% 100.0% 12.0 0.0% MySQL_Data_Stream::buffer2array /opt/proxysql/lib/mysql_data_stream.cpp:975 (discriminator 3) 0.0 0.0% 100.0% 12.0 0.0% MySQL_Data_Stream::read_pkts /opt/proxysql/lib/mysql_data_stream.cpp:892 (discriminator 2) 0.0 0.0% 100.0% 12886.3 21.3% MySQL_Protocol::generate_pkt_row3 (inline) /opt/proxysql/lib/MySQL_Protocol.cpp:1014 0.0 0.0% 100.0% 12886.3 21.3% MySQL_Protocol::generate_pkt_row3 /opt/proxysql/lib/MySQL_Protocol.cpp:990 0.0 0.0% 100.0% 2.0 0.0% MySQL_Protocol::process_pkt_handshake_response /opt/proxysql/lib/MySQL_Protocol.cpp:2065 0.0 0.0% 100.0% 1.0 0.0% MySQL_Protocol::process_pkt_handshake_response /opt/proxysql/lib/MySQL_Protocol.cpp:2066 (discriminator 1) 0.0 0.0% 100.0% 10.1 0.0% MySQL_ResultSet::add_eof /opt/proxysql/lib/MySQL_Protocol.cpp:2829 0.0 0.0% 100.0% 3720.9 6.2% MySQL_ResultSet::add_eof /opt/proxysql/lib/MySQL_Protocol.cpp:2832 0.0 0.0% 100.0% 12886.3 21.3% MySQL_ResultSet::add_row /opt/proxysql/lib/MySQL_Protocol.cpp:2744 0.0 0.0% 100.0% 11071.2 18.3% MySQL_ResultSet::buffer_to_PSarrayOut (inline) /opt/proxysql/lib/MySQL_Protocol.cpp:2903 0.0 0.0% 100.0% 5542.1 9.2% MySQL_ResultSet::buffer_to_PSarrayOut (inline) /opt/proxysql/lib/MySQL_Protocol.cpp:2907 0.0 0.0% 100.0% 16613.3 27.5% MySQL_ResultSet::buffer_to_PSarrayOut /opt/proxysql/lib/MySQL_Protocol.cpp:2895 0.0 0.0% 100.0% 4.0 0.0% MySQL_ResultSet::buffer_to_PSarrayOut /opt/proxysql/lib/MySQL_Protocol.cpp:2900 0.0 0.0% 100.0% 21.5 0.0% MySQL_ResultSet::get_resultset /opt/proxysql/lib/MySQL_Protocol.cpp:2888 0.0 0.0% 100.0% 17.1 0.0% MySQL_ResultSet::init /opt/proxysql/lib/MySQL_Protocol.cpp:2555 0.0 0.0% 100.0% 142.0 0.2% MySQL_STMTs_local_v14::MySQL_STMTs_local_v14 /opt/proxysql/lib/../include/MySQL_PreparedStatement.h:210 0.0 0.0% 100.0% 3.0 0.0% MySQL_Session::MySQL_Result_to_MySQL_wire /opt/proxysql/lib/MySQL_Session.cpp:6499 0.0 0.0% 100.0% 10.0 0.0% MySQL_Session::MySQL_Session /opt/proxysql/lib/MySQL_Session.cpp:446 0.0 0.0% 100.0% 3.0 0.0% MySQL_Session::MySQL_Session /opt/proxysql/lib/MySQL_Session.cpp:450 0.0 0.0% 100.0% 2.0 0.0% MySQL_Session::MySQL_Session /opt/proxysql/lib/MySQL_Session.cpp:452 0.0 0.0% 100.0% 2.0 0.0% MySQL_Session::MySQL_Session /opt/proxysql/lib/MySQL_Session.cpp:502 0.0 0.0% 100.0% 1.0 0.0% MySQL_Session::create_backend /opt/proxysql/lib/MySQL_Session.cpp:623 0.0 0.0% 100.0% 13.0 0.0% MySQL_Session::create_backend /opt/proxysql/lib/MySQL_Session.cpp:629 0.0 0.0% 100.0% 3.0 0.0% MySQL_Session::get_pkts_from_client /opt/proxysql/lib/MySQL_Session.cpp:3416 0.0 0.0% 100.0% 19.0 0.0% MySQL_Session::get_pkts_from_client /opt/proxysql/lib/MySQL_Session.cpp:3515 0.0 0.0% 100.0% 4.0 0.0% MySQL_Session::get_pkts_from_client /opt/proxysql/lib/MySQL_Session.cpp:3572 0.0 0.0% 100.0% 9.0 0.0% MySQL_Session::get_pkts_from_client /opt/proxysql/lib/MySQL_Session.cpp:3631 0.0 0.0% 100.0% 35.0 0.1% MySQL_Session::handler /opt/proxysql/lib/MySQL_Session.cpp:4254 0.0 0.0% 100.0% 16617.3 27.5% MySQL_Session::handler /opt/proxysql/lib/MySQL_Session.cpp:4472 0.0 0.0% 100.0% 3.0 0.0% MySQL_Session::handler /opt/proxysql/lib/MySQL_Session.cpp:4511 0.0 0.0% 100.0% 18.5 0.0% MySQL_Session::handler /opt/proxysql/lib/MySQL_Session.cpp:4607 0.0 0.0% 100.0% 3.0 0.0% MySQL_Session::handler___status_CONNECTING_CLIENT___STATE_SERVER_HANDSHAKE /opt/proxysql/lib/MySQL_Session.cpp:4818 0.0 0.0% 100.0% 4.0 0.0% MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_qpo /opt/proxysql/lib/MySQL_Session.cpp:5469 0.0 0.0% 100.0% 1.0 0.0% MySQL_Session::init /opt/proxysql/lib/MySQL_Session.cpp:513 0.0 0.0% 100.0% 1330.6 2.2% MySQL_Thread::ProcessAllMyDS_AfterPoll /opt/proxysql/lib/MySQL_Thread.cpp:3006 (discriminator 1) 0.0 0.0% 100.0% 18781.5 31.0% MySQL_Thread::ProcessAllMyDS_AfterPoll /opt/proxysql/lib/MySQL_Thread.cpp:3013 0.0 0.0% 100.0% 28.0 0.0% MySQL_Thread::create_new_session_and_client_data_stream /opt/proxysql/lib/MySQL_Thread.cpp:2779 0.0 0.0% 100.0% 1280.6 2.1% MySQL_Thread::create_new_session_and_client_data_stream /opt/proxysql/lib/MySQL_Thread.cpp:2781 (discriminator 3) 0.0 0.0% 100.0% 1.0 0.0% MySQL_Thread::listener_handle_new_connection /opt/proxysql/lib/MySQL_Thread.cpp:4075 0.0 0.0% 100.0% 1329.6 2.2% MySQL_Thread::listener_handle_new_connection /opt/proxysql/lib/MySQL_Thread.cpp:4111 0.0 0.0% 100.0% 16673.8 27.6% MySQL_Thread::process_all_sessions /opt/proxysql/lib/MySQL_Thread.cpp:3762 0.0 0.0% 100.0% 12.0 0.0% MySQL_Thread::process_data_on_data_stream /opt/proxysql/lib/MySQL_Thread.cpp:3497 0.0 0.0% 100.0% 18769.5 31.0% MySQL_Thread::process_data_on_data_stream /opt/proxysql/lib/MySQL_Thread.cpp:3534 0.0 0.0% 100.0% 7.0 0.0% MySQL_Thread::run /opt/proxysql/lib/MySQL_Thread.cpp:3073 0.0 0.0% 100.0% 20112.2 33.2% MySQL_Thread::run /opt/proxysql/lib/MySQL_Thread.cpp:3255 0.0 0.0% 100.0% 16673.8 27.6% MySQL_Thread::run /opt/proxysql/lib/MySQL_Thread.cpp:3271 0.0 0.0% 100.0% 2.0 0.0% MySQL_Thread::run___get_multiple_idle_connections /opt/proxysql/lib/MySQL_Thread.cpp:2922 0.0 0.0% 100.0% 5.0 0.0% MySQL_Thread::run___get_multiple_idle_connections /opt/proxysql/lib/MySQL_Thread.cpp:2923 0.0 0.0% 100.0% 1.0 0.0% PtrArray::PtrArray (inline) /opt/proxysql/lib/../include/gen_utils.h:52 0.0 0.0% 100.0% 1.0 0.0% PtrArray::expand (inline) /opt/proxysql/lib/../include/gen_utils.h:28 0.0 0.0% 100.0% 11092.7 18.3% PtrSizeArray::add (inline) /opt/proxysql/lib/../include/gen_utils.h:139 0.0 0.0% 100.0% 21.5 0.0% PtrSizeArray::copy_add /opt/proxysql/lib/gen_utils.cpp:204 0.0 0.0% 100.0% 11092.7 18.3% PtrSizeArray::expand /opt/proxysql/lib/gen_utils.cpp:176 0.0 0.0% 100.0% 19.0 0.0% Query_Info::begin /opt/proxysql/lib/MySQL_Session.cpp:237 0.0 0.0% 100.0% 19.0 0.0% Query_Processor::query_parser_init /opt/proxysql/lib/Query_Processor.cpp:1891 0.0 0.0% 100.0% 3.0 0.0% Query_Processor_Output::operator new (inline) /opt/proxysql/lib/../include/query_processor.h:143 0.0 0.0% 100.0% 1.0 0.0% SetParser::parse1[abi:cxx11] /opt/proxysql/lib/set_parser.cpp:31 (discriminator 3) 0.0 0.0% 100.0% 3.0 0.0% SetParser::parse1[abi:cxx11] /opt/proxysql/lib/set_parser.cpp:44 (discriminator 2) 0.0 0.0% 100.0% 2.0 0.0% StatCounters::StatCounters (inline) /opt/proxysql/lib/../include/StatCounters.h:32 0.0 0.0% 100.0% 15.0 0.0% __strdup ??:? 0.0 0.0% 100.0% 12951.6 21.4% calloc /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2414 0.0 0.0% 100.0% 37002.1 61.2% clone ??:? 0.0 0.0% 100.0% 49791.4 82.3% imalloc (inline) /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2178 0.0 0.0% 100.0% 49791.4 82.3% imalloc_body (inline) /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2036 0.0 0.0% 100.0% 10453.9 17.3% irealloc_prof (inline) /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2457 0.0 0.0% 100.0% 36839.8 60.9% je_malloc_default /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2209 0.0 0.0% 100.0% 50.8 0.1% ma_alloc_root /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_alloc.c:90 0.0 0.0% 100.0% 22.0 0.0% ma_multi_malloc /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_alloc.c:180 0.0 0.0% 100.0% 10449.9 17.3% ma_net_read /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_net.c:425 0.0 0.0% 100.0% 10449.9 17.3% ma_net_safe_read /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:205 0.0 0.0% 100.0% 12949.6 21.4% ma_pvio_init /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_pvio.c:126 0.0 0.0% 100.0% 22.0 0.0% ma_read_ok_packet /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2438 0.0 0.0% 100.0% 3.0 0.0% ma_read_ok_packet /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2455 0.0 0.0% 100.0% 10449.9 17.3% ma_real_read /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_net.c:401 0.0 0.0% 100.0% 50.8 0.1% mthd_my_read_metadata_ex /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:1180 0.0 0.0% 100.0% 10449.9 17.3% mthd_my_read_one_row /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:1111 0.0 0.0% 100.0% 50.8 0.1% mthd_my_read_query_result /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2559 0.0 0.0% 100.0% 12949.6 21.4% mthd_my_real_connect /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:1611 0.0 0.0% 100.0% 18769.5 31.0% my_context_init /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_context.c:367 0.0 0.0% 100.0% 23488.4 38.8% my_context_spawn /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_context.c:201 0.0 0.0% 100.0% 2.0 0.0% mysql_change_user /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2079 0.0 0.0% 100.0% 11.0 0.0% mysql_change_user /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2080 0.0 0.0% 100.0% 25.0 0.0% mysql_change_user /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2084 0.0 0.0% 100.0% 38.0 0.1% mysql_change_user_start_internal /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_async.c:757 0.0 0.0% 100.0% 10449.9 17.3% mysql_fetch_row /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:2749 0.0 0.0% 100.0% 10449.9 17.3% mysql_fetch_row_start_internal /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_async.c:438 0.0 0.0% 100.0% 18769.5 31.0% mysql_optionsv /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:3208 0.0 0.0% 100.0% 19.0 0.0% mysql_query_digest_and_first_comment /opt/proxysql/lib/c_tokenizer.c:214 0.0 0.0% 100.0% 12949.6 21.4% mysql_real_connect /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_lib.c:1454 0.0 0.0% 100.0% 12949.6 21.4% mysql_real_connect_start_internal /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_async.c:332 0.0 0.0% 100.0% 50.8 0.1% mysql_real_query_start_internal /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/mariadb_async.c:381 0.0 0.0% 100.0% 36793.0 60.8% mysql_worker_thread_func /opt/proxysql/src/main.cpp:417 0.0 0.0% 100.0% 10449.9 17.3% net_realloc (inline) /opt/proxysql/deps/mariadb-client-library/mariadb_client/libmariadb/ma_net.c:132 0.0 0.0% 100.0% 60245.4 99.6% prof_alloc_prep (inline) /opt/proxysql/deps/jemalloc/jemalloc/include/jemalloc/internal/prof_inlines_b.h:157 0.0 0.0% 100.0% 1.0 0.0% re2::CharClassBuilder::GetCharClass /opt/proxysql/deps/re2/re2/re2/regexp.cc:970 0.0 0.0% 100.0% 1.0 0.0% re2::Compiler::Compile /opt/proxysql/deps/re2/re2/re2/compile.cc:1092 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::Compile /opt/proxysql/deps/re2/re2/re2/compile.cc:1136 0.0 0.0% 100.0% 2.0 0.0% re2::Compiler::Finish /opt/proxysql/deps/re2/re2/re2/compile.cc:1153 0.0 0.0% 100.0% 1.0 0.0% re2::FactorAlternationImpl::Round3 /opt/proxysql/deps/re2/re2/re2/parse.cc:1188 0.0 0.0% 100.0% 2.0 0.0% re2::PODArray::PODArray (inline) /opt/proxysql/deps/re2/re2/./re2/pod_array.h:22 0.0 0.0% 100.0% 1.0 0.0% re2::Prog::Flatten /opt/proxysql/deps/re2/re2/re2/prog.cc:569 0.0 0.0% 100.0% 1.0 0.0% re2::Prog::Flatten /opt/proxysql/deps/re2/re2/re2/prog.cc:630 0.0 0.0% 100.0% 1.0 0.0% re2::RE2::Init /opt/proxysql/deps/re2/re2/re2/re2.cc:199 0.0 0.0% 100.0% 3.0 0.0% re2::RE2::Init /opt/proxysql/deps/re2/re2/re2/re2.cc:223 0.0 0.0% 100.0% 4.0 0.0% re2::RE2::RE2 /opt/proxysql/deps/re2/re2/re2/re2.cc:126 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ConcatOrAlternate [clone .localalias] /opt/proxysql/deps/re2/re2/re2/regexp.cc:253 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::FactorAlternation /opt/proxysql/deps/re2/re2/re2/parse.cc:1000 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::Parse /opt/proxysql/deps/re2/re2/re2/parse.cc:2268 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ParseState::DoCollapse /opt/proxysql/deps/re2/re2/re2/parse.cc:1237 (discriminator 6) 0.0 0.0% 100.0% 1.0 0.0% re2::Regexp::ParseState::DoRightParen /opt/proxysql/deps/re2/re2/re2/parse.cc:678 0.0 0.0% 100.0% 1.0 0.0% re2::SparseArray::SparseArray (inline) /opt/proxysql/deps/re2/re2/./re2/sparse_array.h:370 0.0 0.0% 100.0% 10453.9 17.3% realloc /opt/proxysql/deps/jemalloc/jemalloc/src/jemalloc.c:2628 0.0 0.0% 100.0% 25.0 0.0% run_plugin_auth /opt/proxysql/deps/mariadb-client-library/mariadb_client/plugins/auth/my_auth.c:664 0.0 0.0% 100.0% 37002.1 61.2% start_thread ??:? 0.0 0.0% 100.0% 152.0 0.3% std::_Deque_base::_Deque_base (inline) /usr/include/c++/9/bits/stl_deque.h:510 0.0 0.0% 100.0% 9.0 0.0% std::_Deque_base::_M_allocate_map (inline) /usr/include/c++/9/bits/stl_deque.h:634 0.0 0.0% 100.0% 143.0 0.2% std::_Deque_base::_M_allocate_node (inline) /usr/include/c++/9/bits/stl_deque.h:620 0.0 0.0% 100.0% 143.0 0.2% std::_Deque_base::_M_create_nodes (inline) /usr/include/c++/9/bits/stl_deque.h:745 0.0 0.0% 100.0% 10.0 0.0% std::_Deque_base::_M_initialize_map (inline) /usr/include/c++/9/bits/stl_deque.h:719 0.0 0.0% 100.0% 9.0 0.0% std::_Deque_base::_M_initialize_map /usr/include/c++/9/bits/stl_deque.h:707 0.0 0.0% 100.0% 133.0 0.2% std::_Deque_base::_M_initialize_map /usr/include/c++/9/bits/stl_deque.h:719 0.0 0.0% 100.0% 152.0 0.3% std::allocator_traits::allocate (inline) /usr/include/c++/9/bits/alloc_traits.h:444 0.0 0.0% 100.0% 152.0 0.3% std::deque::deque (inline) /usr/include/c++/9/bits/stl_deque.h:915 0.0 0.0% 100.0% 209.0 0.3% std::error_code::default_error_condition ??:? 0.0 0.0% 100.0% 152.0 0.3% std::stack::stack (inline) /usr/include/c++/9/bits/stl_stack.h:157 ```

All jeprof dumps: jeprof.tar.gz

Thanks @renecannao, if you need anymore information please let me know.