processone / ejabberd

Robust, Ubiquitous and Massively Scalable Messaging Platform (XMPP, MQTT, SIP Server)
https://www.process-one.net/en/ejabberd/
Other
6.03k stars 1.5k forks source link

Mnesia overloaded #4028

Closed lokesh411 closed 1 year ago

lokesh411 commented 1 year ago

Before creating a ticket, please consider if this should fit the discussion forum better.

Environment

Errors from error.log/crash.log

2023-04-25 05:04:30.032860+00:00 [warning] <0.577.0>@ejabberd_s2s:handle_info/2:267 Unexpected info: {mnesia_system_event,
                     {mnesia_overload,{dump_log,write_threshold}}}
2023-04-25 05:04:30.032855+00:00 [warning] <0.573.0>@ejabberd_sm_mnesia:handle_info/2:129 Unexpected info: {mnesia_system_event,
                     {mnesia_overload,{dump_log,write_threshold}}}
2023-04-25 05:04:30.058938+00:00 [warning] <0.577.0>@ejabberd_s2s:handle_info/2:267 Unexpected info: {mnesia_system_event,
                     {mnesia_overload,{dump_log,write_threshold}}}
2023-04-25 05:04:30.058938+00:00 [warning] <0.573.0>@ejabberd_sm_mnesia:handle_info/2:129 Unexpected info: {mnesia_system_event,
                     {mnesia_overload,{dump_log,write_threshold}}}
2023-04-25 05:04:30.058949+00:00 [warning] <0.406.0> Mnesia('ejabberd@10.x.x.x'): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}

Bug description

Would want to understand the root cause of mnesia getting overloaded. Whenever this happens in our production servers, we see these logs in ejabberd, we see connections getting dropped and systems are slowing down a bit. Can you please help me in understanding this Thanks in advance

licaon-kter commented 1 year ago

How big is your Mnesia?

You are not using SQL for messages?

iirc, Mnesia should not be used if your message DB exceeds 2Gb

lokesh411 commented 1 year ago

@licaon-kter We have disabled message archive management in our Ejabberd servers, so it won't store any messages. Did i answer your question?

licaon-kter commented 1 year ago

So what's using it? How big is it?

lokesh411 commented 1 year ago

Only sessions are using mnesia mostly as far as i can see

tudoradridan commented 1 year ago

@licaon-kter We have disabled message archive management in our Ejabberd servers, so it won't store any messages. Did i answer your question?

Can you please tell us how did you disabled message archive management? We are trying to do disable this, to leave it only for mod_muc, but the following configuration is not enough:

mod_mam:
  access_preferences: none
  assume_mam_usage: false
  default: never

The messages are still written in archive_msg table. Did you check the size of the archive_msg table in your deployment?

Thanks!

lokesh411 commented 1 year ago

I have removed mod_mam from the table itself, so it shouldn't store any messages ideally. And archive_msg doesn't exists in my prod ejabberd servers

mremond commented 1 year ago

Closing for now, as this is something highly dependant on the setup and use case.