search

sympa-community / sympa

Sympa, Mailing List Management Software
https://www.sympa.community/sympa
GNU General Public License v2.0
237 stars 94 forks source link

SOAP server sometimes dies quietly after ADD or DEL is performed #1857

Open adam12b1 opened 1 week ago

adam12b1 commented 1 week ago

Version

Sympa 6.2.72

Installation method

FreeBSD port/package

Expected behavior

sympa_soap_server.fcgi should remain running

Actual behavior

Once every week or two, our SOAP process just goes away, always at the end of a series of ADD or DEL commands. It logs no errors and reports nothing, just stops running, with the final log lines always like these (email addresses and names all redacted):

Jun 20 04:55:04 npogroups sympasoap[7519]: notice Sympa::WWW::SOAP::authenticateAndRun(REDACTED, 96402334561522, add, kanpnm,REDACTED,REDACTED,1)
Jun 20 04:55:04 npogroups sympasoap[7519]: info Sympa::WWW::SOAP::add(list = REDACTED, email = REDACTED, quiet = 1) From REDACTED via proxy application
Jun 20 04:55:04 npogroups sympasoap[7519]: notice Sympa::Spindle::ProcessRequest::_twist() Processing Sympa::Request <action=add;context=REDACTED;email=REDACTED>

But there is always a long series of ADDs or DELs that succeeds before the process dies, and sometimes the whole series completes fine without any problem. And even the final command succeeds, before the process dies - the user is successfully added or removed.

Steps to reproduce

I am sorry to say, we are unable to reproduce the problem in testing conditions - it only happens (occasionally) on our live production server. I also added some debug logging in Sympa/WWW/SOAP.pm, in the add() and del() routines, immediately after the my $spindle = Sympa::Spindle::ProcessRequest->new() call... but sadly, when the process dies it happens before the code returns from that call. So it never reaches my added logging.

I wish I could offer more details here, but my attempts to gather them have failed. I would be happy to try additional measures if you have any suggestions.

racke commented 1 week ago

Make sure that you have enough RAM on your server. Also please describe your Sympa SOAP setup on your server.

adam12b1 commented 1 week ago

@racke thank you for the quick reply. our server has piles of RAM - 512GB installed, most of which is always free, and we monitor constantly for RAM and swap usage and have seen no spikes. current stats from top here:

last pid: 25652;  load averages:  0.52,  0.56,  0.60   up 14+12:09:50  10:07:51
257 processes: 1 running, 255 sleeping, 1 zombie
CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
Mem: 1339M Active, 2705M Inact, 59G Wired, 104K Buf, 435G Free
ARC: 37G Total, 21G MFU, 4305M MRU, 640K Anon, 721M Header, 12G Other
     19G Compressed, 149G Uncompressed, 7.75:1 Ratio
Swap: 32G Total, 32G Free

we have just this line in our sympa.conf (and similar in all our robot.conf):

soap_url            https://groups.electricembers.net/sympasoap

and only the default sympa.wsdl under /usr/local/share/sympa/defaults/.