dmwm / PHEDEX

CMS data-placement suite
8 stars 18 forks source link

Reduce BlockAllocator memory usage #1039

Open nikmagini opened 7 years ago

nikmagini commented 7 years ago

The BlockAllocator agent is now regularly using 2.8g RSS RAM and sometimes more. It is often crashing after completing a cycle with messages like this:

2016-07-26 15:01:04: BlockAllocator[31705]: getconf PAGESIZE: Cannot allocate memory

Or during a cycle with memory corruption errors:

* glibc detected * perl: double free or corruption (fasttop): 0x0000000044654100 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x75f3e)[0x7f029fe67f3e] /lib64/libc.so.6(+0x78d8d)[0x7f029fe6ad8d] /data/ProdNodes/sw/slc6_amd64_gcc493/external/p5-dbd-oracle/1.23-comp/lib/perl5/x86_64-linux-thread-multi/auto/DBD/Oracle/Oracle.so(ora_st_execute_array+0xd03)[0x7f0298881963] [...]

Should look into reducing its memory footprint - if not, we'll need to consider migrating the Prod Mgmt agents to a vobox with more RAM.