search

crowell / modpagespeed_tmp

Automatically exported from code.google.com/p/modpagespeed
Apache License 2.0
0 stars 0 forks source link

shared-memory cache & memcached produce seg-fault when combined #747

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
I've installed mod_pagespeed 1.6.29.4-3289, the newest one. I've done a clean 
install and I've also done a clean install of Varnish. Then, I configured 
Varnish as described in your guidelines and I also configured MPS as described, 
together with memcached. It works great! System load 0.01 - 0.03 with 200 
hits/second.

But... when I enable ModPagespeedCreateSharedMemoryMetadataCache things go 
wrong: Apache crashes.

Apache error.log: 
[Sun Jul 14 15:31:47 2013] [notice] child pid 17075 exit signal Segmentation 
fault (11)

dmesg: 
[ 3754.765774] apache2[11110]: segfault at 8 ip 00007f6665412e66 sp 
00007f66556fe358 error 4 in libaprutil-1.so.0.3.9[7f6665409000+22000]
[ 6817.123737] apache2[17576] general protection ip:7f66651ee07c 
sp:7f6655d4f350 error:0 in libapr-1.so.0.4.2[7f66651d0000+38000]
[ 8024.609384] apache2[20921] general protection ip:7faab47f7ec1 
sp:7faaa72e8280 error:0 in libaprutil-1.so.0.3.9[7faab47ee000+22000]
[ 8036.949251] apache2[21049]: segfault at 42 ip 00007faab45d3488 sp 
00007faaaf531e20 error 4 in libapr-1.so.0.4.2[7faab45b5000+38000]

In mod_pagespeed_statistics everything seems fine, I see a lot of hits in the 
SharedMemoryMetadataCache. But Apache keeps crashing.

# uname -a
Linux ... 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux

# apache2 -v
Server version: Apache/2.2.16 (Debian)
Server built:   Mar  3 2013 12:10:30

# php -v
PHP 5.4.17-1~dotdeb.0 (cli) (built: Jul  6 2013 16:56:09) 
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
    with Zend OPcache v7.0.2, Copyright (c) 1999-2013, by Zend Technologies

Vanilla Debian Squeeze (later this year I'll upgrade to Wheezy). PHP via FPM. 
Memcache 1.4.5 on localhost. Varnish 3.0.4. All installed via aptitude/apt-get 
and up-to-date.

What should I do? I'm not a developer, and I'm not used to work with strace, 
but I really want to help you and to contribute, if that's needed.
Or maybe if I've done something wrong. Always possible. :)
Which information do you need? And how do I obtain that (for example, if you 
need a dump, how do I create that)?

For now, I disabled ModPagespeedCreateSharedMemoryMetadataCache and everything 
runs fine.
Thank you very much and keep up the good work.

Original issue reported on code.google.com by jmara...@google.com on 22 Jul 2013 at 2:35

GoogleCodeExporter commented 9 years ago 
Anupama: Thanks for trying out mod_pagespeed. Will it be possible for you to 
temporarily disable the Varnish-cache-integration (by deleting the 
"ModPagespeedModifyCachingHeaders off" line and removing all the 
ModPagespeedDownstreamCache* directives from your config), and tell us if 
re-enabling ModPagespeedCreateSharedMemoryMetadataCache continues to cause 
Apache crashes?

Original comment by jmara...@google.com on 22 Jul 2013 at 2:36

GoogleCodeExporter commented 9 years ago 
Hans:

Thanks for the quick reply and your help.

I disabled (commented out) the directives you mentioned. 
But again:
[Mon Jul 15 18:12:55 2013] [notice] child pid 7889 exit signal Segmentation 
fault (11)
Re-enabling ModPagespeedCreateSharedMemoryMetadataCache continues to cause 
Apache crashes.

By the way I have to keep this line disabled, commented out: 
ModPagespeedDownstreamCacheRewritingThresholdPercentage 95
When I enable it, I receive a syntax error while (re)starting Apache:

/etc/init.d/apache2 reload
Syntax error on line 44 of /etc/apache2/mods-enabled/pagespeed.conf:
Invalid command 'ModPagespeedDownstreamCacheRewritingThresholdPercentage', 
perhaps misspelled or defined by a module not included in the server 
configuration
Action 'configtest' failed.
The Apache error log may have more information.
 failed!

Original comment by jmara...@google.com on 22 Jul 2013 at 2:36

GoogleCodeExporter commented 9 years ago 
Maks:

FWIW: turning on 'ModPagespeedInstallCrashHandler on' might offer some
insight on what's going wrong, as it will
cause us to try to get a stack trace when crash occurs; though I am
not certain I'll be able to figure it out
from release binaries. Also, are there any messages mentioning shared
memory in the log?

Original comment by jmara...@google.com on 22 Jul 2013 at 2:37

GoogleCodeExporter commented 9 years ago 
Maks:

Thanks for the backtrace. Finally managed to decode it:

[Mon Jul 15 20:08:11 2013] [alert] [@21547] CRASH with signal:11 at Backtrace:
/usr/lib/apache2/modules/mod_pagespeed.so(+0xe231a)
  base::debug::StackTrace::StackTrace() + 26 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x1a3683)
  net_instaweb::StackTraceString() + 51 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0xc079e)
  signal_handler + 46 in section .text of /tmp/libmod_pagespeed.so
/lib/libc.so.6(+0x35f0c32230)
/usr/lib/libapr-1.so.0(+0x35f281da70)
/usr/lib/libapr-1.so.0(apr_allocator_alloc+0x23)
/usr/lib/libaprutil-1.so.0(apr_bucket_alloc+0x8c)
/usr/lib/libaprutil-1.so.0(+0xbd74)
/usr/lib/libaprutil-1.so.0(apr_brigade_split_line+0xc7)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3b15)
  get_server_line + 37 in section .text of /tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3ca1)
  sendv_and_get_server_line_with_timeout + 129 in section .text of
/tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b5a80)
  apr_memcache2_getp + 272 in section .text of /tmp/libmod_pagespeed.so
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b1017)
  net_instaweb::AprMemCache::Get(std::string const&,
net_instaweb::CacheInterface::Callback*)
  src/net/instaweb/system/apr_mem_cache.cc:199
/usr/lib/apache2/modules/mod_pagespeed.so(+0x2973b7)
  net_instaweb::RewriteContext::OutputCacheCallback::IsFilesystemMetadataCacheCurrent(net_instaweb::CacheInterface*,
std::string const&, net_instaweb::InputInfo const&, long)
  src/net/instaweb/rewriter/rewrite_context.cc:299
/usr/lib/apache2/modules/mod_pagespeed.so(+0x298452)
  net_instaweb::RewriteContext::OutputCacheCallback::IsCachedResultValid(net_instaweb::CachedResult*,
bool*, std::vector<net_instaweb::InputInfo*,
std::allocator<net_instaweb::InputInfo*> >*)
  src/net/instaweb/rewriter/rewrite_context.cc:441
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e0e1d)
  net_instaweb::WriteThroughCallback::ValidateCandidate(std::string
const&, net_instaweb::CacheInterface::KeyState)
  src/net/instaweb/util/write_through_cache.cc:55
/usr/lib/apache2/modules/mod_pagespeed.so(+0x2fc1f1)
  net_instaweb::DelegatingCacheCallback::ValidateCandidate(std::string
const&, net_instaweb::CacheInterface::KeyState)
  src/net/instaweb/util/delegating_cache_callback.cc:46
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31c05c)
  net_instaweb::CacheInterface::ValidateAndReportResult(std::string
const&, net_instaweb::CacheInterface::KeyState,
net_instaweb::CacheInterface::Callback*)
  src/pagespeed/kernel/base/cache_interface.cc:39
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e587f)
  net_instaweb::SharedMemCache<64ul>::GetFromEntry(std::string const&,
net_instaweb::SharedMemCacheData::Sector<64ul>*, int,
net_instaweb::CacheInterface::Callback*) + 351 in section .text of
/tmp/libmod_pagespeed.so
  (inline? ~_Vector_base)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4e59ed)
  net_instaweb::SharedMemCache<64ul>::Get(std::string const&,
net_instaweb::CacheInterface::Callback*) + 301 in section .text of
/tmp/libmod_pagespeed.so
  (inline? std::string::_M_rep)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x30afe6)
  net_instaweb::CacheStats::Get(std::string const&,
net_instaweb::CacheInterface::Callback*)
  src/net/instaweb/util/cache_stats.cc:142
/usr/lib/apache2/modules/mod_pagespeed.so(+0x28b7c8)
  net_instaweb::RewriteContext::Start()
  src/net/instaweb/rewriter/rewrite_context.cc:1391
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2)
  net_instaweb::Function::CallRun()
  src/pagespeed/kernel/base/function.cc:52
/usr/lib/apache2/modules/mod_pagespeed.so(+0x33364c)
  net_instaweb::QueuedWorkerPool::Run(net_instaweb::QueuedWorkerPool::Sequence*,
net_instaweb::QueuedWorker*)
  src/pagespeed/kernel/thread/queued_worker_pool.cc:155
/usr/lib/apache2/modules/mod_pagespeed.so(+0x31caa2)
  net_instaweb::Function::CallRun()
  src/pagespeed/kernel/base/function.cc:52
/usr/lib/apache2/modules/mod_pagespeed.so(+0x33733c)
  net_instaweb::Worker::WorkThread::Run()
  src/pagespeed/kernel/thread/worker.cc:84
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b6c58)
  net_instaweb::PthreadThreadImpl::InvokeRun(void*)
  src/pagespeed/kernel/thread/pthread_thread_system.cc:124
/lib/libpthread.so.0(+0x35efe068ca)
/lib/libc.so.6(clone+0x6d) [0x7ffa1519eb6d]

Original comment by jmara...@google.com on 22 Jul 2013 at 2:37

GoogleCodeExporter commented 9 years ago 
Maks:

And the C frames:

/usr/lib/libaprutil-1.so.0(apr_brigade_split_line+0xc7)
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3b15)
  get_server_line + 37 in section .text of /tmp/libmod_pagespeed.so
  src/third_party/aprutil/apr_memcache2.c:679
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b3ca1)
  sendv_and_get_server_line_with_timeout + 129 in section .text of
/tmp/libmod_pagespeed.so
  src/third_party/aprutil/apr_memcache2.c:715
/usr/lib/apache2/modules/mod_pagespeed.so(+0x4b5a80)
  apr_memcache2_getp + 272 in section .text of /tmp/libmod_pagespeed.so
  src/third_party/aprutil/apr_memcache2.c:896

Original comment by jmara...@google.com on 22 Jul 2013 at 2:37

GoogleCodeExporter commented 9 years ago 
I just installed the new binary release "1.6.29.5-r3346", it didn't fix it for 
me.

Original comment by hanshe...@gmail.com on 25 Jul 2013 at 9:12