search

redis / redis

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
http://redis.io
Other
65.7k stars 23.6k forks source link

Redis Crash #2749

Closed smagnaschi closed 8 years ago

smagnaschi commented 8 years ago

Bug report, this happens regularly onour server but I can't seem to find the cause. Any clue?

=== REDIS BUG REPORT START: Cut & paste starting from here === 32584:M 31 Aug 14:06:37.389 # ------------------------------------------------ 32584:M 31 Aug 14:06:37.389 # !!! Software Failure. Press left mouse button to continue 32584:M 31 Aug 14:06:37.389 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3551 32584:M 31 Aug 14:06:37.389 # (forcing SIGSEGV in order to print the stack trace) 32584:M 31 Aug 14:06:37.389 # ------------------------------------------------ 32584:M 31 Aug 14:06:37.395 # Redis 3.0.2 crashed by signal: 11 32584:M 31 Aug 14:06:37.395 # Failed assertion: (:0) 32584:M 31 Aug 14:06:37.395 # --- STACK TRACE /usr/bin/redis-server :6379(logStackTrace+0x60)[0xb76ce3a0] /usr/bin/redis-server :6379(_redisPanic+0x95)[0xb76cce75] [0xb765040c] /usr/bin/redis-server :6379(_redisPanic+0x95)[0xb76cce75] /usr/bin/redis-server :6379(redisOutOfMemoryHandler+0x4d)[0xb768edfd] /usr/bin/redis-server :6379(zrealloc+0x101)[0xb76986c1] /usr/bin/redis-server :6379(sdsMakeRoomFor+0x4c)[0xb7696d5c] /usr/bin/redis-server :6379(processMultibulkBuffer+0x2fe)[0xb769f61e] /usr/bin/redis-server :6379(processInputBuffer+0x92)[0xb76a0d42] /usr/bin/redis-server :6379(readQueryFromClient+0xe7)[0xb76a0e87] /usr/bin/redis-server :6379(aeProcessEvents+0x1fd)[0xb768b3cd] /usr/bin/redis-server :6379(aeMain+0x2c)[0xb768b6fc] /usr/bin/redis-server :6379(main+0x3d5)[0xb768a175] /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74b3e46] /usr/bin/redis-server *:6379(+0x1b3b1)[0xb768a3b1] 32584:M 31 Aug 14:06:37.407 # --- INFO OUTPUT 32584:M 31 Aug 14:06:37.416 # # Server redis_version:3.0.2 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:2db86889f92bd0fa redis_mode:standalone os:Linux 3.2.0-4-686-pae i686 arch_bits:32 multiplexing_api:epoll gcc_version:4.7.2 process_id:32584 run_id:65e6ca8d495ba8cffe988d57b31239bdcc81269b tcp_port:6379 uptime_in_seconds:772318 uptime_in_days:8 hz:10 lru_clock:14959437 config_file:/etc/redis/redis.conf

Clients

connected_clients:12 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0

Memory

used_memory:1487063088 used_memory_human:1.38G used_memory_rss:1626689536 used_memory_peak:2272810400 used_memory_peak_human:2.12G used_memory_lua:24576 mem_fragmentation_ratio:1.09 mem_allocator:jemalloc-3.6.0

Persistence

loading:0 rdb_changes_since_last_save:8963 rdb_bgsave_in_progress:0 rdb_last_save_time:1441022608 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:11 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok

Stats

total_connections_received:7300891 total_commands_processed:267955358 instantaneous_ops_per_sec:594 total_net_input_bytes:130524123081 total_net_output_bytes:1796574366933 instantaneous_input_kbps:232.33 instantaneous_output_kbps:4009.83 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:10178044 evicted_keys:0 keyspace_hits:107688489 keyspace_misses:23 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:87400 migrate_cached_sockets:0

Replication

role:master connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:12127.05 used_cpu_user:10395.57 used_cpu_sys_children:4129.40 used_cpu_user_children:30765.23

Commandstats

cmdstat_get:calls=93444692,usec=652978726,usec_per_call=6.99 cmdstat_set:calls=14636884,usec=205756348,usec_per_call=14.06 cmdstat_del:calls=175035,usec=7779289,usec_per_call=44.44 cmdstat_exists:calls=140168249,usec=732277517,usec_per_call=5.22 cmdstat_incr:calls=5276354,usec=31613925,usec_per_call=5.99 cmdstat_expire:calls=14243820,usec=134739722,usec_per_call=9.46 cmdstat_keys:calls=10301,usec=3739005549,usec_per_call=362975.00 cmdstat_info:calls=23,usec=11749,usec_per_call=510.83

Cluster

cluster_enabled:0

Keyspace

db0:keys=685509,expires=685230,avg_ttl=172882796 hash_init_value: 1440576902

32584:M 31 Aug 14:06:37.416 # --- CLIENT LIST OUTPUT 32584:M 31 Aug 14:06:37.416 # id=7300822 addr=127.0.0.1:44612 fd=13 name= age=6 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists id=7300855 addr=127.0.0.1:44778 fd=12 name= age=4 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 ob l=0 oll=0 omem=0 events=r cmd=exists id=7300873 addr=127.0.0.1:44967 fd=19 name= age=2 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 ob l=0 oll=0 omem=0 events=r cmd=exists id=7300881 addr=127.0.0.1:45012 fd=8 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl =0 oll=0 omem=0 events=r cmd=expire id=7300885 addr=127.0.0.1:45038 fd=7 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire id=7300886 addr=127.0.0.1:45041 fd=20 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 ob l=0 oll=0 omem=0 events=r cmd=get id=7300887 addr=127.0.0.1:45047 fd=9 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl =0 oll=0 omem=0 events=r cmd=expire id=7300888 addr=127.0.0.1:45050 fd=16 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 ob l=0 oll=0 omem=0 events=r cmd=expire id=7300889 addr=127.0.0.1:45057 fd=10 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 ob l=0 oll=0 omem=0 events=r cmd=exists id=7300890 addr=127.0.0.1:45059 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=3276 8 obl=0 oll=0 omem=0 events=r cmd=incr id=7300891 addr=127.0.0.1:45064 fd=15 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=3276 8 obl=0 oll=0 omem=0 events=r cmd=incr id=7300892 addr=127.0.0.1:45065 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=1418 28 obl=0 oll=0 omem=0 events=r cmd=incr

32584:M 31 Aug 14:06:37.416 # --- CURRENT CLIENT INFO 32584:M 31 Aug 14:06:37.416 # client: id=7300855 addr=127.0.0.1:44778 fd=12 name= age=4 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists 32584:M 31 Aug 14:06:37.416 # argv[0]: 'SET' 32584:M 31 Aug 14:06:37.416 # argv[1]: 'MXM:LYRICS:track_search:8067fd3ff6087d19df58c78051597a18' 32584:M 31 Aug 14:06:37.528 # argv[2]: 'a:2:{s:6:"result";a:10000:{i:0;a:1:{s:5:"track";a:17:{s:8:"track_id";i:74 760523;s:10:"track_name";s:77:"Forever and Always ((Originally Performed By Taylor Swift) [Karaoke Version])";s:1 2:"track_length";i:232;s:12:"instrumental";i:0;s:10:"has_lyrics";i:1;s:13:"has_subtitles";i:1;s:13:"num_favourite ";i:195;s:8:"album_id";i:19753796;s:10:"album_name";s:44:"Best for Musicians No. 796 (Karaoke Version)";s:9:"arti st_id";i:259675;s:11:"artist_name";s:12:"Taylor Swift";s:22:"album_coverart_100x100";s:66:"http://s.mxmcdn.net/im ages-storage/albums/5/1/9/6/2/9/30926915.jpg";s:22:"album_coverart_350x350";s:74:"http://s.mxmcdn.net/images-stor age/albums/5/1/9/6/2/9/30926915_350_350.jpg";s:22:"album_coverart_500x500";s:74:"http://s.mxmcdn.net/images-stora ge/albums/5/1/9/6/2/9/30926915_500_500.jpg";s:15:"track_share_url";s:118:"https://www.musixmatch.com/lyrics/Taylo r-Swift/Forever-and-Always-Originally-Performed-By-Taylor-Swift-Karaoke-Version";s:21:"commontrack_vanity_id";s:8 4:"Taylor-Swift/Forever-and-Always-O 32584:M 31 Aug 14:06:37.528 # --- REGISTERS 32584:M 31 Aug 14:06:37.528 # EAX:00000000 EBX:b7778ac0 ECX:b75fe3c0 EDX:b75fe3ec EDI:00af934f ESI:b774af5c EBP:00000010 ESP:bfe83c10 SS :0000007b EFL:bfe83c10 EIP:b76cce75 CS:00000073 DS :0000007b ES :0000007b FS :00000000 GS:00000033 32584:M 31 Aug 14:06:37.528 # (bfe83c1f) -> b7778ac0 32584:M 31 Aug 14:06:37.528 # (bfe83c1e) -> 00000ddf 32584:M 31 Aug 14:06:37.528 # (bfe83c1d) -> b7752023 32584:M 31 Aug 14:06:37.528 # (bfe83c1c) -> b7742d80 32584:M 31 Aug 14:06:37.528 # (bfe83c1b) -> b768edfd 32584:M 31 Aug 14:06:37.528 # (bfe83c1a) -> 00000000 32584:M 31 Aug 14:06:37.528 # (bfe83c19) -> b7778ac0 32584:M 31 Aug 14:06:37.528 # (bfe83c18) -> b76ccde7 32584:M 31 Aug 14:06:37.528 # (bfe83c17) -> bb3d6900 32584:M 31 Aug 14:06:37.528 # (bfe83c16) -> 8b519200 32584:M 31 Aug 14:06:37.528 # (bfe83c15) -> b7778ac0 32584:M 31 Aug 14:06:37.528 # (bfe83c14) -> 00000ddf 32584:M 31 Aug 14:06:37.528 # (bfe83c13) -> b7752023 32584:M 31 Aug 14:06:37.528 # (bfe83c12) -> b7742d80 32584:M 31 Aug 14:06:37.528 # (bfe83c11) -> b774af5c 32584:M 31 Aug 14:06:37.528 # (bfe83c10) -> 00000003 32584:M 31 Aug 14:06:37.528 # --- FAST MEMORY TEST 32584:M 31 Aug 14:06:37.529 # Bio thread for job type #0 terminated 32584:M 31 Aug 14:06:37.529 # Bio thread for job type #1 terminated 32584:M 31 Aug 14:07:34.245 # Fast memory test PASSED, however your memory can still be broken. Please run a memo ry test for several hours if possible. 32584:M 31 Aug 14:07:34.245 # === REDIS BUG REPORT END. Make sure to include from START to END. ===

sunheehnus commented 8 years ago

Looks like a normal crash for out of memory

antirez commented 8 years ago

Yep, closing.