search

RedisTimeSeries / redistimeseries-py

RedisTimeSeries python client
https://redistimeseries.io
BSD 3-Clause "New" or "Revised" License
99 stars 21 forks source link

Redis 5.0.3 crashed by signal: 11 (Redistimeseries.so with compaction policy set) #79

Closed udayshankarv closed 3 years ago

udayshankarv commented 3 years ago

Redis frequently crashing when compaction policy is enabled.

loadmodule /opt/redis/redistimeseries.so DUPLICATE_POLICY Last COMPACTION_POLICY sum:300s:90d;count:300s:90d;sum:1800s:180d;count:1800s:180d

When the module is loaded without COMPACTION_POLICY, It is a seamless.

=== REDIS BUG REPORT START: Cut & paste starting from here === 14861:M 29 Oct 2020 07:45:30.469 # Redis 5.0.3 crashed by signal: 11 14861:M 29 Oct 2020 07:45:30.469 # Crashed running the instruction at: 0x7fa241f7cb05 14861:M 29 Oct 2020 07:45:30.469 # Accessing address: (nil) 14861:M 29 Oct 2020 07:45:30.469 # Failed assertion: (:0)

------ STACK TRACE ------ EIP: /opt/redis/redistimeseries.so(Uncompressed_UpsertSample+0x125)[0x7fa241f7cb05]

Backtrace: /usr/bin/redis-server 0.0.0.0:6379(logStackTrace+0x44)[0x55df2b955c54] /usr/bin/redis-server 0.0.0.0:6379(sigsegvHandler+0xb5)[0x55df2b9563d5] /lib64/libpthread.so.0(+0x12dd0)[0x7fa2413e5dd0] /opt/redis/redistimeseries.so(Uncompressed_UpsertSample+0x125)[0x7fa241f7cb05] /opt/redis/redistimeseries.so(SeriesUpsertSample+0xda)[0x7fa241f85ada] /opt/redis/redistimeseries.so(SeriesUpsertSample+0x3a8)[0x7fa241f85da8] /opt/redis/redistimeseries.so(+0xcad2)[0x7fa241f80ad2] /opt/redis/redistimeseries.so(TSDB_add+0x162)[0x7fa241f82d42] /usr/bin/redis-server 0.0.0.0:6379(RedisModuleCommandDispatcher+0x66)[0x55df2b983946] /usr/bin/redis-server 0.0.0.0:6379(call+0x93)[0x55df2b90e853] /usr/bin/redis-server 0.0.0.0:6379(processCommand+0x49e)[0x55df2b90fcfe] /usr/bin/redis-server 0.0.0.0:6379(processInputBuffer+0x181)[0x55df2b91fb01] /usr/bin/redis-server 0.0.0.0:6379(aeProcessEvents+0x119)[0x55df2b9081f9] /usr/bin/redis-server 0.0.0.0:6379(aeMain+0x2b)[0x55df2b90865b] /usr/bin/redis-server 0.0.0.0:6379(main+0x3de)[0x55df2b90523e] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7fa2410346a3] /usr/bin/redis-server 0.0.0.0:6379(_start+0x2e)[0x55df2b90558e]

------ INFO OUTPUT ------

Server

redis_version:5.0.3 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:28849dbea6f07cc8 redis_mode:standalone os:Linux 4.18.0-80.el8.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:8.3.1 process_id:14861 run_id:4dfba04838528974a4243b5231d8363e1c62a677 tcp_port:6379 uptime_in_seconds:1 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:10136410 executable:/usr/bin/redis-server config_file:/etc/redis.conf

Clients

connected_clients:1 client_recent_max_input_buffer:0 client_recent_max_output_buffer:0 blocked_clients:0

Memory

used_memory:30728624 used_memory_human:29.31M used_memory_rss:36397056 used_memory_rss_human:34.71M used_memory_peak:30728624 used_memory_peak_human:29.31M used_memory_peak_perc:100.56% used_memory_overhead:1120656 used_memory_startup:791304 used_memory_dataset:29607968 used_memory_dataset_perc:98.90% allocator_allocated:30647824 allocator_active:30879744 allocator_resident:36335616 total_system_memory:8193073152 total_system_memory_human:7.63G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.01 allocator_frag_bytes:231920 allocator_rss_ratio:1.18 allocator_rss_bytes:5455872 rss_overhead_ratio:1.00 rss_overhead_bytes:61440 mem_fragmentation_ratio:1.19 mem_fragmentation_bytes:5838216 mem_not_counted_for_evict:522 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:49694 mem_aof_buffer:522 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:12046 rdb_bgsave_in_progress:0 rdb_last_save_time:1603971929 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 aof_current_size:2840711 aof_base_size:2839964 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0

Stats

total_connections_received:1 total_commands_processed:4 instantaneous_ops_per_sec:0 total_net_input_bytes:969 total_net_output_bytes:48 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:9 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:master connected_slaves:0 master_replid:7cd9b1523a1994fde5ed6227465506d441c0a7bd master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.021736 used_cpu_user:0.213450 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000

Commandstats

cmdstat_info:calls=1,usec=18,usec_per_call=18.00 cmdstat_ts.add:calls=3,usec=494,usec_per_call=164.67

Cluster

cluster_enabled:0

Keyspace

db0:keys=5340,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=10913 addr=172.17.21.2:59224 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=245 qbuf-free=32523 obl=0 oll=0 omem=0 events=r cmd=ts.add

------ CURRENT CLIENT INFO ------ id=10913 addr=172.17.21.2:59224 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=245 qbuf-free=32523 obl=0 oll=0 omem=0 events=r cmd=ts.add argv[0]: 'TS.ADD' argv[1]: 'billing:2088' argv[2]: '1602565827000' argv[3]: '1' argv[4]: 'RETENTION' argv[5]: '86400000' argv[6]: 'LABELS' argv[7]: 'agentname' argv[8]: 'Laura Jackson' argv[9]: 'agentskillname' argv[10]: 'SK1589' argv[11]: 'region' argv[12]: 'N. Virginia' argv[13]: 'server' argv[14]: 'Server1' argv[15]: 'shift' argv[16]: '1' 14861:M 29 Oct 2020 07:45:30.470 # key 'billing:2088' found in DB containing the following object: 14861:M 29 Oct 2020 07:45:30.470 # Object type: 5 14861:M 29 Oct 2020 07:45:30.470 # Object encoding: 0 14861:M 29 Oct 2020 07:45:30.470 # Object refcount: 1

------ REGISTERS ------ 14861:M 29 Oct 2020 07:45:30.470 # RAX:00007fa23d660380 RBX:0000000000000000 RCX:00007fa240000000 RDX:0000000000000002 RDI:00007fffc394abf0 RSI:00007fffc394abcc RBP:00007fa23d660380 RSP:00007fffc394ab70 R8 :00000175205d8040 R9 :0000000000000000 R10:00007fa241f8f838 R11:00007fa24119df20 R12:0000000000000002 R13:00000175205d8040 R14:ffffffffffffffff R15:00007fffc394ac88 RIP:00007fa241f7cb05 EFL:0000000000010246 CSGSFS:002b000000000033 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7f) -> 000055df2b982105 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7e) -> 00007fa23d75e150 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7d) -> 000055df2c5746b0 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7c) -> 00007fa23d660380 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7b) -> 000000002b915e7a 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab7a) -> 0000000000000078 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab79) -> 00007fa240c16100 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab78) -> 0000000000000001 14861:M 29 Oct 2020 07:45:30.470 # (00007fffc394ab77) -> 3ff0000000000000 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab76) -> 0000000000000078 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab75) -> 00007fa241f85ada 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab74) -> ffffffffffffffff 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab73) -> 00000175205d8040 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab72) -> 0000000000000002 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab71) -> 00007fa241f8e780 14861:M 29 Oct 2020 07:45:30.471 # (00007fffc394ab70) -> 000055df2c5747f0

------ FAST MEMORY TEST ------ 14861:M 29 Oct 2020 07:45:30.471 # Bio thread for job type #0 terminated 14861:M 29 Oct 2020 07:45:30.471 # Bio thread for job type #1 terminated 14861:M 29 Oct 2020 07:45:30.471 # Bio thread for job type #2 terminated Preparing to test memory region 55df2bc82000 (2252800 bytes) Preparing to test memory region 55df2c496000 (1622016 bytes) Preparing to test memory region 7fa23c400000 (2097152 bytes) Preparing to test memory region 7fa23c780000 (36175872 bytes) Preparing to test memory region 7fa23eb05000 (2621440 bytes) Preparing to test memory region 7fa23ed86000 (8388608 bytes) Preparing to test memory region 7fa23f587000 (8388608 bytes) Preparing to test memory region 7fa23fd88000 (8388608 bytes) Preparing to test memory region 7fa240800000 (8388608 bytes) Preparing to test memory region 7fa2413cf000 (16384 bytes) Preparing to test memory region 7fa2415ef000 (16384 bytes) Preparing to test memory region 7fa241f8f000 (32768 bytes) *** Preparing to test memory region 7fa241fac000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: Uncompressed_UpsertSample (base: 0x7fa241f7c9e0) Module: /opt/redis/redistimeseries.so (base 0x7fa241f74000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x7fa241f7c9e0 -D -b binary -m i386:x86-64 /tmp/dump.bin

14861:M 29 Oct 2020 07:45:30.765 # dump of function (hexdump of 421 bytes): 4156415541545553488b6f10c706000000004c8b07448b4d10490fbfd94885db0f84ff000000488b450831d2488b084939c87720e9f60000000f1f80000000004863ca48c1e104488b0c084c39c10f837c0000004883c2014839da75e34989dd49c1e5044c39c174764989f6488b75184489ca4989fc4889f148c1e9044839ca751e4889c7488b05741101004883c61048897518ff108b5510488945084989d14a8d34284839da7757f3410f6f0c244183c10131c00f110e44894d1041c706010000005b5d415c415d415ec30f1f40004989d54889d349c1e5044c39c1758a4c01e8f20f104708f20f11400831c05b5d415c415d415ec3660f1f840000000000488d7b014829da48c1e70448c1e204488d3c38e8d8faffff488b7508448b4d104c01eeeb84488b0425000000000f0b74b14c8945004531ed31dbe92affffff9041554989fdbf0100000041544989d4555389f3be200000004883ec08488b052d100100ff104889c54c892889581c83e3017407418b5d1083eb01895d084d85e4741931ffe867f9fffff30f6f00410f110424488b401049894424104883c4084889e85b5d41

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

danni-m commented 3 years ago

@udayshankarv can you open it on https://github.com/RedisTimeSeries/RedisTimeSeries ? this repo is used only for the python client.

danni-m commented 3 years ago

Also, @udayshankarv do you know which version are you using?

udayshankarv commented 3 years ago

thank you @danni-m , I'm using

Redis server v=5.0.3

Name: redistimeseries Version: 1.4.3 Summary: RedisTimeSeries Python Client