Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.02k stars 564 forks source link

[CRASH] when trying to install redisgraph #847

Open Lion5324 opened 5 days ago

Lion5324 commented 5 days ago

Crash report

2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.320 # Server initialized
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.323 * <graph> Enabled role change notification
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.323 * <graph> Starting up RedisGraph version 99.99.99.
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.324 * <graph> Thread pool created, using 12 threads.
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.324 * <graph> Maximum number of OpenMP threads set to 12
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.324 * <graph> Query backlog size: 1000
2024-06-29 23:49:04 1:1:M 29 Jun 2024 20:49:04.324 * Module 'graph' loaded from /opt/keydb/modules/redisgraph.so
2024-06-29 23:49:04 
2024-06-29 23:49:04 
2024-06-29 23:49:04 === KEYDB BUG REPORT START: Cut & paste starting from here ===
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.324 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.324 # Accessing address: 0x2a8
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.324 # Crashed running the instruction at: 0x5589b62007eb
2024-06-29 23:49:04 1:36:M 29 Jun 2024 20:49:04.324 * Thread 0 alive.
2024-06-29 23:49:04 1:37:M 29 Jun 2024 20:49:04.324 * Thread 1 alive.
2024-06-29 23:49:04 
2024-06-29 23:49:04 ------ STACK TRACE ------
2024-06-29 23:49:04 EIP:
2024-06-29 23:49:04 keydb-server *:6379(unlinkClient(client*)+0x2fb) [0x5589b62007eb]
2024-06-29 23:49:04 
2024-06-29 23:49:04 Backtrace:
2024-06-29 23:49:04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fc5861f9420]
2024-06-29 23:49:04 keydb-server *:6379(unlinkClient(client*)+0x2fb) [0x5589b62007eb]
2024-06-29 23:49:04 keydb-server *:6379(freeClient(client*)+0x29e) [0x5589b6203c0e]
2024-06-29 23:49:04 keydb-server *:6379(RM_FreeThreadSafeContext(RedisModuleCtx*)+0x1e) [0x5589b616063e]
2024-06-29 23:49:04 /opt/keydb/modules/redisgraph.so(CronTask_streamFinishedQueries+0x3d5) [0x7fc581c0e3a5]
2024-06-29 23:49:04 /opt/keydb/modules/redisgraph.so(CronTask_RecurringTask+0x26) [0x7fc581c0dc16]
2024-06-29 23:49:04 /opt/keydb/modules/redisgraph.so(+0x240775) [0x7fc581c0d775]
2024-06-29 23:49:04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fc5861ed609]
2024-06-29 23:49:04 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fc586112353]
2024-06-29 23:49:04 
2024-06-29 23:49:04 ------ REGISTERS ------
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # 
2024-06-29 23:49:04 RAX:0000000000000000 RBX:0000000000000000
2024-06-29 23:49:04 RCX:0000000000000000 RDX:0000000000000000
2024-06-29 23:49:04 RDI:00007fc583809340 RSI:00007fc583809340
2024-06-29 23:49:04 RBP:00007fc583809340 RSP:00007fc5811c8da0
2024-06-29 23:49:04 R8 :00007fc584e00900 R9 :0000000000000030
2024-06-29 23:49:04 R10:0000000000000320 R11:00007fc5811cb700
2024-06-29 23:49:04 R12:0000000000000000 R13:00007fc5838093f8
2024-06-29 23:49:04 R14:00007fc5811c8ea0 R15:00007fc5811c8e98
2024-06-29 23:49:04 RIP:00005589b62007eb EFL:0000000000010246
2024-06-29 23:49:04 CSGSFS:002b000000000033
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8daf) -> 00005589b6231c01
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8dae) -> 00007fc58380d5d4
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8dad) -> 00005589b6203c0e
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8dac) -> 00007fc5811c8e98
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8dab) -> 00007fc5811c8ea0
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8daa) -> 00007fc5838093f8
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da9) -> 00007fc58380d5d4
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da8) -> 00007fc583809340
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da7) -> 00007fc584c09bd0
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da6) -> 00007fc583829070
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da5) -> 20f93ab533b63900
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da4) -> 0000000000000000
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da3) -> 00005589b6230769
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da2) -> 00007fc5838290a0
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da1) -> 00007fc583809340
2024-06-29 23:49:04 1:11:M 29 Jun 2024 20:49:04.325 # (00007fc5811c8da0) -> 00007fc583829070
2024-06-29 23:49:04 
2024-06-29 23:49:04 ------ INFO OUTPUT ------
2024-06-29 23:49:04 # Server
2024-06-29 23:49:04 redis_version:6.3.4
2024-06-29 23:49:04 redis_git_sha1:7e7e5e57
2024-06-29 23:49:04 redis_git_dirty:1
2024-06-29 23:49:04 redis_build_id:bb8fc59400781b64
2024-06-29 23:49:04 redis_mode:standalone
2024-06-29 23:49:04 os:Linux 5.15.153.1-microsoft-standard-WSL2 x86_64
2024-06-29 23:49:04 arch_bits:64
2024-06-29 23:49:04 multiplexing_api:epoll
2024-06-29 23:49:04 atomicvar_api:atomic-builtin
2024-06-29 23:49:04 gcc_version:9.4.0
2024-06-29 23:49:04 process_id:1
2024-06-29 23:49:04 process_supervised:no
2024-06-29 23:49:04 run_id:67a66401c10d1ddcb140bc9078b9b2511ce065c4
2024-06-29 23:49:04 tcp_port:6379
2024-06-29 23:49:04 server_time_usec:1719694144325681
2024-06-29 23:49:04 uptime_in_seconds:1
2024-06-29 23:49:04 uptime_in_days:0
2024-06-29 23:49:04 hz:10
2024-06-29 23:49:04 configured_hz:10
2024-06-29 23:49:04 lru_clock:8418111
2024-06-29 23:49:04 executable:/data/keydb-server
2024-06-29 23:49:04 config_file:/etc/keydb/keydb.conf
2024-06-29 23:49:04 availability_zone:
2024-06-29 23:49:04 features:cluster_mget
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Clients
2024-06-29 23:49:04 connected_clients:0
2024-06-29 23:49:04 cluster_connections:0
2024-06-29 23:49:04 maxclients:10000
2024-06-29 23:49:04 client_recent_max_input_buffer:0
2024-06-29 23:49:04 client_recent_max_output_buffer:0
2024-06-29 23:49:04 blocked_clients:0
2024-06-29 23:49:04 tracking_clients:0
2024-06-29 23:49:04 clients_in_timeout_table:0
2024-06-29 23:49:04 current_client_thread:182698393
2024-06-29 23:49:04 thread_0_clients:0
2024-06-29 23:49:04 thread_1_clients:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Memory
2024-06-29 23:49:04 used_memory:2241176
2024-06-29 23:49:04 used_memory_human:2.14M
2024-06-29 23:49:04 used_memory_rss:0
2024-06-29 23:49:04 used_memory_rss_human:0B
2024-06-29 23:49:04 used_memory_peak:2241176
2024-06-29 23:49:04 used_memory_peak_human:2.14M
2024-06-29 23:49:04 used_memory_peak_perc:inf%
2024-06-29 23:49:04 used_memory_overhead:2238488
2024-06-29 23:49:04 used_memory_startup:2238488
2024-06-29 23:49:04 used_memory_dataset:2688
2024-06-29 23:49:04 used_memory_dataset_perc:100.00%
2024-06-29 23:49:04 allocator_allocated:0
2024-06-29 23:49:04 allocator_active:0
2024-06-29 23:49:04 allocator_resident:0
2024-06-29 23:49:04 total_system_memory:16732319744
2024-06-29 23:49:04 total_system_memory_human:15.58G
2024-06-29 23:49:04 used_memory_lua:37888
2024-06-29 23:49:04 used_memory_lua_human:37.00K
2024-06-29 23:49:04 used_memory_scripts:0
2024-06-29 23:49:04 used_memory_scripts_human:0B
2024-06-29 23:49:04 number_of_cached_scripts:0
2024-06-29 23:49:04 maxmemory:0
2024-06-29 23:49:04 maxmemory_human:0B
2024-06-29 23:49:04 maxmemory_policy:noeviction
2024-06-29 23:49:04 allocator_frag_ratio:-nan
2024-06-29 23:49:04 allocator_frag_bytes:0
2024-06-29 23:49:04 allocator_rss_ratio:-nan
2024-06-29 23:49:04 allocator_rss_bytes:0
2024-06-29 23:49:04 rss_overhead_ratio:-nan
2024-06-29 23:49:04 rss_overhead_bytes:0
2024-06-29 23:49:04 mem_fragmentation_ratio:-nan
2024-06-29 23:49:04 mem_fragmentation_bytes:0
2024-06-29 23:49:04 mem_not_counted_for_evict:0
2024-06-29 23:49:04 mem_replication_backlog:0
2024-06-29 23:49:04 mem_clients_slaves:0
2024-06-29 23:49:04 mem_clients_normal:0
2024-06-29 23:49:04 mem_aof_buffer:0
2024-06-29 23:49:04 mem_allocator:jemalloc-5.2.1
2024-06-29 23:49:04 active_defrag_running:0
2024-06-29 23:49:04 lazyfree_pending_objects:0
2024-06-29 23:49:04 lazyfreed_objects:0
2024-06-29 23:49:04 storage_provider:none
2024-06-29 23:49:04 available_system_memory:unavailable
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Persistence
2024-06-29 23:49:04 loading:0
2024-06-29 23:49:04 current_cow_size:0
2024-06-29 23:49:04 current_cow_size_age:0
2024-06-29 23:49:04 current_fork_perc:0.00
2024-06-29 23:49:04 current_save_keys_processed:0
2024-06-29 23:49:04 current_save_keys_total:0
2024-06-29 23:49:04 rdb_changes_since_last_save:0
2024-06-29 23:49:04 rdb_bgsave_in_progress:0
2024-06-29 23:49:04 rdb_last_save_time:1719694143
2024-06-29 23:49:04 rdb_last_bgsave_status:ok
2024-06-29 23:49:04 rdb_last_bgsave_time_sec:-1
2024-06-29 23:49:04 rdb_current_bgsave_time_sec:-1
2024-06-29 23:49:04 rdb_last_cow_size:0
2024-06-29 23:49:04 aof_enabled:0
2024-06-29 23:49:04 aof_rewrite_in_progress:0
2024-06-29 23:49:04 aof_rewrite_scheduled:0
2024-06-29 23:49:04 aof_last_rewrite_time_sec:-1
2024-06-29 23:49:04 aof_current_rewrite_time_sec:-1
2024-06-29 23:49:04 aof_last_bgrewrite_status:ok
2024-06-29 23:49:04 aof_last_write_status:ok
2024-06-29 23:49:04 aof_last_cow_size:0
2024-06-29 23:49:04 module_fork_in_progress:0
2024-06-29 23:49:04 module_fork_last_cow_size:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Stats
2024-06-29 23:49:04 total_connections_received:0
2024-06-29 23:49:04 total_commands_processed:0
2024-06-29 23:49:04 instantaneous_ops_per_sec:0
2024-06-29 23:49:04 total_net_input_bytes:0
2024-06-29 23:49:04 total_net_output_bytes:0
2024-06-29 23:49:04 instantaneous_input_kbps:0.00
2024-06-29 23:49:04 instantaneous_output_kbps:0.00
2024-06-29 23:49:04 rejected_connections:0
2024-06-29 23:49:04 sync_full:0
2024-06-29 23:49:04 sync_partial_ok:0
2024-06-29 23:49:04 sync_partial_err:0
2024-06-29 23:49:04 expired_keys:0
2024-06-29 23:49:04 expired_stale_perc:0.00
2024-06-29 23:49:04 expired_time_cap_reached_count:0
2024-06-29 23:49:04 expire_cycle_cpu_milliseconds:0
2024-06-29 23:49:04 evicted_keys:0
2024-06-29 23:49:04 keyspace_hits:0
2024-06-29 23:49:04 keyspace_misses:0
2024-06-29 23:49:04 pubsub_channels:0
2024-06-29 23:49:04 pubsub_patterns:0
2024-06-29 23:49:04 latest_fork_usec:0
2024-06-29 23:49:04 total_forks:0
2024-06-29 23:49:04 migrate_cached_sockets:0
2024-06-29 23:49:04 slave_expires_tracked_keys:0
2024-06-29 23:49:04 active_defrag_hits:0
2024-06-29 23:49:04 active_defrag_misses:0
2024-06-29 23:49:04 active_defrag_key_hits:0
2024-06-29 23:49:04 active_defrag_key_misses:0
2024-06-29 23:49:04 tracking_total_keys:0
2024-06-29 23:49:04 tracking_total_items:0
2024-06-29 23:49:04 tracking_total_prefixes:0
2024-06-29 23:49:04 unexpected_error_replies:0
2024-06-29 23:49:04 total_error_replies:0
2024-06-29 23:49:04 dump_payload_sanitizations:0
2024-06-29 23:49:04 total_reads_processed:0
2024-06-29 23:49:04 total_writes_processed:0
2024-06-29 23:49:04 instantaneous_lock_contention:0
2024-06-29 23:49:04 avg_lock_contention:0.000000
2024-06-29 23:49:04 storage_provider_read_hits:0
2024-06-29 23:49:04 storage_provider_read_misses:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Replication
2024-06-29 23:49:04 role:master
2024-06-29 23:49:04 connected_slaves:0
2024-06-29 23:49:04 master_failover_state:no-failover
2024-06-29 23:49:04 master_replid:a1c5b072246ed775d16fb6c91aeb177cea103c50
2024-06-29 23:49:04 master_replid2:0000000000000000000000000000000000000000
2024-06-29 23:49:04 master_repl_offset:0
2024-06-29 23:49:04 second_repl_offset:-1
2024-06-29 23:49:04 repl_backlog_active:0
2024-06-29 23:49:04 repl_backlog_size:1048576
2024-06-29 23:49:04 repl_backlog_first_byte_offset:0
2024-06-29 23:49:04 repl_backlog_histlen:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 # CPU
2024-06-29 23:49:04 used_cpu_sys:0.020420
2024-06-29 23:49:04 used_cpu_user:0.010210
2024-06-29 23:49:04 used_cpu_sys_children:0.000000
2024-06-29 23:49:04 used_cpu_user_children:0.000107
2024-06-29 23:49:04 server_threads:2
2024-06-29 23:49:04 long_lock_waits:0
2024-06-29 23:49:04 used_cpu_sys_main_thread:0.000000
2024-06-29 23:49:04 used_cpu_user_main_thread:0.001198
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Modules
2024-06-29 23:49:04 module:name=graph,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Commandstats
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Errorstats
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Cluster
2024-06-29 23:49:04 cluster_enabled:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 # Keyspace
2024-06-29 23:49:04 
2024-06-29 23:49:04 # KeyDB
2024-06-29 23:49:04 mvcc_depth:0
2024-06-29 23:49:04 
2024-06-29 23:49:04 ------ CLIENT LIST OUTPUT ---

Aditional information I am trying to run a benchmark for keydb with redisgraph. but I can't make redisgraph module to work on keydb, I tried multiple version already.

  1. OS distribution and version Docker from: eqalpha/keydb:alpine_x86_64_v6.3.4 also tried latest
  2. Steps to reproduce (if any)

after building redisgraph I tried this:

Set the entry point to run KeyDB with the RedisGraph module and debug logging

CMD ["sh", "-c", "keydb-server /etc/keydb/keydb.conf --loadmodule /opt/keydb/modules/redisgraph.so --loglevel debug"]

with debug level to see where it crashes.

keithchew commented 4 days ago

hi @Lion5324

As far as I know, redis modules from redis 7 have changes that will make them incompatible with KeyDB. In addition, depending on which module you want to use, support within KeyDB will vary. Eg you need to do a lot of work to get RediSearch to work, less for RedisJson, etc.

If you have to use RedisGraph, I would recommend first trying the module from redis 6. If you still experience crashes, then you will need to trace the code from the stack trace and patch (KeyDB or module) accordingly.

Just looking at your trace, it could be similar to a crash I found a while back:

https://github.com/Snapchat/KeyDB/issues/777

My current workaround is to uncomment the assert. But in your case, it looks like the code did not reach that point, so you will need to do a bit of logging to track it down.