search

microsoftarchive / redis

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes
http://redis.io
Other
20.81k stars 5.37k forks source link

Urgent issue with production - Redis 3.0.501 deletes all keys on sporadic saves (* DB saved on disk) #488

Closed DanielMunch closed 8 years ago

DanielMunch commented 8 years ago

We are experiencing an urgent issue in production, where "* DB saved on disk" deletes all stored keys in the rdb file. The background save / fork seems for run as expected.

[932] 10 Jun 15:08:11.082 # Granting read/write access to 'NT AUTHORITY\NetworkService' on: "C:\Trackunit\redis\Service" "c:\trackunit\redis\dump\" [932] 10 Jun 15:08:11.082 # Redis successfully installed as a service. Redis 3.0.501 (00000000/0) 64 bit Running in standalone mode Port: 6379 PID: 4836

[4836] 13 Jun 09:31:45.417 # Server started, Redis version 3.0.501 [4836] 13 Jun 09:32:21.747 * DB loaded from disk: 36.319 seconds [4836] 13 Jun 09:32:21.747 * The server is now ready to accept connections on port 6379 [4836] 13 Jun 09:32:22.419 - DB 0: 10440383 keys (10440383 volatile) in 25165824 slots HT. [4836] 13 Jun 09:32:22.419 - DB 1: 544146 keys (544146 volatile) in 1572864 slots HT.

Config is set to SAVE 3600 1, and this is triggering ok, as far as I can see there are no issues here. ie;

[1400] 13 Jun 13:02:33.292 - DB 0: 10156611 keys (10156611 volatile) in 16777216 slots HT. [1400] 13 Jun 13:02:33.292 - DB 1: 528267 keys (528267 volatile) in 1048576 slots HT. [1400] 13 Jun 13:02:33.292 - 2 clients connected (0 slaves), 2947733928 bytes in use [1400] 13 Jun 13:02:34.710 # fork operation complete [1400] 13 Jun 13:02:35.085 * Background saving terminated with success [1400] 13 Jun 13:02:39.132 - DB 0: 10156593 keys (10156593 volatile) in 16777216 slots HT. [1400] 13 Jun 13:02:39.132 - DB 1: 528259 keys (528259 volatile) in 1048576 slots HT.

Then I see every few days there are saves made with a different log entry, and this save WIPES the stored keys in production, this is a disaster.

[1400] 24 Jul 15:50:50.118 - DB 0: 1215196 keys (1215195 volatile) in 2097152 slots HT. [1400] 24 Jul 15:50:50.118 - DB 1: 57829 keys (57829 volatile) in 65536 slots HT. [1400] 24 Jul 15:50:50.118 - 36 clients connected (0 slaves), 356430096 bytes in use [1400] 24 Jul 15:50:54.106 - Accepted 89.248.172.9:47362 [1400] 24 Jul 15:50:56.280 * DB saved on disk [1400] 24 Jul 15:50:56.296 - Client closed connection [1400] 24 Jul 15:50:56.312 - Accepted 89.248.172.9:47417 [1400] 24 Jul 15:50:56.327 - Client closed connection [1400] 24 Jul 15:50:56.343 - Accepted 89.248.172.9:47419 [1400] 24 Jul 15:50:56.374 - Client closed connection [1400] 24 Jul 15:50:57.704 - DB 0: 197 keys (196 volatile) in 256 slots HT. [1400] 24 Jul 15:50:57.704 - DB 1: 17 keys (17 volatile) in 32 slots HT. [1400] 24 Jul 15:50:57.704 - 36 clients connected (0 slaves), 2533672 bytes in use

and..

[1400] 23 Jul 19:05:57.861 - DB 0: 1065461 keys (1065460 volatile) in 2097152 slots HT. [1400] 23 Jul 19:05:57.861 - DB 1: 46469 keys (46469 volatile) in 65536 slots HT. [1400] 23 Jul 19:05:57.861 - 36 clients connected (0 slaves), 314862440 bytes in use [1400] 23 Jul 19:06:01.768 - Accepted 89.248.172.9:42046 [1400] 23 Jul 19:06:03.550 * DB saved on disk [1400] 23 Jul 19:06:03.565 - Client closed connection [1400] 23 Jul 19:06:03.581 - Accepted 89.248.172.9:42212 [1400] 23 Jul 19:06:03.597 - Client closed connection [1400] 23 Jul 19:06:03.628 - Accepted 89.248.172.9:42215 [1400] 23 Jul 19:06:03.643 - Client closed connection [1400] 23 Jul 19:06:05.081 - DB 0: 197 keys (196 volatile) in 256 slots HT. [1400] 23 Jul 19:06:05.081 - DB 1: 10 keys (10 volatile) in 16 slots HT. [1400] 23 Jul 19:06:05.081 - 36 clients connected (0 slaves), 2347040 bytes in use

I have the entire 170 mb log file available if needed.

cli INFO;

127.0.0.1:6379> info

Server

redis_version:3.0.501 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:ba05b51e58eb9205 redis_mode:standalone os:Windows arch_bits:64 multiplexing_api:WinSock_IOCP process_id:1400 run_id:c8de48e2f3416548a36e58fe697709773a82a863 tcp_port:6379 uptime_in_seconds:3703193 uptime_in_days:42 hz:10 lru_clock:9896997 config_file:C:\Trackunit\redis\Service\redis.windows.conf

Clients

connected_clients:37 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0

Memory

used_memory:640185160 used_memory_human:610.53M used_memory_rss:640147376 used_memory_peak:3443551960 used_memory_peak_human:3.21G used_memory_lua:36864 mem_fragmentation_ratio:1.00 mem_allocator:jemalloc-3.6.0

Persistence

loading:0 rdb_changes_since_last_save:134473 rdb_bgsave_in_progress:0 rdb_last_save_time:1469512303 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:9 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok

Stats

total_connections_received:174 total_commands_processed:822433029 instantaneous_ops_per_sec:314 total_net_input_bytes:40841658928 total_net_output_bytes:76599625112 instantaneous_input_kbps:18.45 instantaneous_output_kbps:24.64 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:45162957 evicted_keys:0 keyspace_hits:595121785 keyspace_misses:84083202 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:49037 migrate_cached_sockets:0

Replication

role:master connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:24808.31 used_cpu_user:18698.80 used_cpu_sys_children:0.00 used_cpu_user_children:0.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=2102643,expires=2102642,avg_ttl=568353539 db1:keys=105136,expires=105136,avg_ttl=563796227 127.0.0.1:6379>

redis.windows.config.txt

Config attached. redis.windows.config.txt

DanielMunch commented 8 years ago

Issue caused by malicious access by hacker.