Closed mazuryv closed 2 years ago
i cant get useful information from error message,can you give me more details?os version?how did you run redis?config file?how long happended?every time happended or sometimes?
I used 2 Windows Server 2019 Standart as OS. Configuration: First server - master + sentinel Second one - replica + 2 sentinels. Master.conf
port 6379
requirepass "secret"
masterauth "secret"
loglevel notice
logfile "Logs/redis_log.txt"
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
save 3600 1 300 100 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
dbfilename RedisDump_NotDeleteDuringRedeploy.rdb
maxmemory 256mb
appendonly no
maxmemory-policy allkeys-lru
Replica.conf
port 6379
requirepass "secret"
masterauth "secret"
loglevel notice
logfile "Logs/redis_log.txt"
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
save 3600 1 300 100 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
dbfilename RedisDump_NotDeleteDuringRedeploy.rdb
maxmemory 256mb
appendonly no
maxmemory-policy allkeys-lru
replicaof 10.98.96.15 6379
There were Errors in event viewer of OS related to this error:
Faulting application name: redis-server.exe, version: 7.0.0.0, time stamp: 0x62d77e36
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007fe11e44b005
Faulting process id: 0x1334
Faulting application start time: 0x01d8ac0050977fb3
Faulting application path: C:\Redis\redis-server.exe
Faulting module path: unknown
Report Id: 799844a2-6432-4ef3-a1e2-b94474679838
Faulting package full name:
Faulting package-relative application ID:
After files RedisDump_NotDeleteDuringRedeploy.rdb were deleted from both servers (with stopped services) - this error does not happen again.
Besides. They are redeployed every morning and all sync services are down. I have specified restore after fail and services are started regularly. Info from log:
[008564] 10 Aug 04:25:44.209 # RedisWin is starting ......
[008564] 10 Aug 04:25:44.209 # RedisWin version=7.0.4, bits=64, commit=07/20/2022, modified=0, pid=8564, just started
[008564] 10 Aug 04:25:44.209 # Configuration loaded
[008564] 10 Aug 04:25:44.209 * monotonic clock: POSIX clock_gettime
[008564] 10 Aug 04:25:44.224 * Running mode=standalone, port=6379.
[008564] 10 Aug 04:25:44.224 # Server initialized
[008564] 10 Aug 04:25:44.224 * Loading RDB produced by version 7.0.4
[008564] 10 Aug 04:25:44.224 * RDB age 6856 seconds
[008564] 10 Aug 04:25:44.224 * RDB memory usage when created 0.00 Mb
[008564] 10 Aug 04:25:44.240 # Accepting client connection: accept: Either the application has not called WSAStartup, or WSAStartup failed.
[009972] 10 Aug 04:25:44.677 # RedisWin is starting ......
[009972] 10 Aug 04:25:44.677 # RedisWin version=7.0.4, bits=64, commit=07/20/2022, modified=0, pid=9972, just started
[009972] 10 Aug 04:25:44.677 # Configuration loaded
[009972] 10 Aug 04:25:44.677 * monotonic clock: POSIX clock_gettime
[009972] 10 Aug 04:25:44.693 * Running mode=standalone, port=6379.
Thank you.
Thanks,according to your config file,i will test it ,maybe need some days,if i find out issue,i will inform u here. meanwhile you also can do a test for me,you stop sentinel service and observe what happend.i want to know bcz of replicas or sentinel issue.
When I have stopped redis sentinels on Replica and Master, nothing happened. After 3 minutes, they were started. Sentinel logs:
000440] 16 Aug 07:58:30.853 * Sentinel new configuration saved on disk
[000440] 16 Aug 09:42:00.403 # +sdown sentinel 5991b798926b4e8bf4fb10de83680fdea4a96925 10.98.96.16 26378 @ *** 10.98.96.15 6379
[000440] 16 Aug 09:42:02.469 # +sdown sentinel 1c75f8be0c662b06f287609ca1da958b83a0de0c 10.98.96.16 26379 @ *** 10.98.96.15 6379
[008788] 16 Aug 09:44:57.580 # RedisWin is starting ......
[008788] 16 Aug 09:44:57.580 # RedisWin version=7.0.4, bits=64, commit=07/20/2022, modified=0, pid=8788, just started
[008788] 16 Aug 09:44:57.580 # Configuration loaded
[008788] 16 Aug 09:44:57.580 * monotonic clock: POSIX clock_gettime
[008788] 16 Aug 09:44:57.580 * Running mode=sentinel, port=26379.
[008788] 16 Aug 09:44:57.580 # Sentinel ID is 478d20a154d51a4df1a0c03785a33ea0ec7d1b68
[008788] 16 Aug 09:44:57.580 # +monitor master *** 10.98.96.15 6379 quorum 2
** - is my master group name
Hello. I have new errors. They were thrown without any reasons: Master
[002784] 16 Aug 22:45:59.070 * 100 changes in 300 seconds. Saving...
[002784] 16 Aug 22:45:59.086 * Background saving started by pid 260
=== REDIS BUG REPORT START: Cut & paste starting from here ===
[000260] 16 Aug 22:45:59.940 # --- ABORT
[000260] 16 Aug 22:45:59.940 # --- STACK TRACE
redis-server.exe!((null):0)(0x00000016, 0x7FE142D22E89, 0x140119B10, 0x140262490)
redis-server.exe!((null):0)(0x00000016, 0x140119B10, 0x140119B10, 0x0014F1E0)
redis-server.exe!((null):0)(0x0014ED01, 0xAFF600000000, 0x00000000, 0x6634616638752F71)
redis-server.exe!((null):0)(0x6634616638752F74, 0x140204160, 0x6634616638752F71, 0x0000005D)
redis-server.exe!((null):0)(0x00000018, 0x6634616638752F74, 0x0014F1E0, 0x06C00030)
redis-server.exe!((null):0)(0x6634616638752F74, 0x7FE109001571, 0x0014F1E0, 0x14006E573)
redis-server.exe!((null):0)(0x6634616638752F74, 0x7FE142D22E89, 0x0014F1E0, 0x0014F1E0)
redis-server.exe!((null):0)(0x7FE10682C4F0, 0x00000003, 0x07800003, 0x00000022)
redis-server.exe!((null):0)(0x0014F1E0, 0x00000004, 0x7FE106C0A103, 0x0014F1E0)
redis-server.exe!((null):0)(0x7FE10682C4F0, 0x06C00000, 0x0014F140, 0x06C00000)
redis-server.exe!((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
redis-server.exe!((null):0)(0x00612CE0, 0x00CE0000, 0x00000000, 0x00612CE0)
redis-server.exe!((null):0)(0x00CE0000, 0x00CE0000, 0x00000000, 0x000001F8)
redis-server.exe!((null):0)(0x00000234, 0x00000000, 0x00000234, 0x2AAAAAAAAAAAAAAB)
redis-server.exe!((null):0)(0x00000000, 0x00000000, 0x005F4CF0, 0x00000000)
redis-server.exe!((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[000260] 16 Aug 22:45:59.955 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report this bug by following the instructions at:
https://github.com/zkteco-home/redis-windows/wiki/Submitting-an-Issue
Suspect RAM error? Use redis-server --test-memory to verify it.
[002784] 16 Aug 22:46:00.388 # fork operation failed
[002784] 16 Aug 22:46:00.529 # Background saving terminated by signal 1
[002784] 16 Aug 22:46:05.025 * 100 changes in 300 seconds. Saving...
[002784] 16 Aug 22:46:05.025 * Background saving started by pid 4988
[002784] 16 Aug 22:46:07.780 # Connection with replica 10.98.96.16:6379 lost.
[002784] 16 Aug 22:46:08.295 # fork operation complete
[002784] 16 Aug 22:46:08.311 * Background saving terminated with success
[002784] 16 Aug 22:46:19.087 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
[002784] 16 Aug 22:46:19.087 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:19.087 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:19.087 * REPLICAOF 10.98.96.16:6379 enabled (user request from 'id=417 addr=10.98.96.16:52665 laddr=10.98.96.15:6379 fd=54 name=sentinel-5991b798-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=199 qbuf-free=20275 argv-mem=4 multi-mem=179 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 tot-mem=22703 events=r cmd=exec user=default redir=-1 resp=2')
[002784] 16 Aug 22:46:19.087 # CONFIG REWRITE executed with success.
[002784] 16 Aug 22:46:19.103 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:19.103 * Master replied to PING, replication can continue...
[002784] 16 Aug 22:46:19.103 * Trying a partial resynchronization (request abb4c134a7c21ee29f4dd618d722f48f7bc967f8:1298906116598).
[002784] 16 Aug 22:46:19.118 * Full resync from master: 437fa9e1db1d02af20f06949173f16d1c1f5b306:1298863741652
[002784] 16 Aug 22:46:20.541 * MASTER <-> REPLICA sync: receiving 128420441 bytes from master to disk
[002784] 16 Aug 22:46:21.441 * Discarding previously cached master state.
[002784] 16 Aug 22:46:21.441 * MASTER <-> REPLICA sync: Flushing old data
[002784] 16 Aug 22:46:21.441 * MASTER <-> REPLICA sync: Loading DB in memory
[002784] 16 Aug 22:46:21.457 * Loading RDB produced by version 7.0.4
[002784] 16 Aug 22:46:21.457 * RDB age 2 seconds
[002784] 16 Aug 22:46:21.457 * RDB memory usage when created 0.00 Mb
[002784] 16 Aug 22:46:21.457 # I/O error trying to sync with MASTER: connection lost
[002784] 16 Aug 22:46:21.457 * Reconnecting to MASTER 10.98.96.16:6379 after failure
[002784] 16 Aug 22:46:21.457 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:21.472 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:21.472 * Master replied to PING, replication can continue...
[002784] 16 Aug 22:46:21.472 * Partial resynchronization not possible (no cached master)
[002784] 16 Aug 22:46:21.488 * Full resync from master: 437fa9e1db1d02af20f06949173f16d1c1f5b306:1298863747163
[002784] 16 Aug 22:46:22.957 * MASTER <-> REPLICA sync: receiving 128420646 bytes from master to disk
[002784] 16 Aug 22:46:23.269 # RDB file was saved with checksum disabled: no check performed.
[002784] 16 Aug 22:46:23.269 * Done loading RDB, keys loaded: 39, keys expired: 0.
[002784] 16 Aug 22:46:23.269 * MASTER <-> REPLICA sync: Finished with success
[002784] 16 Aug 22:46:23.347 # Connection with master lost.
[002784] 16 Aug 22:46:23.347 * Caching the disconnected master state.
[002784] 16 Aug 22:46:23.347 * Reconnecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:23.347 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:24.379 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:24.379 # Sending command to master in replication handshake: -Writing to master: No error
[002784] 16 Aug 22:46:24.472 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:24.472 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:25.498 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:25.498 # Sending command to master in replication handshake: -Writing to master: No error
[002784] 16 Aug 22:46:25.560 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:25.560 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:26.577 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:26.577 # Sending command to master in replication handshake: -Writing to master: No error
[002784] 16 Aug 22:46:26.655 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:26.655 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:27.671 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:27.671 # Sending command to master in replication handshake: -Writing to master: No error
[002784] 16 Aug 22:46:27.749 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:27.749 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:28.765 * Non blocking connect for SYNC fired the event.
[002784] 16 Aug 22:46:28.765 # Sending command to master in replication handshake: -Writing to master: No error
[002784] 16 Aug 22:46:28.843 * Connecting to MASTER 10.98.96.16:6379
[002784] 16 Aug 22:46:28.843 * MASTER <-> REPLICA sync started
[002784] 16 Aug 22:46:29.766 * Discarding previously cached master state.
[002784] 16 Aug 22:46:29.766 # Setting secondary replication ID to 437fa9e1db1d02af20f06949173f16d1c1f5b306, valid up to offset: 1298863747164. New replication ID is 0d82d40d67c1623d35f1a9ba466bdc6fb2a776f5
[002784] 16 Aug 22:46:29.766 * MASTER MODE enabled (user request from 'id=417 addr=10.98.96.16:52665 laddr=10.98.96.15:6379 fd=54 name=sentinel-5991b798-cmd age=20 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=188 qbuf-free=20286 argv-mem=4 multi-mem=169 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 tot-mem=22693 events=r cmd=exec user=default redir=-1 resp=2')
[002784] 16 Aug 22:46:29.766 # CONFIG REWRITE executed with success.
[002784] 16 Aug 22:47:04.098 * Replica 10.98.96.16:6379 asks for synchronization
[002784] 16 Aug 22:47:04.098 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '75101e130d74f2d860769244184f311aeb5173bc', my replication IDs are '0d82d40d67c1623d35f1a9ba466bdc6fb2a776f5' and '437fa9e1db1d02af20f06949173f16d1c1f5b306')
Replica:
[009180] 16 Aug 22:45:56.044 * 100 changes in 300 seconds. Saving...
[009180] 16 Aug 22:45:56.044 * Background saving started by pid 9752
[009180] 16 Aug 22:45:57.466 # fork operation complete
[009180] 16 Aug 22:45:57.529 * Background saving terminated with success
[009180] 16 Aug 22:46:07.789 # Connection with master lost.
[009180] 16 Aug 22:46:07.789 * Caching the disconnected master state.
[009180] 16 Aug 22:46:07.789 * Discarding previously cached master state.
[009180] 16 Aug 22:46:07.789 # Setting secondary replication ID to abb4c134a7c21ee29f4dd618d722f48f7bc967f8, valid up to offset: 1298863711516. New replication ID is 437fa9e1db1d02af20f06949173f16d1c1f5b306
[009180] 16 Aug 22:46:07.789 * MASTER MODE enabled (user request from 'id=130 addr=10.98.96.16:61507 laddr=10.98.96.16:6379 fd=39 name=sentinel-1c75f8be-cmd age=46427 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=188 qbuf-free=20286 argv-mem=4 multi-mem=169 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 tot-mem=22693 events=r cmd=exec user=default redir=-1 resp=2')
[009180] 16 Aug 22:46:07.789 # CONFIG REWRITE executed with success.
[009180] 16 Aug 22:46:08.352 # CONFIG REWRITE executed with success.
[009180] 16 Aug 22:46:19.103 * Replica 10.98.96.15:6379 asks for synchronization
[009180] 16 Aug 22:46:19.103 * Partial resynchronization not accepted: Requested offset for second ID was 1298906116598, but I can reply up to 1298863711516
[009180] 16 Aug 22:46:19.103 * Starting BGSAVE for SYNC with target: disk
[009180] 16 Aug 22:46:19.119 * Background saving started by pid 7352
[009180] 16 Aug 22:46:20.509 # fork operation complete
[009180] 16 Aug 22:46:20.540 * Background saving terminated with success
[009180] 16 Aug 22:46:21.212 * Synchronization with replica 10.98.96.15:6379 succeeded
[009180] 16 Aug 22:46:21.462 # Connection with replica 10.98.96.15:6379 lost.
[009180] 16 Aug 22:46:21.478 * Replica 10.98.96.15:6379 asks for synchronization
[009180] 16 Aug 22:46:21.478 * Full resync requested by replica 10.98.96.15:6379
[009180] 16 Aug 22:46:21.478 * Starting BGSAVE for SYNC with target: disk
[009180] 16 Aug 22:46:21.478 * Background saving started by pid 9412
[009180] 16 Aug 22:46:22.947 # fork operation complete
[009180] 16 Aug 22:46:22.962 * Background saving terminated with success
=== REDIS BUG REPORT START: Cut & paste starting from here ===
[009180] 16 Aug 22:46:22.978 # --- EXCEPTION_ACCESS_VIOLATION
[009180] 16 Aug 22:46:22.978 # --- STACK TRACE
redis-server.exe!((null):0)(0x1401E2B40, 0x00E9FF60, 0x00E9E9A0, 0x00E9E9A0)
redis-server.exe!((null):0)(0x00E9E9A0, 0x00E9FF01, 0x1401197F0, 0x140123990)
KERNELBASE.dll!UnhandledExceptionFilter((null):0)(0x00E9EFC0, 0x7FFD72163090, 0x00000000, 0x00000000)
ntdll.dll!memset((null):0)(0x00000000, 0x7FFD72023100, 0x00E9F630, 0x00E9EF68)
ntdll.dll!_C_specific_handler((null):0)(0x00000000, 0x00E9EF50, 0x00000000, 0x140000000)
ntdll.dll!_chkstk((null):0)(0x00E9EF50, 0x00000000, 0x7FFD6F9EBC3C, 0x7FFD6F940000)
ntdll.dll!RtlWalkFrameChain((null):0)(0x00E9F1E0, 0x00000008, 0x7FE11A404D00, 0x1400C1C92)
ntdll.dll!KiUserExceptionDispatcher((null):0)(0x14002CBE1, 0x00000000, 0x140093856, 0x0213BBA0)
ntdll.dll!KiUserExceptionDispatcher((null):0)(0x00000000, 0x140093856, 0x0213BBA0, 0x00000002)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x7FE10B8002A0, 0x00000000, 0x00000000, 0x00000000)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000000, 0x11B0000001B, 0x00000002, 0x00000002)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x000023DC, 0x1401DC600, 0x14025AA50, 0x7FE109400EA0)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x0213BBA0, 0x0213BBA0, 0x0213BBA0, 0x00000002)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x01E47680, 0x01E47680, 0x01E47680, 0x0213BBA0)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[009180] 16 Aug 22:46:22.978 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report this bug by following the instructions at:
https://github.com/zkteco-home/redis-windows/wiki/Submitting-an-Issue
Suspect RAM error? Use redis-server --test-memory to verify it.
[007408] 16 Aug 22:46:53.653 # RedisWin is starting ......
[007408] 16 Aug 22:46:53.653 # RedisWin version=7.0.4, bits=64, commit=07/20/2022, modified=0, pid=7408, just started
When I have stopped redis sentinels on Replica and Master, nothing happened. After 3 minutes, they were started. Sentinel logs:
000440] 16 Aug 07:58:30.853 * Sentinel new configuration saved on disk [000440] 16 Aug 09:42:00.403 # +sdown sentinel 5991b798926b4e8bf4fb10de83680fdea4a96925 10.98.96.16 26378 @ *** 10.98.96.15 6379 [000440] 16 Aug 09:42:02.469 # +sdown sentinel 1c75f8be0c662b06f287609ca1da958b83a0de0c 10.98.96.16 26379 @ *** 10.98.96.15 6379 [008788] 16 Aug 09:44:57.580 # RedisWin is starting ...... [008788] 16 Aug 09:44:57.580 # RedisWin version=7.0.4, bits=64, commit=07/20/2022, modified=0, pid=8788, just started [008788] 16 Aug 09:44:57.580 # Configuration loaded [008788] 16 Aug 09:44:57.580 * monotonic clock: POSIX clock_gettime [008788] 16 Aug 09:44:57.580 * Running mode=sentinel, port=26379. [008788] 16 Aug 09:44:57.580 # Sentinel ID is 478d20a154d51a4df1a0c03785a33ea0ec7d1b68 [008788] 16 Aug 09:44:57.580 # +monitor master *** 10.98.96.15 6379 quorum 2
** - is my master group name
i didnt get useful information. "nothing happenen" what means? i want to know one thing,if you stop sentinel service,can redis work fine?
I tested it with your configration,i didnt start sintinel service. i didnt find error.so can you tell me some detail?for ex.when will your error happen?it is easy to reproduce or difficulty?
Sorry, but I can`t prepare conditions when this error is thrown. Unfortunately, I only periodically find it in logs. Besides, I specified that services are started after fault and have only minute delays for my aims. Not only that, but I am installing version 7.0.5 and try it a bit later.
Version 7.0.4 latest release
Could you please investigate it?