Closed donspaulding closed 10 years ago
I spent a couple of hours this morning trying to find why my clients were aborting with invalid protocol exceptions and will roll back to 2.8.2 to see if this fixes it
Hello, back to work today, investigating. Thanks for reporting!
Hi again, I see here that you are using a 2.4.8 master with 2.8.3 slaves. While this should be possible in the first synchronization stage, this is not safe in the long run, or at least not guaranteed to be safe at all.
I wonder if @pedigree has the same setup.
Btw what happens here AFAIK is that the 2.8.3 slave sends the newlines that the 2.4.8 master is not able to understand, replying with an error that is interpreted by 2.8.3 as a protocol violation.
This is easy to fix in this specific instance since 2.4.8 masters can be easily detected as they are flagged as not understanding PSYNC. However there is to check what happens with, for instance, 2.8.2 masters.
Investigating further the issue in order to provide a more general fix.
I have a 1 x 2.6.17 -> (stunnel over WAN) -> 1 x 2.8.2 (patched with interval sets) -> 3 x 2.8.2 (patched with interval sets, was gold 2.8.3)
I've been concentrating on issues with hiphop hhvm just not connecting after 24 hours of stress testing but I am not longer seeing protocol errors being logging with a 2.8.2 slave, or in the 2.8.3 middle slave/master
I'll install a 2.8.3 for testing but it is a custom code base as interval sets has dropped my system load so much that it's now going to have to stay.
On 08/01/2014 12:11, Salvatore Sanfilippo wrote:
Hi again, I see here that you are using a 2.4.8 master with 2.8.3 slaves. While this should be possible in the first synchronization stage, this is not safe in the long run, or at least not guaranteed to be safe at all.
I wonder if @pedigree https://github.com/pedigree has the same setup.
Btw what happens here AFAIK is that the 2.8.3 slave sends the newlines that the 2.4.8 master is not able to understand, replying with an error that is interpreted by 2.8.3 as a protocol violation.
This is easy to fix in this specific instance since 2.4.8 masters can be easily detected as they are flagged as not understanding PSYNC. However there is to check what happens with, for instance, 2.8.2 masters.
Investigating further the issue in order to provide a more general fix.
— Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/1488#issuecomment-31826192.
@pedigree thanks for the additional info.
All: here is a clarification of the issue. The new problem was actually exposed by commit 0ff078d8d07c98f7c38af8858b4ec29c5623e7b2 and not by the fix for issue #1449.
But the new behavior is actually more correct, it just manages to expose a different bug at all, that is, slaves of old masters that don't support PSYNC, still trying to use REPLCONF ACK
commands against masters.
The fix is easy and should be ready in a matter of minutes. Working at it right now.
Hello, the above commit fixes the issue (tested). Redis 2.8.4 will be released in the next days, however you should be able to cherry pick the commit in your branch or use the latest 2.8 commit from Github.
Thanks, Salvatore
As im doing it via the web interface, I'm not sure how to but will continue to update the interval set fork until I can convince you its a great pull commit :)
On 08/01/2014 13:33, Salvatore Sanfilippo wrote:
Hello, the above commit fixes the issue (tested). Redis 2.8.4 will be released in the next days, however you should be able to cherry pick the commit in your branch or use the latest 2.8 commit from Github.
Thanks, Salvatore
— Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/1488#issuecomment-31830860.
Thanks very much for this @antirez. We'll test this after 2.8.4 hits the Ubuntu PPA we're using and follow up if we run into further issues.
Hi,
I am wondering if it is really fixed. I could reproduce the issue with redis 2.8.4 as well as redis 2.8.7. I am using master from redis 2.4.17 and was able to create slave easily on redis 2.4.17.
However, when I create a SLAVEOF using either 2.8.4 or 2.8.7, I get the following issue:
[20087] 10 Mar 14:11:19.551 * MASTER <-> SLAVE sync started [20087] 10 Mar 14:11:19.552 * Non blocking connect for SYNC fired the event. [20087] 10 Mar 14:11:19.552 * Master replied to PING, replication can continue... [20087] 10 Mar 14:11:19.553 * Partial resynchronization not possible (no cached master) [20087] 10 Mar 14:11:19.553 * Master does not support PSYNC or is in error state (reply: -ERR unknown command 'PSYNC') [20087] 10 Mar 14:11:19.553 * Retrying with SYNC...
role:slave master_host:10.101.4.255 master_port:6402 master_link_status:down master_last_io_seconds_ago:-1 master_sync_in_progress:0 slave_repl_offset:1 master_link_down_since_seconds:1394486339 slave_priority:100 slave_read_only:1 connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
Your console output above doesn't show the same error that started this issue (Protocol error from client
). It's normal for your 2.8 replica to fail on PSYNC
(not supported by 2.4) then fail back to SYNC
.
Do you have more output showing the failure to sync against your master?
Do you have logs or console output from both the master and the attempted replica?
And, just curious, but why still use a 2.4 master? Has your master just been up for a year and a half without having to be restarted? You'll have memory improvements, performance fixes, and fewer edge case issues like this by upgrading everything to a recent 2.8.x.
Basically, I am not able to create slave at all from either redis 2.4.17 or redis 2.6.11.
We want to migrate from redis 2.4 to redis 2.8.7. Since our master is still running on 2.4.17. we want to have slave on 2.8.7 before switchover.
Regarding 2.4 master, we switchover master from one server to another server every day. But both are running on redis 2.4.17. Hence master/slave replica works fine.
Do you have more output showing the failure to sync against your master? ==> Yes. it is same output which goes in the infinite loop.
redis log from the slave server:
23008] 10 Mar 15:22:41.007 * MASTER <-> SLAVE sync started [23008] 10 Mar 15:22:41.007 * Non blocking connect for SYNC fired the event. [23008] 10 Mar 15:22:41.008 * Master replied to PING, replication can continue... [23008] 10 Mar 15:22:41.008 * Partial resynchronization not possible (no cached master) [23008] 10 Mar 15:22:41.008 * Master does not support PSYNC or is in error state (reply: -ERR unknown command 'PSYNC') [23008] 10 Mar 15:22:41.008 * Retrying with SYNC... [23008] 10 Mar 15:22:46.017 # Opening the temp file needed for MASTER <-> SLAVE synchronization: Read-only file system [23008] 10 Mar 15:22:46.918 * Connecting to MASTER uid2.dw.sc.gwallet.com:6415 [23008] 10 Mar 15:22:46.918 * MASTER <-> SLAVE sync started [23008] 10 Mar 15:22:46.918 * Non blocking connect for SYNC fired the event. [23008] 10 Mar 15:22:46.919 * Master replied to PING, replication can continue... [23008] 10 Mar 15:22:46.919 * Partial resynchronization not possible (no cached master) [23008] 10 Mar 15:22:46.919 * Master does not support PSYNC or is in error state (reply: -ERR unknown command 'PSYNC') [23008] 10 Mar 15:22:46.919 * Retrying with SYNC... [23008] 10 Mar 15:22:51.930 # Opening the temp file needed for MASTER <-> SLAVE synchronization: Read-only file system [23008] 10 Mar 15:22:52.832 * Connecting to MASTER uid2.dw.sc.gwallet.com:6415 [23008] 10 Mar 15:22:52.832 * MASTER <-> SLAVE sync started
redis log from master:
23642] 10 Mar 15:23:28 * Slave ask for synchronization [23642] 10 Mar 15:23:28 * Starting BGSAVE for SYNC [23642] 10 Mar 15:23:28 * Background saving started by pid 24427 [24427] 10 Mar 15:23:29 * DB saved on disk [23642] 10 Mar 15:23:29 * Background saving terminated with success [23642] 10 Mar 15:23:29 - DB 15: 985062 keys (985062 volatile) in 1048576 slots HT. [23642] 10 Mar 15:23:29 - 0 clients connected (1 slaves), 127312240 bytes in use [23642] 10 Mar 15:23:33 - Reading from client: Connection reset by peer [23642] 10 Mar 15:23:34 - Accepted 10.101.4.143:57643 [23642] 10 Mar 15:23:34 * Slave ask for synchronization [23642] 10 Mar 15:23:34 * Starting BGSAVE for SYNC [23642] 10 Mar 15:23:34 * Background saving started by pid 24428 [23642] 10 Mar 15:23:34 - DB 15: 985062 keys (985062 volatile) in 1048576 slots HT. [23642] 10 Mar 15:23:34 - 0 clients connected (1 slaves), 127312240 bytes in use [24428] 10 Mar 15:23:35 * DB saved on disk [23642] 10 Mar 15:23:35 * Background saving terminated with success [23642] 10 Mar 15:23:39 - Reading from client: Connection reset by peer [23642] 10 Mar 15:23:39 - DB 15: 985062 keys (985062 volatile) in 1048576 slots HT. [23642] 10 Mar 15:23:39 - 0 clients connected (0 slaves), 127303608 bytes in use [23642] 10 Mar 15:23:40 - Accepted 10.101.4.143:57645 [23642] 10 Mar 15:23:40 * Slave ask for synchronization
Please let me know if you need more information.
Thanks for the additional information!
Redis has a sneaky way of showing errors: they are denoted by #
in the log. So all of your lines with # Opening the temp file needed for MASTER <-> SLAVE synchronization: Read-only file system
are an error — your filesystem has become read only, which means your replica can't injest the RDB from the master.
Try some fsck on your replica's file system and see if you can wake it up again.
Thank you very much. I got it fixed now. It was my problem. You were right, there was permission issue in writing new rdb file.
Thanks for your quick response. I was able to create slaveof from redis 2.4.17.
I met the same problem with redis 2.8.8
log of slave:
[2584] 17 Apr 16:37:15.041 # Server started, Redis version 2.8.8 [2584] 17 Apr 16:37:15.042 * The server is now ready to accept connections on port 6379 [2584] 17 Apr 16:37:16.043 - 0 clients connected (0 slaves), 750712 bytes in use [2584] 17 Apr 16:37:16.044 * Connecting to MASTER 192.168.100.129:6379 [2584] 17 Apr 16:37:16.045 * MASTER <-> SLAVE sync started [2584] 17 Apr 16:37:16.070 * Non blocking connect for SYNC fired the event. [2584] 17 Apr 16:37:16.071 * Master replied to PING, replication can continue... [2584] 17 Apr 16:37:16.071 * Partial resynchronization not possible (no cached master) [2584] 17 Apr 16:37:16.072 * Full resync from master: b18b2915b1380087ff1a0016f9cf2994fa696e3f:1 [2584] 17 Apr 16:37:16.171 * MASTER <-> SLAVE sync: receiving 18 bytes from master
log of master
[3807] 17 Apr 16:37:05.546 # Server started, Redis version 2.8.8 [3807] 17 Apr 16:37:05.547 * DB loaded from disk: 0.000 seconds [3807] 17 Apr 16:37:05.547 * The server is now ready to accept connections on port 6379 [3807] 17 Apr 16:37:06.546 - 0 clients connected (0 slaves), 750680 bytes in use [3807] 17 Apr 16:37:11.599 - 0 clients connected (0 slaves), 750680 bytes in use [3807] 17 Apr 16:37:16.045 - Accepted 192.168.100.130:39467 [3807] 17 Apr 16:37:16.069 * Slave asks for synchronization [3807] 17 Apr 16:37:16.069 * Full resync requested by slave. [3807] 17 Apr 16:37:16.069 * Starting BGSAVE for SYNC [3807] 17 Apr 16:37:16.074 * Background saving started by pid 3811 [3811] 17 Apr 16:37:16.090 * DB saved on disk [3811] 17 Apr 16:37:16.090 * RDB: 6 MB of memory used by copy-on-write [3807] 17 Apr 16:37:16.168 * Background saving terminated with success [3807] 17 Apr 16:37:16.179 * Synchronization with slave succeeded [3807] 17 Apr 16:37:16.672 - 0 clients connected (1 slaves), 1820184 bytes in use
it seems that master doesn't suppor PSYNC command: [root@os_1 forfun]# redis-cli 127.0.0.1:6379> help PSYNC
127.0.0.1:6379> help SYNC
SYNC - summary: Internal command used for replication since: 1.0.0 group: server
127.0.0.1:6379>
Is there any suggestion? Thanks a lot
I don't see any error in your instances, @tdk2nyt. PSYNC
is a command used only internally. As the log already says Partial resynchronization not possible (no cached master)
. This means there was never a full sync between your master and slave. partial sync can only happen if after a full sync the link between master & slave goes down for a short time. Please read the replication documentation for more info (section "Partial resynchronization")
Thanks badboy! Finally I checked the code, found I misunderstood PSYNC: PSYNC only works if slave disconnect with master for a short while, if slave restart, PSYNC not work.
hello tdk2nyt ,how do us know if PSYNC is working?
If PSYNC is working, you will not see lines like:
* Master does not support PSYNC or is in error state (reply: -ERR unknown command 'PSYNC')
is there any PSYNC loginfo we you see
We get a "Protocol error from client" error when replication completes with 2.8.3 slaves. Rolling back to version 2.8.2 makes the error go away. Furthermore, we have slaves on the same master that are using older versions of redis, and do not exhibit this issue. A first glance makes me think this is a regression introduced by #1449.
I'm not sure which data to collect to help you replicate the issue, so please let me know if there's more information I can provide.
I believe the
cmd=ping
line shows different commands depending on exactly what the first command is that a client runs after the sync completes.Here's the master server's
INFO
:And the slave's
INFO
:And finally our slave's
redis.conf
: