nysenthil
commented
7 years ago Hi Amir, There was a C++ pointer bug in the Redis-Cluster wrapper client library that caused the PE to crash on a cluster master node outage. I fixed it in the Redis-Cluster wrapper code that we ship in the DPS toolkit. I also made changes in the DPS toolkit to gracefully handle a master node failure in a Redis cluster. Users can do the following in their Streams application code for continued operation when a Redis cluster auto-reconfigures itself during cluster master node outages.
1) In the no-sql-kv-store-servers.cfg file, users can now provide more than one Redis cluster master node information. (It is simply a good idea to specify all the master nodes and then all the slave nodes so that there are more possibilities for the DPS toolkit code to reconnect when a Redis cluster undergoes auto reconfiguration during a master node outage.)
Machine1:7001 Machine2:7002 Machine3:7003 Machine4:7004
2) In the Streams application code, whenever a DPS API is called, its return error code can be checked to see if there is a sudden connection failure in the Redis cluster. If a connection failure is detected, application code should call a local user written function to reconnect to the Redis cluster as shown below.
mutable uint64 err = 0ul; rstring myKey = “TestKey”; rstring myValue = “TestValue”; dpsPut(storeHandle, myKey, myValue, err);
// DPS_CONNECTION_ERROR = 102, DL_CONNECTION_ERROR = 501 if(err == 102ul || err == 501ul) { // When this function returns true, you are reconnected to the // Redis cluster and you can retry your failed API. // When it returns false, it means you are not connected to the Redis cluster. reconnectWithRedisCluster(100); }
public boolean reconnectWithRedisCluster(int32 maxConnectionAttemptCnt) { mutable int32 attemptCnt = 0;
while(dpsReconnect() == false) { if (++attemptCnt >= maxConnectionAttemptCnt) { // Giving up after the specified number of reconnection attempts. return(false); }
// Wait before trying to connect again.
block(2.0);}
// Successfully reconnected to the Redis cluster. return(true); }
3) We will do additional testing in our lab and make a new release in the master branch in another 3 weeks. In the meantime, a development version of the DPS toolkit with this fix can be obtained from here for your local testing.
https://github.com/IBMStreams/streamsx.dps/tree/develop
4) That open source Redis cluster client C++ wrapper pointer bug will certainly haunt other Redis cluster users outside the context of Streams as well. So, I alerted the author of that open source code to apply that simple fix in his original code.
https://github.com/shinberg/cpp-hiredis-cluster/issues/25#issuecomment-326489160
a-haroun
brandtol
Hello,
We have an issue when using the redis-cluster mode, when the node on which we are connected disconnects, the PE goes unhealthy! it does not try to connect to other nodes of the cluster. I am wondering what is the advantage of using a cluster in this case?
Thanks in advance, Amir.