Add support for Redis over TLS

[avigadmi]

Redis is now created with TLS enabled by default on IBM Cloud. When trying to run streams job with DPS for Redis over SSL I get the following errors:

18 Feb 2018 07:42:36.073 [7241] ERROR #splapptrc,J[2],P[5],redis_nw89pvc390n.redis_nw89pvc390n.RedisWriter,DistributedProcessStore M[DistributedProcessStore.cpp:connectToDatabase:342] - Cannot connect to database. Details: 'Unable to authenticate to the redis server(s). Possible connection breakage. Server closed the connection'. 18 Feb 2018 07:42:36.077 [7241] ERROR #splapptrc,J[2],P[5],redis_nw89pvc390n.redis_nw89pvc390n.RedisWriter,spl_operator M[RedisWriter.cpp:processRaw:83] - Exception in operator redis_nw89pvc390n.redis_nw89pvc390n.RedisWriter in port 0 18 Feb 2018 07:42:36.077 [7241] ERROR #splapptrc,J[2],P[5],redis_nw89pvc390n.redis_nw89pvc390n.RedisWriter,spl_operator M[RedisWriter.cpp:processRaw:84] - Exception: SPL::SPLRuntimeException (Cannot connect to database. Details: 'Unable to authenticate to the redis server(s). Possible connection breakage. Server closed the connection'.) at DistributedProcessStore::connectToDatabase

[brandtol]

The DPS toolkit uses the hiredis C client to connect to Redis. As far as I see this client does not support SSL. So for the time being please consider disabling this option in the Redis service. We need to investigate further what options are available to support SSL. Switching the client library may result in larger effort.

[brandtol]

Currently, I see these options :

1) Fork the hiredis client and add SSL support using OpenSSL (which is preinstalled anyway)

2) Use the qredsiclient which supports SSL (https://github.com/uglide/qredisclient) Cons: has dependencies to the Qt library, comes under GPL v2 license

3) Try to package some SSL tunneling software (like stunnel) with the toolkit and start that up on first connection Cons: unclear if this will work, might be a hack

4) Add a new Java operator to the DPS toolkit that uses the lettuce client (https://github.com/lettuce-io/lettuce-core), and supports limited functions (get/put). Lettuce is under Apache license, so should be fine.

I would prefer 1) or 4) Are there other options ?

[erelsibm]

I agree that 1 and 4 seem to be preferable, please note that streams designer is using hmset command so we'll need to have it available if option 4 is selected. There might be an advantage to option 4 as java skills are more common than C++ and lettuce seems to be more active and recommended than hiredis.

[ddebrunner]

See #31 also.

[erelsibm]

@ddebrunner at this time dps does support authentication and we've been using it for several months. the issue is now with SSL support which is currently not supported.

[mikespicer]

1 and 4 seem attractive. This toolkit is used in scenarios with high rates so performance should be a consideration.

[nysenthil]

After 20 months of silence on this issue, one of our top IBM Streams customers in the U.S expressed their dire need for the TLS support in the streamsx.dps toolkit as they are doing a wholesale migration of several dozens of their on-prem IBM Streams applications to AWS. Many of their applications heavily use the streamsx.dps toolkit for using Redis as an external data store. They have to now move their on-prem Redis as well to the AWS Elasticache Redis service which can only work with clients that support TLS. This is very much like the case as reported above by @avigadmi. To help that top customer, I have been doing several experiments for the past three months to support TLS in the streamsx.dps toolkit for it to work with Redis public cloud services such as the IBM Compose Redis, AWS Elasticache Redis etc. I'm beginning to see some early success with that effort. If my thorough tests over the next few weeks yield reliable results with the major Redis public cloud service providers, I will incorporate the additional code to support TLS for Redis and release a new version of streamsx.dps toolkit in 1Q2020.

[anouri]

support TLS for Redis is a very important feature of streamsx.dps to connect to the Redis cluster. We are working on this issue and we will deliver this feature in the next release.

[nysenthil]

Almost three years after this issue was opened, a few favorable things happened in the Redis land. In Aug/2020, hiredis library v1.0.0 came out with full TLS/SSL support for the single instance Redis. Soon after that, a C++11 based redis-plus-plus library (that wraps hiredis) added full TLS/SSL support for the redis-cluster. Both of those events helped me to spend the next two months in a partial re-write of the IBM Streams DPS toolkit with 5000 lines of code to add TLS/SSL support for the single instance Redis and for the clustered Redis. I just now released a new DPS toolkit version 4.1.2 that takes care of this issue.