Spike: Select another persistent solution for Caching.

[balhar-jakub]

Is your feature request related to a problem? Please describe. The dockerized deployment of the Zowe needs to have a Caching service supporting the run off-platform.

Describe the solution you'd like Understand the options we have and figure out which one to use and have a small POC with it to verify our expectations

Consider ActiveMQ, ETCD, MongoDB, Redis, Memcached, Hazelcast, ehcache, etc.

Some requirements to consider:

• security - secure transport
• startup performance - ideally want to embed solution into java app rather than having to startup another process/container
• performance - native clients perform better than REST
• is persistence a hard requirement? We need resilience if a node crashes, but may not need to keep data through a full restart

Describe alternatives you've considered

[anton-brezina]

We ( @jandadav and @balhar-jakub ) would like to refine the following and then meet with external expert.

• How do we persist the data?
• How to best embed Active MQ to caching service?
• Mapping from key-value to MQ
• Learn more about how the Active MQ handles the concurrency
• How to build asolution that is tolerant to integrity issues?

This will be something between a spike and a PoC in the end. A performance benchmark could be a part of this as well for our awarenes and to spot obvious problems soon.

[CarsonCook]

ActiveMQ: Built for messaging, not specifically for caching/persistence. Architecture would be one service instance sends a persistent message to other service instances which then use that message to update their own local cache stores. Another option would be to use a message receiver that stores the data using JDBC, but this is SQL and is overhead when we could just simply communicate with a database directly.

Activemq can run on a wide variety of environments, including z/OS Can run embedded within Spring boot, however it may be best practices to not run embedded (research this further if activemq is chosen) Native Java client as well as REST API available Can configure to transport with ssl

Redis: Redis should be used on Linux systems, with no real support on other systems. Hostbridge offers redis on Z as a commercial option. Can easily configure maximum memory to use, and has reject, TTL, and best effort less recently used eviction. Redis Cluster and Redis Sentinel are options for high availability. Works in memory, with periodic disk dumps or a disk log for persistence. Key-value data model that can handle data structures Single-threaded, can scale horizontally via clustering spring-boot-starter-data-redis 3 clause BSD license Joe Winchester has worked with Hostbridge in the past, potentially go via him in figuring out if Zowe can use Redis Can run with TLS transport

memcached: Seems to have support on ibm z least recently used eviction Just works in server memory, no disk storage Memcached is optimized to use less memory for metadata. Only stores strings, so must serialize data structures Multi-threaded, can scale with more computer resources memcached-spring-boot-starter In general less features and less flexible than redis Revised BSD license Can transport with TLS

Hazelcast: Works in memory, can write to disk via a db but have to set up db and write code to export cache to db Seems very similar in design to memcached, but with more features and interfaces, like RESTful API, consistency configuration, partitioning, replication, etc. Apache license Open source version does not have secure transport, enterprise license does

MongoDB Most popular noSQL DB, "JSON like document storage" NoSchema so very flexible with data to be stored Built in replication/failover and horizontal scaling Only available on zLinux (plus distributed platforms) https://spring.io/projects/spring-data-mongodb Only Community edition is free to use (lacks encrypting stored data and connection to LDAP) Data stored to disk by default but can switch to "in-memory engine" General impression when comparing with Redis is that MongoDB is faster to learn but less flexible. (general stack overflow impression is that Redis is typically the more suitable solution unless rapid development is essential) SSPL license (Previously GPL) Can transport with TLS

Etcd Can't run on z/OS Can run embedded in spring Key-value store Can use TLS in transport Made to store k8s objects, not flexible data types Only stores data on disk, not in RAM, may not be performant

[balhar-jakub]

So basically we are looking at either:

• Redis
• Memcached
• Hazelcast

[jandadav]

Hazelcast IMDG cluster with Dynamic discovery and Persistence

Test repository: https://github.com/jandadav/hazelcast-cluster

Purpose

To provide a POC for implementing caching service under Zowe/Api Mediation Layer

Requirements

Brief summary of caching service requirements and how Hazelcast fits into each one

Embeddability / Number of processes to manage

Can be embedded to springboot app, reducing number of processes to just the caching api.

Runs on zOS

Pure java impl, should run without problem. Not tested

Data Persistence

Data not persisted, opensource does not provide mechanisms for persistent data.

Handlers can be made and hooked on internal evenr model, that can serialize the data on events (inserted, deleted, .....) and custom serialization logic can be implemented.

Discovery of clients

Hazelcast provides native integration with Eureka discovery. The integration works very well and is easy to configure.

Normally, Hazelcast uses multicast for discovery, which might be problematic for customers in production environment. Luckilly the discovery plugin can circumvent this problem. Other discovery plugins are provided for other discovery solutions as well.

Native Client

Yes, but with caveat that secure transport is not available. Can be implemented but effort is significant.

Security

Opensource solution does not offer TLS security on transport. Alternative solution usin AT-TLS or NGINX/TOMCAT as reverse proxy providing TLS should be possible (not tested)

Custom implementation could be made.

Customizing the nio networking: Effort is significant. Experimented with customization of Hazelcast instance and the mechanism is done in the repo.

Replacing the networking with blocking secure socket: Effort unknown

Performance

Profiling of memmory vs data should be done to understand capacity.

Capacity

As above.

Multiple clients combine heap and extend capacity.

Replication

Very fast replication in cluster. Backups are kept so when instance goes down data does not get lost.

Spring Integration

Nice integration wiht SpringBoot. Autoconfigure or custom creation can be done with beans. Very Spring like!

License

Apache License, Version 2.0

Commercial support

Available

Summary

While some aspects still remain to be tested, Hazelcast seems like solid option that fits on almost all requirements.

The only challenge is securing the transport. Some options are availabe and need to be evaluated.

[lchudinov]

Speaking of Redis: I ported Redis 5 (including Redis Cluster and Redis Sentinel) to z/os last year. Just let me know if you're interested in the port.

[jackjia-ibm]

I don't have experience on Hazelcast and just learned from this thread so I would rather not comment on it. David's comments look great to me. Except for Mongodb, which I'm not sure, all other choices listed are cross platforms as far I learned. To compile etcd, we can expect a go compiler on Z really soon, so I also consider it as cross-platform solution.

For other options:

• activemq: this is my 3rd choice with the benefit that we can control the deployment completely with relatively small efforts. But the performance on z/OS is my concern. I would select a tool created with C/C++ as first choice, then golang which can be compiled to binary on the platform. Java based is ok to me but not as ideal when considering starting time and runtime performance.
• memcached: I wouldn't recommend this as a good choice except this tool itself has been evolved during the year. The main reason is it's a pure memory solution, we need to find our own way to persistent data. This conflicts with the purpose here to persist state data. Also maintaining a memcached cluster needs experience, or need to be properly documented. There are some catches I wouldn't expect every sysadm can pick up and do it correctly at the first setup. Memcached has it's own data retention mechanism, If we also implement our own data retention mechanism, it increases complexity to troubleshooting issues like why data is missing from the store. The main scenario for memcached is using it as "cache", which means there is another more reliable persistence method, and data in memecached can be wiped out and re-initialized without any data lost.
• redis: it's my second choice. With default setup, the data is snapshot to disk to avoid data lost. The default interval is very short, but you still have a chance to lose data updated within that interval. This is a trade-off for performance gain by sacrificing part of data integrity. It's good for storing non-critical data. If we say the data stored in caching service should be 100% accurate, this won't be an ideal choice. If we configure redis to keep AOF persistence logs, it will decrease performance and will be very likely conflicting with the customer's existing redis instances. I also have concern when customer asks to use their existing redis cluster.
• mongodb: I wouldn't recommend this for now except for have other extended goals to use mongodb. It feels too complicated to maintain a mongodb cluster comparing to what we want to achieve, storing small amount of data. The performance of mongodb itself and the relationship among the entities hit me badly in the past. But if we just want to store key/value data, we should be ok. I heard people questioned on choosing mongodb, since they may say I already have PostgreSQL, or mysql, or DB2 in our environment, can you use that. It will be hard to argue with customers we need a dedicated mongodb for zowe.
• etcd: This is still my first choice. It's lightweight and popular on distributed systems. The data storage is trustworthy and we don't expect data lost in production environment. The gRPC protocol itself can potentially increase our connection performance, but we need better way to take advantage of it. I believe we can take full control how etcd is deployed and customers could be relatively easily convinced we need our own dedicated etcd instances, since we can see etcd in every k8s cluster.

Overall there are 2 questions are most important to me:

1. Can we tolerate data lost when the persistent instance fails? Do we want to keep 100% ensured data integrity?
2. Can we control the deployment of the cluster? A customer deployed/maintained cluster would be a concern for me to provide support for it, except for we say if you use your own redis/mongodb cluster, we don't provide support on that. If we share a cluster with other product, we have more chance to be challenged about how it can be configured in general and how the security should be setup.

Ideally, we can enable other developers to create plugins to extend persistent method, so people can create DB2 extension if they want. I understand this is out of the topic here, but could benefit us for long term.

[CarsonCook]

Redis

Requirements

Embeddability/Number of processes to manage

Need to run a separate redis server process in addition to the caching service.

Runs on zOS

Will with Hostbridge support, a commercial option.

Data persistence

Periodic disk dumps or logging to persist data. Can (although unlikely) lose data between the dump intervals, as a trade-off for the better performance. Ideal for non-critical data.

Discovery of clients

Redis cluster: each node auto-discovery other nodes Redis Sentinel: use sentinel nodes to discovery other nodes

Native client

Yes

Security

Has TLS support.

Performance

Seems to be highly performant, more-so than memcached. May not scale as well as Hazelcast.

Capacity

Will need to profile. Can optimize key length to make more use of memory. Can partition data to increase available memory, at the cost of multi-key transactions.

Replication

Fast replication via Redis Cluster or Redis Sentinel. Uses leader-follower architecture with exact copies of the leader.

Spring integration

Nice integration with Spring Boot. Typical Spring bean configuration.

License

BSD license

Commercial support

Available via Hostbridge on z/OS, or Redis Enterprise.

Summary

Redis is very popular and should be easy to spin up as a PoC. May not be able to scale as well as we want, and inability to run on z/OS without a commercial license may be a problem. Data durability is also a concern, although 100% durability may not be a true requirement.

Memcached

Requirements

Embeddability/Number of processes to manage

Separate process to run a memcached server.

Runs on zOS

Docker images exist for IBM Z, but then obviously this requires docker.

Data persistence

Works in memory, no real data persistence. Can configure to re-start without losing data, but this isn't "real" persistence.

Discovery of clients

Native client

Yes

Security

TLS support.

Performance

Seems to be beat by redis and Hazelcast, but would need to profile.

Capacity

Will need to profile. Combines memory across web servers.

Replication

Does not support replication. Would have to manually replicate data.

Spring integration

Has spring boot integration. Bean configuration available.

License

BSD license

Commercial support

No, rely on opensource support.

Summary

Older technology that isn't easy to use. Limited features compared to other options, would require complex administration and/or custom extension.

ETCD

Requirements

Embeddability/Number of processes to manage

Requires another process to run the etcd server.

Runs on zOS

No. However there may be a go compiler ready on Z soon, which would allow etcd to run on z/OS

Data persistence

Yes, writes to disk by default.

Discovery of clients

Has discovery client implementation in spring boot.

Native client

Yes

Security

TLS in transport.

Performance

We can expect it to be performant, though will need to profile it. Some benchmarks show it being as performant as redis despite not working in memory.

Capacity

To avoid performance degradation, capacity is limited by RAM, 8 GB typically enough.

Replication

Cluster for availability, after cluster consensus request is accepted and replicated across the cluster.

Spring integration

Spring boot integration.

License

Apache license 2.0

Commercial support

No, rely on open source support.

Summary

Ideal option for containerized environment, given its use in Kubernetes. Really good, robust features, and should be performant. Only problem is lack of z/OS options.

[jandadav]

@lchudinov Yes please, is there a repository where we can take a look?

[jandadav]

So far we are looking at the following decision

[lchudinov]

@jandadav I am waiting for approval to push the Redis port to the Github.

[lchudinov]

By the way, "SSL/TLS is supported by Redis starting with version 6": https://redis.io/topics/encryption. Redis 5, which we have, doesn't support TLS.

[lchudinov]

@jandadav Please checkout https://github.com/lchudinov/redis and run ./build-zos.sh script. By default it builds 64-bit ASCII version. Let me know if you have any questions.

[jandadav]

from @Ichudinov: By the way, Golang for z/os goes now with a doc named "porting guide", where they demonstrate how to port etcd so etcd now should work on z/os, though I haven't tried Leonty Chudinov go version go1.14.9 zos/s390x

[jandadav]

The question about additional address spaces has been discussed on HA call, and it has not been identified as a showstopper to have more address spaces, even though not ideal.

[balhar-jakub]

Just based on the previous knowledge and comparison my personal preferences are: 1) Hazelcast - AT-TLS on platform and Reverse proxy setup off-platform 2) Redis - Off-platform works well for on platform support may be needed 3) ETCD - More or less same as Redis, but in Redis case we may get additional support outside of Zowe

[balhar-jakub]

@jandadav Could you please add a Row uses ZIIP to your table? It seems that only Hazelcast does from the Hazelcast, Redis and ETCD

[jalel01]

Quick update from the discussion with Redis yesterday. Redis indicated an interest to participate in Zowe. As such, they have a working port for z/os which they will share with us in the coming weeks once ready. Support details for this port are yet to be finalized.

Once we have the port for z/os we will proceed with a PoC. In the meantime, we may look closer at the Redis off-platform solution.

[balhar-jakub]

@jalel01 Can we close this issue now as the decision has been made? And that is to do the PoC using Redis.