PoC Redis with TLS as Caching backend for Openstack

[fdobrovolny]

Subtask for #457

As an SCS Developer, I want to try implementing PoC Redis with TLS so that I can identify the scope of work needed.

Definition of Ready:

• [ ] User Story is small enough to be finished within one sprint
• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [ ] All acceptance criteria are met
• [ ] Changes have been reviewed
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

Status

component	Redis cache PoC implemented
keystone	yes w/ TLS
nova	yes w/ TLS
neutron	yes w/ TLS
cinder	yes w/TLS
glance	yes w/TLS
placement	yes w/TLS
horizon	yes w/TLS
heat	yes w/TLS

yes - Redis backend implemented no - Redis cache implementation not started yet in progress - Redis cache implementattion in progress no cache - this component does not use caching

[berendt]

The target is to use Redis as an alternative where Memcached is currently used. Redis itself is already deployable with TLS.

At the moment Memcached is used as Caching Backend for the Keystone Auth Tokens. This is an example configuration from the aodh service. At the moment the encryption is done on the content layer, but not on the transport layer.

[keystone_authtoken]
service_type = alarming
memcache_security_strategy = ENCRYPT
memcache_secret_key = {{ memcache_secret_key }}
memcached_servers = {% for host in groups['memcached'] %}{{ 'api' | kolla_address(host) | put_address_in_context('memcache') }}:{{ memcached_port }}{% if not loop.last %},{% endif %}{% endfor %}

[fdobrovolny]

We identified the following services as using memcached:

• keystone
• nova
• neutron
• cinder
• glance
• Placement
• heat

One thing that we run into is that there might be an issue with the oslo.cache library, as it seems like it is not passing SSL options to the Redis client.

@berendt

Redis itself is already deployable with TLS.

Do you mean by this that Redis is capable of TLS or that there is already a role to deploy Redis with TLS?

[berendt]

We have https://github.com/openstack/kolla-ansible/tree/master/ansible/roles/redis in kolla-ansible for the Redis deployment. I think it should already support TSL.

[fdobrovolny]

I was aware of this role, but was wondering if there is any special osism or SCS role for redis, because I was not able to find any way to configure it using the kolla-ansible as the tls options are not in the config https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/redis/templates/redis.conf.j2, nor is it possible to specify them as command line argument. https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/redis/tasks/check-containers.yml#L4-L12

[berendt]

For OpenStack and required services by OpenStack we use Kolla-Ansible. This includes services like MariaDB, RabbitMQ, Redis, ... If TLS is not yet enabled there, then it is probably not there after all and must be implemented first. Then I had remembered it wrong.

[MatusJenca2]

A few notes regarding progess on this issue

Redis Encryption

• Redis does support transport TLS encryption, however it needs to be compiled with the option enabled
• Redis does NOT support encryption at rest in the standalone version according to this thread. It does so in AWS,GCP and Azure accordig to the official documentation

  Redis TLS configuration in Openstack

• There is a technical problem configuring Redis TLS in Openstack
• Openstack uses dogpile.cache to comunicate with redis backend
• dogpile.cache.redis requires a parameter connection_kwargs - a dictionary containing additional options, such as ssl_certfile
• Arguments to dogpile.cache.redis are provided through config using backend_arguments oprion - a dictionary of arguments
• Therefore, connection_kwargs must be provided in a recursive dictionary - something which is not supported by Openstack config
• A suggestion from Openstack IRC does not work

  [cache]
  backend = dogpile.cache.redis
  backend_argument=url:rediss://127.0.0.1:6443
  tls_certfile = /opt/stack/redis.crt
  tls_keyfile = /opt/stack/redis.key
  tls_cafile = /opt/stack/ca.crt
  tls_enabled = true

• tls_* options are listed in openstack documentation, however they don't seem to do anything
• A workaround suggestion would be adding a wrapper class into oslo.cache library, something like this

  
  # Preview only
  from dogpile.cache.backends import redis as redis_backend
  from oslo_cache import core
  __all__ = [
  'RedisTLSBackend'
  ]
  class RedisTLSBackend(redis_backend.RedisBackend):
  def __init__(self,arguments):
      arguments.update(
          connection_kwargs=dict(
              ssl_certfile=arguments['ssl_certfile'],
              ssl_keyfile=arguments['ssl_keyfile'],
              ssl_ca_certs=arguments['ssl_ca_certs']
          )
      )
      return super(RedisTLSBackend, self).__init__(arguments)

- This bypasses the need of recursive dictionary in config
- The config after this patch would look something like this

[cache] backend = oslo_cache.redis_tls enabled = True backend_argument = url:rediss://127.0.0.1:6443 backend_argument = ssl_certfile:/opt/stack/redis.crt backend_argument = ssl_keyfile:/opt/stack/redis.key backend_argument = ssl_ca_certs:/opt/stack/ca.crt

## SSL timeout for Nova
- Nova does not work with redis cache, even after proposed oslo patch (`ConnectionError: timed out`)
- According to [this issue](https://github.com/eventlet/eventlet/issues/692), eventlet library raises non-standart error  - `SSLError` as opposed to `socked.timeout` which is caught in [redis-py](https://github.com/redis/redis-py) 
- This was fixed in [this commit](https://github.com/eventlet/eventlet/commit/caf9f9983a43c537efff5c4c9ff1d543af6e1220) in eventlet, however it is not in pip repository yet
- The workaround would be installing eventlet from master branch
```bash
pip uninstall -y eventlet 
pip install https://github.com/eventlet/eventlet/archive/master.zip

Progress

• [x] Create the suggested patch in local environment i.e. devstack
• [x] Test locally with all relevant OS components
• [x] Ensure that TLS is really utilized

[MatusJenca2]

I managed to run a proof of concept in devstack:

• install devstack with heat enabled
• add the proposed fix to oslo.cache and install it to devstack venv /opt/stack/data/venv

  
  # oslo.cache/oslo_cache/backends/redis-tls.py
  from dogpile.cache.backends import redis as redis_backend
  from oslo_cache import core
  __all__ = [
  'RedisTLSBackend'
  ]
  class RedisTLSBackend(redis_backend.RedisBackend):
  def __init__(self,arguments):
      arguments.update(
          connection_kwargs=dict(
              ssl_certfile=arguments['ssl_certfile'],
              ssl_keyfile=arguments['ssl_keyfile'],
              ssl_ca_certs=arguments['ssl_ca_certs']
          )
      )
      return super(RedisTLSBackend, self).__init__(arguments)

- add the configuration option to `oslo.cache/oslo_cache/_opts.py` and `oslo.cache/setup.cfg`
```python
    ...    
    cfg.StrOpt('backend', default=_DEFAULT_BACKEND,
                   choices=['oslo_cache.memcache_pool',
                            'oslo_cache.dict',
                            'oslo_cache.mongo',
                            'oslo_cache.etcd3gw',
                            'oslo_cache.redis_tls',
    ...

[entry_points]
oslo.config.opts =
    oslo.cache = oslo_cache._opts:list_opts

dogpile.cache =
    oslo_cache.mongo = oslo_cache.backends.mongo:MongoCacheBackend
    oslo_cache.memcache_pool = oslo_cache.backends.memcache_pool:PooledMemcachedBackend
    oslo_cache.dict = oslo_cache.backends.dictionary:DictCacheBackend
    oslo_cache.etcd3gw = oslo_cache.backends.etcd3gw:Etcd3gwCacheBackend
    oslo_cache.redis_tls = oslo_cache.backends.redis_tls:RedisT

• install oslo.cache in edit mode

  pip install -e </path/to/oslo,cache>

• install django redis for Horizon pip install django-redis into the venv
• upgrade eventlet

  pip uninstall -y eventlet 
  pip install https://github.com/eventlet/eventlet/archive/master.zip

• edit all relarted .conf files e.g. /etc/keystone/keystone.conf

  
  backend = oslo_cache.redis_tls
  enabled = True
  backend_argument = url:rediss://127.0.0.1:6443 #path to the certificates
  backend_argument = ssl_certfile:/opt/stack/redis.crt
  backend_argument = ssl_keyfile:/opt/stack/redis.key
  backend_argument = ssl_ca_certs:/opt/stack/ca.crt

- restart devstack
```bash
sudo systemctl restart devstack@*

• edit /opt/stack/horizon/openstack_dashboard/settings.py

  CACHES = {
  "default": {
      "BACKEND": "django_redis.cache.RedisCache",
      "LOCATION": "rediss://127.0.0.1:6443",
      "OPTIONS": {
          "CONNECTION_POOL_KWARGS":{
              "ssl_ca_certs": "/opt/stack/ca.crt", # path to the certificates
              "ssl_certfile": "/opt/stack/redis.crt",
              "ssl_keyfile": "/opt/stack/redis.key"
              },
          "REDIS_CLIENT_KWARGS": {
              "ssl": True
  
          }
      }
  }
  }

• restart apache

  sudo systemctl restart apache2