TLS Client Connection Support for Memcached

Neaj-Morshad-101 commented 1 month ago

Hello, I have successfully set up a Memcached server with TLS enabled. The server is running with the following configuration:

Memcached is running with the -Z flag to enable TLS and using the following options:

ssl_chain_cert=/etc/ssl/certs/ssl_cert.pem
ssl_key=/etc/ssl/private/ssl_key.pem

Memcached version: 1.6.31

TLS Configuration: Enabled using the -Z option
Environment: Docker

Steps followed:

Set up the Memcached server with TLS enabled using the following Docker setup:

docker run -d \
  --name memcached-server \
  -p 11211:11211 \
  -v /path/to/server.crt:/etc/ssl/certs/ssl_cert.pem \
  -v /path/to/server.key:/etc/ssl/private/ssl_key.pem \
  memcached:1.6.31 \
  memcached -Z \
  -o ssl_chain_cert=/etc/ssl/certs/ssl_cert.pem \
  -o ssl_key=/etc/ssl/private/ssl_key.pem

Certificates are generated using the following commands - openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=memcached-server/O=kubedb" openssl genrsa -out server.key 2048 openssl req -new -key server.key -out server.csr -subj "/CN=memcached-server/O=kubedb" openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365
TLS certificate (server.crt) and key (server.key) are configured and mounted.

Tried connecting to the server using a client with TLS (e.g., openssl s_client -connect localhost:11211 -CAfile /path/to/ca.crt -servername memcached-server).

CONNECTED(00000003)
depth=1 CN = memcached, O = kubedb
verify return:1
depth=0 CN = memcached-server, O = kubedb
verify return:1
---
Certificate chain
0 s:CN = memcached-server, O = kubedb
i:CN = memcached, O = kubedb
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Oct 21 05:51:28 2024 GMT; NotAfter: Oct 21 05:51:28 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC2DCCAcACFBqxLTWYsUnoYzRck297U5vOrtlWMA0GCSqGSIb3DQEBCwUAMCUx
EjAQBgNVBAMMCW1lbWNhY2hlZDEPMA0GA1UECgwGa3ViZWRiMB4XDTI0MTAyMTA1
NTEyOFoXDTI1MTAyMTA1NTEyOFowLDEZMBcGA1UEAwwQbWVtY2FjaGVkLXNlcnZl
cjEPMA0GA1UECgwGa3ViZWRiMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA45frIRQXMaDpMgx/nMtKd5X+q1oeCWnrlj7yVfrOM6udOlPXsMKjy646AcLW
JoAsS5s986+oMDq7cSJ8OPnQkT8nBNwYP6xNzLbsxSCerhYeFHsPyCCoj/HUN6ZU
0nHcGKaExkO2obM3SVssr0b5UCp6spkychQpBCteS/FAxAoI25ejjeg21LQqYRqq
eMYXgVaBhSnv7y4yM5GDcU0U6YAyrJTKPM4IUdfX6JlbBkZ7becll/3HKFlZIZ45
q9II789C8iU4fcucSMP6wgdz8LStwSmPzKT75705Mb4ZF1bG/McMjonP/1kS/s5c
TLQhrLLJQMrAaAtIamMoz392jQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCnpVOc
JXEF/NtsKQ31ZYN8Ua3u0OC3kRmRhZa2F5EkjcQCEYOVJTE6AUJ+LbT8Em3n9O/J
JeRmwtIboOH0ozreR9WPsT4mQ5nIoJVLFkVmmfazB6kvCcFch34n4KMHLfsomgSp
nRgg2g5VLsHedijJm27oL6Myb8CEjT7ueXmAxhdKcv6emM8whTuCxDlw5Qac8746
HHvMsFFIOo4cNYA9I39rS1ELOOYsI5F+Mc4CQFP1Szt/QH88RpIMIIrUCORFFUqr
tQV1S9ZSZjIeGIxyGafWdJV5JdvA0XzbMP9vCE0UIVW8ouMCaBKXySAPGz9d76dP
13PI4NZSuIJH8y/T
-----END CERTIFICATE-----
subject=CN = memcached-server, O = kubedb
issuer=CN = memcached, O = kubedb
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1288 bytes and written 398 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Session-ID: AFF46F24A77D269E8CCA17ADFB80BD7144074BAD78764A9E3D180D2DA0E0FD61
Session-ID-ctx: 
Resumption PSK: B44DA57700911E1939883EC28A06D17A14DCE42EB009CFC73D63B8BB027CA7E12DC8ACCB46C1E4C84A217B38EA9E6859
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - dc c8 4d 59 13 b1 8f 48-9d c6 e3 fd c6 37 b6 f2   ..MY...H.....7..
0010 - f2 a7 f8 41 23 32 1c 50-ff d0 c0 47 98 eb ed 7e   ...A#2.P...G...~
0020 - 1a 83 e8 f5 91 94 a3 98-74 b9 04 e4 2f 51 e6 0f   ........t.../Q..
0030 - f9 80 e3 91 fc f1 aa b4-a4 3d b7 5e 67 4b 9d 3e   .........=.^gK.>
0040 - 9a cd 55 37 9f 07 15 6f-da 0e 9a 28 52 46 9d 1c   ..U7...o...(RF..
0050 - 48 b9 45 17 4d 92 88 5b-eb aa f3 be e1 7e f1 05   H.E.M..[.....~..
0060 - 7f 29 e3 66 df 50 58 81-0f 72 4c b4 1b dd 8c 5a   .).f.PX..rL....Z
0070 - 3d 41 4e 72 10 01 39 f9-41 65 9c 47 fd de c9 0b   =ANr..9.Ae.G....
0080 - b1 f0 4a 8c 89 64 b0 d6-ef 09 40 84 1e af 30 b5   ..J..d....@...0.
0090 - 51 b5 af 6d bf bb 47 c8-7e eb c9 62 43 c9 ef a0   Q..m..G.~..bC...
00a0 - 5c d1 2f b8 b1 ec e9 b1-a3 75 bd fd 02 9b 2b bb   \./......u....+.
00b0 - ad cd 7d 8b cd bb 17 ce-20 30 1a 12 77 14 09 88   ..}..... 0..w...
00c0 - 00 f3 4e d8 9d b0 44 58-e8 78 7c 10 ad e9 a0 2a   ..N...DX.x|....*

Start Time: 1729514937
Timeout   : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Session-ID: DDD92EB596EDAD13AAE6668AC71E6413042CA98A06F1A1C4DA3BC0AA826184BF
Session-ID-ctx: 
Resumption PSK: A5E6ED6248DAC4C596E9DB1F4A6457A6E6133D0C507C5BB7B12397C1BE405C3D55836482AE03702921D147C7CDCE778B
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - dc c8 4d 59 13 b1 8f 48-9d c6 e3 fd c6 37 b6 f2   ..MY...H.....7..
0010 - ca e8 16 9e de 89 4e d2-8d 9c 2a d4 41 15 1b b7   ......N...*.A...
0020 - 59 aa 77 f3 d1 90 be 3c-cb cc 68 62 a1 74 71 56   Y.w....<..hb.tqV
0030 - 5c d0 b8 aa 73 10 71 27-f4 ae b0 e9 0a 50 20 ba   \...s.q'.....P .
0040 - 93 5c fa ae 4d f5 88 a0-26 c9 85 14 2e de 02 c2   .\..M...&.......
0050 - 50 99 8b e8 ec 93 55 3f-73 c9 80 1e a0 9e 32 81   P.....U?s.....2.
0060 - 68 43 67 2f d5 ff fa 6f-ab 84 2f ec 81 d0 f5 ba   hCg/...o../.....
0070 - 03 5f 84 1b 0c c8 94 81-90 44 16 83 0d a9 88 e3   ._.......D......
0080 - 62 71 e8 f5 2a de 55 93-b8 fd 1d 68 e2 1d e9 6e   bq..*.U....h...n
0090 - c8 b2 f0 94 d6 28 79 bb-ec 5d fe 81 63 65 8d b2   .....(y..]..ce..
00a0 - 2a 5c a3 b4 3e 7a 79 f6-05 7c 4d 8b bc f1 ad ea   *\..>zy..|M.....
00b0 - 57 60 dd ce 5f 6c ad f0-06 19 79 d9 e6 84 5d 9f   W`.._l....y...].
00c0 - 82 b4 ba 7c e2 7a 44 51-b1 42 24 5a 9b 04 16 e9   ...|.zDQ.B$Z....

Start Time: 1729514937
Timeout   : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK

 echo "stats settings" | openssl s_client -connect localhost:11211 -CAfile /path/to/ca.crt -servername memcached-server -quiet ```

The command returns the Memcached stats as expected:
```bash
depth=1 CN = memcached, O = kubedb
verify return:1
depth=0 CN = memcached-server, O = kubedb
verify return:1
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 0
STAT inter NULL
STAT verbosity 1
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT shutdown_command no
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT num_threads_per_udp 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT auth_enabled_ascii no
STAT item_size_max 1048576
STAT maxconns_fast yes
STAT hashpower_init 0
STAT slab_reassign yes
STAT slab_automove 1
STAT slab_automove_ratio 0.80
STAT slab_automove_window 30
STAT slab_chunk_max 524288
STAT lru_crawler yes
STAT lru_crawler_sleep 100
STAT lru_crawler_tocrawl 0
STAT tail_repair_time 0
STAT flush_enabled yes
STAT dump_enabled yes
STAT hash_algorithm murmur3
STAT lru_maintainer_thread yes
STAT lru_segmented yes
STAT hot_lru_pct 20
STAT warm_lru_pct 40
STAT hot_max_factor 0.20
STAT warm_max_factor 2.00
STAT temp_lru no
STAT temporary_ttl 61
STAT idle_timeout 0
STAT watcher_logbuf_size 262144
STAT worker_logbuf_size 65536
STAT read_buf_mem_limit 0
STAT track_sizes no
STAT inline_ascii_response no
STAT ext_item_size 512
STAT ext_item_age 4294967295
STAT ext_low_ttl 0
STAT ext_recache_rate 2000
STAT ext_wbuf_size 4194304
STAT ext_compact_under 0
STAT ext_drop_under 0
STAT ext_max_sleep 1000000
STAT ext_max_frag 0.80
STAT slab_automove_freeratio 0.010
STAT ext_drop_unread no
STAT ssl_enabled yes
STAT ssl_chain_cert /etc/ssl/certs/ssl_cert.pem
STAT ssl_key /etc/ssl/private/ssl_key.pem
STAT ssl_verify_mode 0
STAT ssl_keyformat 1
STAT ssl_ciphers NULL
STAT ssl_ca_cert NULL
STAT ssl_wbuf_size 16384
STAT ssl_session_cache no
STAT ssl_kernel_tls no
STAT ssl_min_version tlsv1.2
STAT num_napi_ids (null)
STAT memory_file (null)
STAT client_flags_size 4
END

I have been using telnet to connect with memcached server, But with tls enabled, I can't connect like earlier, what flags / options are there for connecting with tls enabled? example:

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
Connection closed by foreign host.

I have been unable to find a way to connect to the Memcached server using TLS from available clients. It seems that there is no flag or option in the clients to establish a TLS-secured connection.

Could anyone provide guidance or information about TLS support in Memcached clients in this schenario?

Thank you!

dormando commented 1 month ago

What client are you trying to use? Not all clients support TLS; you might need to contact the client author there and ask about adding support.

You can use socat to connect to memcached when it's running TLS. There're a lot of tutorials online about how to do this.

It's also possible to use our proxy as a TLS gateway of sorts: Use your non-TLS client to connect to a proxy running on localhost, and then the proxy connects to memcached over TLS. I need to add a small guide for this as we've been working on updating the docs and I haven't gotten there yet: https://docs.memcached.org/features/proxy/ - if this could work let me know and I'll prioritize the additional guide.

Evanraisul commented 1 month ago

We try to use > socat , but the set command is not working. Always getting "CLIENT_ERROR bad data chunk" this error. Other commands are working perfectly. Here is the logs:

socat -d -d - OPENSSL:localhost:11211,cert=/home/evan/go/src/evanraisul/memcached_TLS/client.crt,key=/home/evan/go/src/evanraisul/memcached_TLS/client.key,cafile=/home/evan/go/src/evanraisul/memcached_TLS/ca.crt,verify=1

2024/10/23 11:54:30 socat[11928] N reading from and writing to stdio 2024/10/23 11:54:30 socat[11928] W OpenSSL: Warning: this implementation does not check CRLs 2024/10/23 11:54:30 socat[11928] N successfully connected from local address AF=2 127.0.0.1:50280 2024/10/23 11:54:30 socat[11928] N trusting certificate, commonName matches 2024/10/23 11:54:30 socat[11928] N SSL proto version used: TLSv1.3 2024/10/23 11:54:30 socat[11928] N SSL connection using TLS_AES_256_GCM_SHA384 2024/10/23 11:54:30 socat[11928] N SSL connection compression "none" 2024/10/23 11:54:30 socat[11928] N SSL connection expansion "none" 2024/10/23 11:54:30 socat[11928] N starting data transfer loop with FDs [0,1] and [6,6] set foo 0 999 3 bar

CLIENT_ERROR bad data chunk 2024/10/23 11:54:45 socat[11928] N write(1, 0x60edddcfe000, 29) completed get foo END 2024/10/23 11:54:52 socat[11928] N write(1, 0x60edddcfe000, 5) completed version VERSION 1.6.31

dormando commented 1 month ago

The data portion of a set needs to end with a \r\n, which is automatically added using telnet. ie: the command is actually: set foo 0 999 3\r\n bar\r\n

guessing you're inputting \n into socat or something. you can try copy/pasting or something else; this isn't really a support forum for socat :)

dormando commented 1 month ago

you can also set up socat as a listener doing tcp<->tls and then telnet to the socat. do you need this for debugging purposes or for something else?

Evanraisul commented 1 month ago

Actually, we want to run Memcached on Kubernetes. Already running, but need to run Memcached with TLS support. Before this we were using telnet, Memcached was running perfectly without TLS.

socat -d -d - TCP:localhost:11211

2024/10/23 13:03:55 socat[13841] N reading from and writing to stdio 2024/10/23 13:03:55 socat[13841] N opening connection to AF=2 127.0.0.1:11211 2024/10/23 13:03:55 socat[13841] N successfully connected from local address AF=2 127.0.0.1:50664 2024/10/23 13:03:55 socat[13841] N starting data transfer loop with FDs [0,1] and [5,5] version 2024/10/23 13:03:58 socat[13841] N write(5, 0x6268e8496000, 8) completed VERSION 1.6.31 2024/10/23 13:03:58 socat[13841] N write(1, 0x6268e8496000, 16) completed set foo 0 999 3\r\n bar\r\n2024/10/23 13:04:09 socat[13841] N write(5, 0x6268e8496000, 20) completed CLIENT_ERROR bad command line format 2024/10/23 13:04:09 socat[13841] N write(1, 0x6268e8496000, 38) completed version 2024/10/23 13:04:24 socat[13841] N write(5, 0x6268e8496000, 15) completed ERROR 2024/10/23 13:04:24 socat[13841] N write(1, 0x6268e8496000, 7) completed

dormando commented 1 month ago

You're just using telnet to test it right? Are you using a real client to access it normally?

You can tell memcached to listen on multiple ports, some without TLS, ie: -l 0.0.0.0:11211 -l notls:127.0.0.1:11511 if that helps?

Evanraisul commented 1 month ago

I am using gomemcache Client to access it. But, before using that client I have to make sure that from Memcached site it is all okay and the TLS is working correctly. That's why before I was using telnet. But, telnet doesn't support TLS. So, I need "socat" or others to make sure how a normal user will connect to Memcached with TLS.

No, some ports without TLS will not help me. I need all (one or multiple) ports with TLS.

Evanraisul commented 1 month ago

Setting up socat as a listener doing tcp<->tls and then telnet to the socat, worked successfully for Memcached with TLS Support. Thanks for the support.

memcached / memcached

TLS Client Connection Support for Memcached #1181

Memcached version: 1.6.31

Steps followed: