Closed InputOutputZ closed 1 year ago
caojiajun
commented
1 year ago you should config upstream resource in rediss-cluster://xxx instead of redis-cluster://xxx
https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/auth/redis-resources.md
caojiajun
commented
1 year ago if client access proxy as a standalone redis：
cluster-mode-enable should setting falseproxy.cluster.mode.nodes=xxx should delete in configthis is frontend:
cluster-mode-enable:true means multi proxy nodes disguised as a redis cluster
proxy.cluster.mode.nodes=xxx means the proxy nodes list to proxy discovery each other.
tls-port means client access proxy in tls connect
proxy-frontend-tls-provider-class-name means the frontend tls provider
proxy.frontend.tls.xxxxxx configs means the frontend tls config
this is upstream/backend:
resource define the upstream redis addr, support redis:// redis-sentinel:// redis-cluster:// and rediss:// rediss-sentinel:// rediss-cluster:// proxy-upstream-tls-provider-class-name means the provider of upstream tlsproxy.upstream.tls.xxxx means the config of upstream tls
InputOutputZ
commented
1 year ago if client access proxy as a standalone redis：
cluster-mode-enableshould setting falseproxy.cluster.mode.nodes=xxxshould delete in configthis is frontend:
cluster-mode-enable:truemeans multi proxy nodes disguised as a redis clusterproxy.cluster.mode.nodes=xxxmeans the proxy nodes list to proxy discovery each other.tls-portmeans client access proxy in tls connectproxy-frontend-tls-provider-class-namemeans the frontend tls providerproxy.frontend.tls.xxxxxxconfigs means the frontend tls configthis is upstream/backend:
resourcedefine the upstream redis addr, supportredis://redis-sentinel://redis-cluster://andrediss://rediss-sentinel://rediss-cluster://proxy-upstream-tls-provider-class-namemeans the provider of upstream tlsproxy.upstream.tls.xxxxmeans the config of upstream tls
Hi caojiajun,
Thanks so much for clarifying it, finally I got it working. Just in case anyone else wondered how here is my working configuration with typical 6 nodes Redis Cluster establishing successful connection using TLS.
application.yml
server:
port: 7619
spring:
application:
name: camellia-redis-proxy-server
camellia-redis-proxy:
cluster-mode-enable: false
cport: -1
tls-port: 7619
console-port: 0
monitor-enable: false
netty-transport-mode: epoll
proxy-dynamic-conf-loader-class-name: com.netease.nim.camellia.redis.proxy.conf.FileBasedProxyDynamicConfLoader
proxy-frontend-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.frontend.DefaultProxyFrontendTlsProvider
password: mypassword
username: default
transpond:
local:
resource: rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384
redis-conf:
proxy-upstream-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.upstream.DefaultProxyUpstreamTlsProvidercamellia-redis-proxy.properties
proxy.frontend.tls.ca.cert.file.path=/etc/redis/cert/ca.pem
proxy.frontend.tls.cert.file.path=/etc/redis/cert/redis.pem
proxy.frontend.tls.key.file.path=/etc/redis/cert/rediskey.pem
proxy.upstream.tls.ca.cert.file.path=/etc/redis/cert/ca.pem
proxy.upstream.tls.cert.file.path=/etc/redis/cert/redis.pem
proxy.upstream.tls.key.file=/etc/redis/cert/rediskey.pem
proxy.upstream.tls.config=[{"resource":"rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384","ca.cert.file.path":"/etc/redis/cert/ca.pem","cert.file.path":"/etc/redis/cert/redis.pem","key.file.path":"/etc/redis/cert/rediskey.pem"}]@caojiajun btw predixy supports custom commands, and I was able to get client list command working and wonder if you have anything similar? as for now when I run client list I get " (error) syntax error "?
With thanks.
Zakaria.
caojiajun
commented
1 year ago https://redis.io/commands/client-list/ this command?
you want this command resp return the clients of proxy or clients of upstream redis?
caojiajun
commented
1 year ago your application.yml seems wrong indent, redis-conf is the child node of transpond
server:
port: 7619
spring:
application:
name: camellia-redis-proxy-server
camellia-redis-proxy:
password: mypassword
tls-port: 7619
proxy-frontend-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.frontend.DefaultProxyFrontendTlsProvider
netty-transport-mode: epoll
proxy-dynamic-conf-loader-class-name: com.netease.nim.camellia.redis.proxy.conf.FileBasedProxyDynamicConfLoader
transpond:
type: local
local:
type: simple
resource: rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384
redis-conf:
proxy-upstream-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.upstream.DefaultProxyUpstreamTlsProvider
https://redis.io/commands/client-list/this command?you want this command resp return the clients of proxy or clients of upstream redis?
Yes, I would love to have it?
caojiajun
commented
1 year ago clients of proxy or clients of upstream redis
which one?
InputOutputZ
commented
1 year ago
clients of proxyorclients of upstream rediswhich one?
clients of upstream or proxy, can I get both of them?
caojiajun
commented
1 year ago
clients of proxyorclients of upstream rediswhich one?
clients of upstream or proxy, can I get both of them?
the reply fields of clients of proxy and clients of upstream redis is different, it's a problem.
caojiajun
commented
1 year ago client list return clients of upstream redis which passthrough the reply of redis
define a new command proxy client list return clients of proxy
What do you think of this policy？
InputOutputZ
commented
1 year ago
client listreturnclients of upstream rediswhich passthrough the reply of redis define a new commandproxy client listreturnclients of proxyWhat do you think of this policy？
I think it's better to return clients of proxy because it's more accurate, and so far to retrieve client ip addresses connected via proxy is only possible through full transparent proxying and it's not simple to achieve e.g. using haproxy and kernel full transparent modules, having I struggled to get it working.
Btw with that being said, not sure if my conclusion is correct, but I think the TLS implementation is not 100% correct. I set my Redis Cluster to require SSL client authentication, and I connect from my local machine to the cluster using the proxy, and it doesn't require client certificate but only CA file, while it does from the upstream i.e. cluster server?
Here is screenshot comparisons:-
Connection to cluster using upstream terminal, and it's working as it should.
Connection to cluster using camellia proxy from my local machine, it should close connection unless valid client certificate was provided?
I wonder if it could be something wrong in my configuration actually?
Anyways, again thanks for all your work.
caojiajun
commented
1 year ago you should setting no-tls-port to 0 to disabled it, no-tls-port priority higher than tls-port
server:
port: 0
spring:
application:
name: camellia-redis-proxy-server
camellia-redis-proxy:
password: mypassword
tls-port: 7619
proxy-frontend-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.frontend.DefaultProxyFrontendTlsProvider
netty-transport-mode: epoll
proxy-dynamic-conf-loader-class-name: com.netease.nim.camellia.redis.proxy.conf.FileBasedProxyDynamicConfLoader
transpond:
type: local
local:
type: simple
resource: rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384
redis-conf:
proxy-upstream-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.upstream.DefaultProxyUpstreamTlsProviderif using haproxy before camellia-redis-proxy, from the perspective of proxy, client-ip is haproxy-ip rather than real client ip, unless camellia-redis-proxy support proxy-protocol, which currently not supported
caojiajun
commented
1 year ago caojiajun
commented
1 year ago in v1.2.14, you can use info clients to get client connect num
caojiajun
commented
1 year ago you should setting no-tls-port to 0 to disabled it, no-tls-port priority higher than tls-port
server: port: 0 spring: application: name: camellia-redis-proxy-server camellia-redis-proxy: password: mypassword tls-port: 7619 proxy-frontend-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.frontend.DefaultProxyFrontendTlsProvider netty-transport-mode: epoll proxy-dynamic-conf-loader-class-name: com.netease.nim.camellia.redis.proxy.conf.FileBasedProxyDynamicConfLoader transpond: type: local local: type: simple resource: rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384 redis-conf: proxy-upstream-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.upstream.DefaultProxyUpstreamTlsProvider
my fault, it's a bug SSL bidirectional authentication not effective in 1.2.14 i will fix it in v1.2.15
InputOutputZ
commented
1 year ago you should setting no-tls-port to 0 to disabled it, no-tls-port priority higher than tls-port
server: port: 0 spring: application: name: camellia-redis-proxy-server camellia-redis-proxy: password: mypassword tls-port: 7619 proxy-frontend-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.frontend.DefaultProxyFrontendTlsProvider netty-transport-mode: epoll proxy-dynamic-conf-loader-class-name: com.netease.nim.camellia.redis.proxy.conf.FileBasedProxyDynamicConfLoader transpond: type: local local: type: simple resource: rediss-cluster://default:mypassword@127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381,127.0.0.1:6382,127.0.0.1:6383,127.0.0.1:6384 redis-conf: proxy-upstream-tls-provider-class-name: com.netease.nim.camellia.redis.proxy.tls.upstream.DefaultProxyUpstreamTlsProvidermy fault, it's a bug SSL bidirectional authentication not effective in 1.2.14 i will fix it in v1.2.15
Perfect, if there is anything I can help with, please let me know.
Thanks so much for all your work and looking forward for a better Camellia Redis proxy.
caojiajun
commented
1 year ago the ssl bug has fixed in #133
these two new features, could you pls create PR to support it?
InputOutputZ
commented
1 year ago Btw, I created this service to run the proxy server in RHEL7, and it works perfectly, it could be good idea to include it in the package quick start guide: https://github.com/netease-im/camellia/blob/master/docs/redis-proxy/quickstart/quick-start-package.md.
Configuring steps.
1- Move uncompressed camellia-redis-proxy-1.2.14.tar.gz to /root/camelliaRP 2- vi /etc/systemd/system/camellia_redisproxy.service 3-
[Unit]
Description=Camellia Redis Proxy service.
After=network-online.target
Wants=network-online.target
[Service]
SuccessExitStatus=143
Type=simple
WorkingDirectory=/root/camelliaRP
ExecStart=/usr/bin/java -XX:+UseG1GC -Xms4096m -Xmx4096m -server org.springframework.boot.loader.JarLauncher
ExecStop=/bin/kill -15 $MAINPID
KillMode=mixed
[Install]
WantedBy=multi-user.target
4- systemctl enable camellia_redisproxy.service --now
With thanks.
InputOutputZ
commented
1 year ago the ssl bug has fixed in #133
131 #132 these two new features, could you pls create PR to support it?
Perfect, I will test #133 fix and update you there but for #131 and #132 I don't think I'm qualified enough to work on a PR? but maybe I'm, not sure, while I'm sure I can help with testing for now, since I don't fully understand the codebase?
caojiajun
commented
1 year ago you can create a PR to add it to quick_start documents, thank you very much!
caojiajun
commented
1 year ago the ssl bug has fixed in #133
131 #132 these two new features, could you pls create PR to support it?
Perfect, I will test #133 fix and update you there but for #131 and #132 I don't think I'm qualified enough to work on a PR? but maybe I'm, not sure, while I'm sure I can help with testing for now, since I don't fully understand the codebase?
thank you for your testing!
InputOutputZ
commented
1 year ago you can create a PR to add it to quick_start documents, thank you very much!
I will do this just now, and btw I was trying to find way to compile package for 1.15 and I couldn't. I wonder which SDK do you use for this? would it be something like Eclipse and if there is any quick guide or would you recommend for me to wait until you release 1.15 package?
caojiajun
commented
1 year ago InputOutputZ
commented
1 year ago Perfect, thanks for this.
Module: camellia-redis-proxy Version: 1.2.14 Content:
Hi there,
I just wanted to say thanks so much for the dev work to bring this proxy and I have a question, I wonder if there is anything wrong in my configuration since I couldn't figure out the reason I get these errors when I run start.sh
I checked redis cluster log and I get this error whenever I run start.sh
34824:S 18 Aug 2023 17:52:53.601 # Error accepting a client connection: error:0A00010B:SSL routines::wrong version numberThese are my configurations:-
application.yml
camellia-redis-proxy.properties
Looking forward for any hint, @caojiajun with thanks.
Zakaria.