Add redis sentinel-backed failover client

[furkansenharputlu]

It should be enabled by if redis.UniversalOptions.MasterName != "" like in redis.NewUniversalClient() function. It should be implemented and tested for all the following projects:

• [x] Gateway
• [x] Dashboard
• [x] Pump
• [x] MDCB

[buger]

Merged Gateway, now needs updating rest of projects (updated PR description to have checklist)

[maciejwojciechowski]

need to reopen. have some issues with pump and gateway. pump

tyk@MBP-Maciej tyk-pump (master) $ ./tyk-pump 
[Jan 24 10:28:57]  INFO main: ## Tyk Analytics Pump, v0.8.3 ##
[Jan 24 10:28:57]  INFO --> [REDIS] Creating single-node client
[Jan 24 10:28:57] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `version-check-pump`, `v0.8.3`, 
[Jan 24 10:28:57]  INFO mongo-pump: -- No max batch size set, defaulting to 10MB
[Jan 24 10:28:57]  INFO mongo-pump: -- No max document size set, defaulting to 10MB
[Jan 24 10:28:57]  WARN mongo-pump: Collection (tyk_analytics) already exists. Capping could result in data loss. Ignoring
[Jan 24 10:28:57]  INFO main: Init Pump: MongoDB Pump
[Jan 24 10:28:57]  INFO main: Init Pump: MongoDB Aggregate Pump
[Jan 24 10:28:57]  INFO mongo-pump-selective: -- No max batch size set, defaulting to 10MB
[Jan 24 10:28:57]  INFO mongo-pump-selective: -- No max document size set, defaulting to 10MB
[Jan 24 10:28:57]  INFO main: Init Pump: MongoDB Selective Pump
[Jan 24 10:28:57]  INFO main: 'dont_purge_uptime_data' set to false, attempting to start Uptime pump! MongoDB Pump
[Jan 24 10:28:57]  INFO mongo-pump: -- No max batch size set, defaulting to 10MB
[Jan 24 10:28:57]  INFO mongo-pump: -- No max document size set, defaulting to 10MB
[Jan 24 10:28:57]  WARN mongo-pump: Collection (tyk_uptime_analytics) already exists. Capping could result in data loss. Ignoring
[Jan 24 10:28:57]  INFO main: Init Uptime Pump: MongoDB Pump
[Jan 24 10:28:57]  INFO main: Starting purge loop @5(s)
[Jan 24 10:29:02]  WARN redis: Connection dropped, connecting..
[Jan 24 10:29:02] ERROR redis: Multi command failed: ERR unknown command `MULTI`, with args beginning with: 
[Jan 24 10:29:02]  WARN redis: Connection dropped, connecting..
[Jan 24 10:29:02] ERROR redis: Multi command failed: ERR unknown command `lrange`, with args beginning with: `analytics-tyk-system-analytics`, `0`, `-1`, 
[Jan 24 10:29:07] ERROR redis: Multi command failed: ERR unknown command `del`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:29:07] ERROR redis: Multi command failed: ERR unknown command `EXEC`, with args beginning with: 
[Jan 24 10:29:12] ERROR redis: Multi command failed: ERR unknown command `MULTI`, with args beginning with: 
[Jan 24 10:29:12] ERROR redis: Multi command failed: ERR unknown command `lrange`, with args beginning with: `host-checker:tyk-uptime-analytics`, `0`, `-1`, 

gateway

tyk@MBP-Maciej tyk (release-2.9) $ ./tyk 
[Jan 24 10:30:36]  INFO main: Tyk API Gateway v2.9.2
[Jan 24 10:30:36]  INFO --> [REDIS] Using single node mode
[Jan 24 10:30:36] ERROR webhooks: Failed to parse URL! parse : empty url
[Jan 24 10:30:36] ERROR webhooks: Init failed for this webhook, invalid URL, URL must be absolute
[Jan 24 10:30:36]  INFO main: Setting up analytics normaliser
[Jan 24 10:30:36]  INFO main: PIDFile location set to: ./tyk-gateway.pid
[Jan 24 10:30:36]  WARN main: The control_api_port should be changed for production
[Jan 24 10:30:36]  WARN checkup: Insecure configuration allowed: allow_insecure_configs: true
[Jan 24 10:30:36]  WARN checkup: File descriptor limit 256 too low for production use. Min 80000 recommended.
    This could have a significant negative impact on performance.
    Please refer to https://tyk.io/docs/deploy-tyk-premise-production/#file-handles for further guidance.
[Jan 24 10:30:36]  WARN checkup: Default secret `352d20ee67be67f6340b4c0605b044b7` should be changed for production.
[Jan 24 10:30:36]  WARN checkup: Default node_secret `352d20ee67be67f6340b4c0605b044b7` should be changed for production.
[Jan 24 10:30:36] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `host-checker:PollerActiveInstanceID`, `d54d122b-772f-4049-9fbe-99b754141e40`, 
[Jan 24 10:30:36]  INFO host-check-mgr: Starting Poller
[Jan 24 10:30:36] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `redis-test-0ec00adc-df98-4aeb-9fad-10cd39d17cb8`, `test`, 
[Jan 24 10:30:36]  INFO Waiting for Redis connection pools to be ready currRetry=0
[Jan 24 10:30:37] ERROR Error trying to get TTL: ERR unknown command `ttl`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:30:37] ERROR Could not EXPIRE key: ERR unknown command `expire`, with args beginning with: `analytics-tyk-system-analytics`, `60`, 
[Jan 24 10:30:37] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `redis-test-aa9e6b02-1ce1-402a-9f9b-7d9fd3fe5831`, `test`, 
[Jan 24 10:30:37]  INFO Waiting for Redis connection pools to be ready currRetry=1
[Jan 24 10:30:38] ERROR Error trying to get TTL: ERR unknown command `ttl`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:30:38] ERROR Could not EXPIRE key: ERR unknown command `expire`, with args beginning with: `analytics-tyk-system-analytics`, `60`, 
[Jan 24 10:30:38] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `redis-test-f4df34fb-8526-4181-b932-e2772f3ef2f8`, `test`, 
[Jan 24 10:30:38]  INFO Waiting for Redis connection pools to be ready currRetry=2
[Jan 24 10:30:39] ERROR Error trying to get TTL: ERR unknown command `ttl`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:30:39] ERROR Could not EXPIRE key: ERR unknown command `expire`, with args beginning with: `analytics-tyk-system-analytics`, `60`, 
[Jan 24 10:30:39] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `redis-test-cd085221-e8b6-4d56-9690-b4cba62e8fb3`, `test`, 
[Jan 24 10:30:39]  INFO Waiting for Redis connection pools to be ready currRetry=3
[Jan 24 10:30:40] ERROR Error trying to get TTL: ERR unknown command `ttl`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:30:40] ERROR Could not EXPIRE key: ERR unknown command `expire`, with args beginning with: `analytics-tyk-system-analytics`, `60`, 
[Jan 24 10:30:40] ERROR Error trying to set value: ERR unknown command `set`, with args beginning with: `redis-test-6fae07c9-e737-4269-abd5-561b846336ba`, `test`, 
[Jan 24 10:30:40]  INFO Waiting for Redis connection pools to be ready currRetry=4
[Jan 24 10:30:41] ERROR Error trying to get TTL: ERR unknown command `ttl`, with args beginning with: `analytics-tyk-system-analytics`, 
[Jan 24 10:30:41] ERROR Could not EXPIRE key: ERR unknown command `expire`, with args beginning with: `analytics-tyk-system-analytics`, `60`, 
[Jan 24 10:30:41] ERROR Waiting for Redis connection pools failed
[Jan 24 10:30:41] FATAL main: Redis connection pools are not ready. Exiting...

pump.txt tyk.txt

[furkansenharputlu]

The problem for the gateway is that it is not merged into release branch. Now, it is solved.

[buger]

@furkansenharputlu what about pump?

[furkansenharputlu]

I will send a PR for pump. Also, I will send small config fixes for all projects.

[maciejwojciechowski]

verified