Sentinel: Error: READONLY You can't write against a read only slave???

[CQBinh]

I have this redis replication with sentinel architect:

5 servers: .1, .2, .3, .4 and .5

• .1: nodejs app, redis master, redis sentinel
• .2: nodejs app, redis slave
• .3: nodejs app, redis slave
• .4: redis sentinel
• .5: redis sentinel

Sentinel handle failover as expected. redis-cli -h x.y.z.5 -p 26379 sentinel get-master-addr-by-name mymaster is always return .1 server.

My code to connect to this replication:

let sentinels = [
  { host: process.env.REDIS_SENTINEL_1, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_2, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_3, port: process.env.REDIS_PORT }
]
store = new Redis({
  name: 'mymaster',
  sentinels: sentinels
})
store.on('ready', () => {
  store.set('foo', new Date())
  store.get('foo', function (err, result) {
    console.log('redis: get foo: ', result)
  });
});

Then, .1 nodejs app run ok, but in .2 and .3, the error occurs with this log:

 events.js:161
       throw er; // Unhandled 'error' event
       ^

 Error: READONLY You can't write against a read only slave.

P/s: my redis servers are binding in both server IP and 127.0.0.1. Because, without 127.0.0.1, I meet this error:

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379

So, I guess that, my app isn't connect to redis server via sentinel, but it connect directly with localhost.

Any help are appriciate!

Update: log when I turn on the DEBUG=ioredis:*:

2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connecting -> connect
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connect -> ready
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:connection send 1 commands in offline queue
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis write command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | events.js:161
2|MyApp     |       throw er; // Unhandled 'error' event
2|MyApp     |       ^
2|MyApp     | Error: READONLY You can't write against a read only slave.
2|MyApp     |     at JavascriptReplyParser._parseResult (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:43:16)
2|MyApp     |     at JavascriptReplyParser.try_parsing (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:114:21)
2|MyApp     |     at JavascriptReplyParser.run (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:126:22)
2|MyApp     |     at JavascriptReplyParser.execute (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:107:10)
2|MyApp     |     at Socket.<anonymous> (/home/demo/demo_api/source/node_modules/redis/index.js:131:27)
2|MyApp     |     at emitOne (events.js:96:13)
2|MyApp     |     at Socket.emit (events.js:189:7)
2|MyApp     |     at readableAddChunk (_stream_readable.js:176:18)
2|MyApp     |     at Socket.Readable.push (_stream_readable.js:134:10)
2|MyApp     |     at TCP.onread (net.js:551:20)
2|MyApp     | [nodemon] app crashed - waiting for file changes before starting...
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)

Update 2: one more thing, I run the app in my local machine with the production environment and everything work like charm!!!

[luin]

Please try to add a statement (debug('resolved: %s:%s', resolved.host, resolved.port);) to log the resolved address:

in lib/connectors/sentinel_connector.js to see whether the sentinels return the correct master address.

[CQBinh]

It returns the same result with when I query via cli: x.y.z.1:6379.

Still returns the master IP.

[CQBinh]

Update 2: one more thing, I run the app in my local machine with the production environment and everything work like charm!!!

[luin]

I'm not sure but the issue may due to the configuration since it seems the correct address has been resolved to the client.

[CQBinh]

Yeah, finally found.

It's my mistake when an old code use the redis module and try to connect directly to localhost.

Replace it to ioredis instance and it work like charm :)

Anw, thank for your quickly help.