moleculerjs / moleculer

:rocket: Progressive microservices framework for Node.js
https://moleculer.services/
MIT License
6.16k stars 587 forks source link

Redis cacher broker.cacher.clean error (Redis cluster) #864

Closed xypherbo closed 3 years ago

xypherbo commented 3 years ago

Prerequisites

Please answer the following questions for yourself before submitting an issue.

Current Behavior

Built-in cache type Redis is trying to call DEL command to remove cache data from slave node when using Redis cluster and got error

Expected Behavior

It should scan and delete data from master node only

Failure Information

Action call is failed

Steps to Reproduce

  1. Using Redis cluster in cacher module (Cluster must have a slave)
  2. Call some action to have a cache inside Redis cluster
  3. Trying to call broker.cacher.clean(<cache key>)
  4. You will get error Redis 'scanDel' error. Pattern: MOL-<actionName>:<params> and del MOVED xxxx <master hostname>

Reproduce code snippet

const broker = new ServiceBroker({
    logger: console,
    cacher: {
      type: 'redis',
      options: {
        cluster: { 
           nodes: [<redis cluster nodes master+slave>] },
      },
    },
});

broker.createService({
    version: 1,
    name: "dummy",
    actions: {
       test: {
          cache: {
            keys: ['param1'],
          },
          params: {
            param1: { type: 'string' },
          },
          async handler(ctx) {
            return { success: 1}
          },
        },
      cleanCache: {
          async handler(ctx) {
             broker.cacher.clean('v1.dummy.test:<your param1>')
          },
      }
    }
});

Context

Failure Logs

  ioredis:cluster cluster slots result [0]: slots 2815~5544 served by [ '172.18.20.248:6379', '172.18.41.151:6379' ] +0ms
  ioredis:cluster cluster slots result [1]: slots 9017~11747 served by [ '172.18.20.248:6379', '172.18.41.151:6379' ] +1ms
  ioredis:cluster cluster slots result [2]: slots 8192~9016 served by [ '172.18.43.160:6379', '172.18.13.87:6379' ] +0ms
  ioredis:cluster cluster slots result [3]: slots 11748~16383 served by [ '172.18.43.160:6379', '172.18.13.87:6379' ] +0ms
  ioredis:cluster cluster slots result [4]: slots 0~2814 served by [ '172.18.6.41:6379', '172.18.24.204:6379' ] +1ms
  ioredis:cluster cluster slots result [5]: slots 5545~8191 served by [ '172.18.6.41:6379', '172.18.24.204:6379' ] +0ms

  ioredis:cluster:connectionPool   { host: '172.18.43.160', port: 6379, readOnly: false },
  ioredis:cluster:connectionPool   { host: '172.18.13.87', port: 6379, readOnly: true },
  ioredis:cluster:connectionPool   { host: '172.18.6.41', port: 6379, readOnly: false },
  ioredis:cluster:connectionPool   { host: '172.18.24.204', port: 6379, readOnly: true },
  ioredis:cluster:connectionPool   { host: '172.18.20.248', port: 6379, readOnly: false },
  ioredis:cluster:connectionPool   { host: '172.18.41.151', port: 6379, readOnly: true }

  ioredis:redis write command[172.18.20.248:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +28ms
  ioredis:redis write command[172.18.6.41:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +0ms
  ioredis:redis write command[172.18.24.204:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +1ms
  ioredis:redis write command[172.18.41.151:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +0ms
  ioredis:redis write command[172.18.43.160:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +0ms
  ioredis:redis write command[172.18.13.87:6379]: 0 -> scan([ '0', 'MATCH', '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54', 'COUNT', '100' ]) +1ms
  ioredis:redis write command[172.18.20.248:6379]: 0 -> del([ '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54' ]) +1ms
  ioredis:redis write command[172.18.41.151:6379]: 0 -> del([ '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54' ]) +1ms

{
  command: {
    name: 'del',
    args: [
      '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54'
    ]
  },
  level: 'error',
  message: 'del MOVED 2886 172.18.20.248:6379',
  stack: 'ReplyError: MOVED 2886 172.18.20.248:6379\n' +
    '    at parseError (/app/node_modules/redis-parser/lib/parser.js:179:12)\n' +
    '    at parseType (/app/node_modules/redis-parser/lib/parser.js:302:14)',
  timestamp: '2021-02-03T15:30:51.005Z'
}

{
  level: 'error',
  nodeID: 'd6f19037dafa-446',
  ns: '',
  mod: 'cacher',
  command: {
    name: 'del',
    args: [
      '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54'
    ]
  },
  pattern: '{MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54',
  message: "Redis 'scanDel' error. Pattern: {MOL}-v3.comment._getByPublicId:5b028e4a673f81000fb040e7|601abce409aa8bc670434d54",
  timestamp: '2021-02-03T15:30:51.007Z'
}

Suggestion

Inside moleculer/src/cachers/redis.js

          _clusterScanDel(pattern) {
        const scanDelPromises = [];
        const nodes = this.client.nodes(); <-- here has to put option "master" into

        nodes.forEach(node => {
            scanDelPromises.push(this._nodeScanDel(node, pattern));
        });

        return this.broker.Promise.all(scanDelPromises);
    }
intech commented 3 years ago

@xypherbo fixed in this PR https://github.com/moleculerjs/moleculer/pull/865