Periodic accumulation of service load with an error "NATS error. 'Maximum Payload Violation'"

[JeStasG]

Hello, please help! We have the following configuration ServiceBroker config.js:

const SafeJSONSerializer = require(appRoot + '/helpers/safe-json-serializer.js');

module.exports = {
    namespace: process.env.NAME,
    nodeID: process.env.NAME +'_'+ process.pid,
    logger: true,
    logLevel: "warn",
    transporter: "nats.local:4222",
    requestTimeout: 1200000,
    requestRetry: 0,
    cacher: "Memory",
    registry: {
        strategy: "RoundRobin"
    },
    circuitBreaker: {
        enabled: true,
        threshold: 0.5,
        minRequestCount: 20,
        windowTime: 60,
        halfOpenTime: 5 * 1000,
        check: err => err && err.code >= 500
    },
    serializer: new SafeJSONSerializer(),
    metrics: true,
    statistics: true,
    heartbeatInterval: 5,
    heartbeatTimeout: 15
}

Start to be made several instans by means of PM2 pm2.json:

{
    "apps" : [{
        "script"      : "../index.js",
        "name"        : "moleculer_prod",
        "instances"   : "8",
        "exec_mode"   : "cluster",
        "watch"       : true,
        "ignore_watch": ["node_modules"],
        "max_memory_restart" : "250M",
        "env"         : {
          "NAME": "prod",
        }
      }]
  }

The main script index.js:

const { ServiceBroker } = require('moleculer');
const brokerConfig = require('config');
const servicesList= [
    "web.js",
    "data.js",
    "auth.js",
   ....
]
const broker = new ServiceBroker(brokerConfig);
broker.loadServices(appRoot + '/services', servicesList);
broker.start();

Each service of a web.js(moleculer-web) has its own port assigned on the basis of the default port + process.env.NODE_APP_INSTANCE. With the subsequent balancing through NGINX.

Periodically falls with an error:

NATS client is reconnecting.
NATS error. 'Maximum Payload Violation'

Please tell me where to look if the configuration NATS service in maximum payload is:

#maximum payload
max_payload 1048576

The indicated error does not appear immediately, but after 2-3 days after starting the services.

[JeStasG]

How it looks visually through prometheus+grafana: 1.Moleculer: 2.NATS: 3.At 9:58 - "pm2 restart all" and functioning restored.

[JeStasG]

In fact, servicesList is done in order to run its own set of services on each server. For example:

registry = [
    {
         service: "someService1.js",
         server: "someServer1"
    },
    .....
    {
        service: "someService2.js",
        server: "someServer2"
    }
];
let servicesList = registry.filter({ server: process.env.SERVER }).map(i => i.service);

broker.loadServices(appRoot + '/services', servicesList);

Not documented (in the documentation I did not see), but through trial and error it turned out to be a working solution.

[icebob]

It means you reached the limit of NATS packet. By default is 1MB. You can increase it in NATS server config, but maybe you had better turn on verbose log in NATS server and check which packet is huge & reaches the limit.

[JeStasG]

Thanks, max_payload corrected and watch the result. Can you please tell me if there is a method for restarting the service without restarting the broker?

[icebob]

Check the hot-reload code: https://github.com/moleculerjs/moleculer/blob/master/src/service-broker.js#L605

[JeStasG]

I mean force by analogy broker.loadService. For example - broker.reloadService("service name").

[xynon]

@JeStasG That's exactly what the hot-reload does.

utils.clearRequireCache(service.__filename);

To clear the require.cache - so you can require the service file again - if you dont have changes then you dont need this line

then

return this.destroyService(service).then(() =>
    this.loadService(service.__filename));

Destroy services do Service._stop and remove the service from the broker registry then you have a clean setup to load the service again.

[JeStasG]

Thanks!

[kevyworks]

For some reason, hotReloadService is cached haven't tested it further but, correct source change is loaded until specifically doing a broker.stop and reloading it..