ARS breaking at 10k and 5k clients + error logging doubt

[david-garay]

Dear Pier,

While doing some load testing experiments, I tried to generate 10k and 5k client configuration files. I get an error from stdout (see below), but no error log. Note: ~2.7k clients config generation do work, though.

I tried configuring log.ini in the config.d/ folder, but somehow didn't see any results (I configured: keys=stderr,/var/log/arouteserver.log, and with keys=stderr,file, with no joy).

Printout is:

ARouteServer 2019-06-29 16:05:33,433 INFO RPKI ROAs: 89343 total, 6347 from not allowed TAs, 70199 used for IPv4 ARouteServer 2019-06-29 16:05:33,644 INFO Updating entries from the ARIN Whois DB dump... ARouteServer 2019-06-29 16:05:37,073 INFO Updating entries from the Registro.br Whois DB dump... ARouteServer 2019-06-29 16:05:37,489 ERROR An error occurred: please refer to the log for details.

Clients file enclosed for background info. Any suggestions on how to troubleshoot further this issue logs.zip ? Br,

David

experiments.zip

[pierky]

Hello David,

could you try this? https://arouteserver.readthedocs.io/en/latest/USAGE.html#resources-and-memoryerror-error-messages

IIRC I found a similar behaviour in the past, while running some load test, and that fixed them (even though the specific MemoryError message is not mentioned in the output).

[david-garay]

Hi,

Thanks for the quick response. I don't seem to have any limits, at any rate ulimit is not installed and /etc/security/limits.conf has nothing on. Running with sudo would anyways perhaps disregard any configured limits, which in my case I confirmed not to fix the problem.

Perhaps enabling logging would enlighten the issue. I will look with more time if I can get it to work Thanks for your support!

David

[pierky]

Hello David,

for the log.ini configuration, I can suggest you to start from the one that is provided within the repo, then change the following lines:

• in [handlers] section, keys should be stderr,file (keys=stderr,file);
• in [logger_root] section, level should be DEBUG (level=DEBUG);
• in the same [logger_root] section, handlers shoud be stderr,file (handlers=stderr,file).

This should allow you to have an arouteserver.log file with all the DEBUG info collected by the tool during the execution.

Currently I'm trying to reproduce the issue. I'm using the asns and clients sections from the 5k YML file, and a simple general.yml file made of only the two mandatory values:

cfg:
  rs_as: 999

  # The route server's router ID.
  router_id: "192.0.2.2"

Could you share your general.yml configuration too? The one that I found in the experiments.zip file is not really compliant with ARouteServer, so I'm not sure which features are you using during the load tests.

Thanks

[pierky]

Hello @david-garay, I've been able to generate a config in the way described above. Looking forward to hearing your general.yml configuration. Thanks

[david-garay]

Hello @pierky,

Excuses for the the late response - took the kids to the park.

Enclosed the general.yml. Thanks in advance,

DAvid general.zip

[pierky]

Hi again @david-garay, I used your general.yml file (which is basically the default one distributed with the tool, just with a differente router ID and route server ASN, isn't it?) and I've been able to run the 5K test without problems.

Could you try running the same test again, with the logging settings that I've reported above?

I'll run the 10K load test ASAP.

[david-garay]

Dear @pierky, long time no see

I somehow am failing to generate the debug files, despite checking rights and fixing the log.ini file with the settings you suggested. I managed to get DEBUG output with the --logging-level flag, but still there is little to see.

I am attaching the standard output, for the remote possibility it might help. Cheers,

DAvid output_5k.log