Application process dies after intensive requests.

[rclabs-dev]

I'm using RPKI Validator to validate updates in a period of 24 hours using RESTful. The updates are in the order of 1.800,000 records. Occurs that after checking than more 100,000 records the application process dies and the logs do not indicate what happened (in fact is random but always occurred after 100,000).

I tried to increase the memory for use (jvm.memory.maximum) and set the parameter "rtr.close-on-error" to false, but without success.

I set my application to makes requisitions via localhost in periods from 10, 20, 30, 40 and 50 milliseconds, but it is indifferent, the error occurs always.

I'm running it using Ubuntu 12.04 with Java 8.

There is how to enable auditing for errors at runtime, as well as to disable access and validator LOGs to improve performance?

Thanks.

[AlexanderBand]

Could you please let us know which version you are using. Is it the latest one, v. 2.22?

[rclabs-dev]

Alexander, thanks for your reply.

Yes, I'm using v2.22.

Thanks!

[timbru]

Hi,

The API that you are using was not really built with this kind of load in mind to be honest. Normally the evaluation of route validity is done inside an RPKI enabled router or for example Bird or Quagga - the RPKI validator then just communicates "validated roa prefixes" using rpki-rtr typically.

Can you explain a bit more about what your use case is?

All that being said, if you have an error log for us or could provide some example code that we can use to replicate the issue here, then we can have a look to see if this is something we can easily fix. Even though this is not the primary use of the validator we obviously want to fix things if it can be done with reasonable effort.

If you prefer you can contact me directly at tim@ripe.net.

Cheers Tim

[rclabs-dev]

Hi Tim, Thank you for your reply. About the case that I opened in github. I'm PhD student at Lisbon University. In the moment I'm researching some BGP security problems. We have tried to improve the BGP Security gaps using SDN scenaries.  For this, we developed a application using ONOS Controller to improve several things regarding BGP security. We have done some tests and one of them is that through a emulated network we send using ExaBGP a 24-hour routing information dataset to SDN Controller (in true to the other Speaker in emulated network, but which cross a OpenFlow switch controlled by ONOS). In the SDN Controller our application gets the BGP UPDATE and validates the ROA through queries in the RPKI  Validator using RESTful via localhost connection. But always after a long validate time the RPKI Validator process simply died and the LOGs don't said nothing about any fail. Thus, I hope cleared up the motive by which we are using RESTful instead RTR protocol. Is more simple for us. We exported all the ROAs from RPKI Validator, so we made a simulation to obtain the desired results, but we also want to simulate a real situation. But due the RPKI Validator dies, we didn't conclude the tests.  We are grateful if you give us some help about it. Only to information, in Lisbon University my advisor is Professor Fernando Ramos. Thanks a lot. Best regards, Regivaldo Costa Doe Sangue. Doe Vida.

  De: Tim Bruijnzeels <notifications@github.com>

Para: RIPE-NCC/rpki-validator rpki-validator@noreply.github.com Cc: rcosta regivaldocosta@yahoo.com.br; Author author@noreply.github.com Enviadas: Terça-feira, 2 de Agosto de 2016 11:10 Assunto: Re: [RIPE-NCC/rpki-validator] Application process dies after intensive requests. (#19)

Hi,The API that you are using was not really built with this kind of load in mind to be honest. Normally the evaluation of route validity is done inside an RPKI enabled router or for example Bird or Quagga - the RPKI validator then just communicates "validated roa prefixes" using rpki-rtr typically.Can you explain a bit more about what your use case is?All that being said, if you have an error log for us or could provide some example code that we can use to replicate the issue here, then we can have a look to see if this is something we can easily fix. Even though this is not the primary use of the validator we obviously want to fix things if it can be done with reasonable effort.If you prefer you can contact me directly at tim@ripe.net.Cheers Tim— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[timbru]

Hi,

We have a hunch that the problem may be related to the Linux Out-of-memory (OOM) killer. Could you check if you see suspicious messages in /var/log/messages?

We recommend that you make sure that the JVM has at least 1.5GB of heap memory as configured by default. However, because Java has other memory pools (some of which are used fir each concurrent thread) the total memory usage is higher than this. If this exceeds your system memory, the process will be killed.

You could try to lower the memory usage for the JVM in the configuration file, but then java itself may run out of memory or become slow.

In your situation I would rather recommend that you give the JVM more than enough memory, possibly even more (2GB) to save it from having to do garbage collection when you subject it to high load - and you give it enough CPUs, and make sure that your OS still has memory to spare.

How much memory the JVM will need exactly is fairly complicated to deduct*. But if as a rule of fist you make sure that the OS has twice the memory that you configure in the validator config file you should be okay.

Please let us know if this solved your problem.

*: Java7 -> https://plumbr.eu/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx

[rclabs-dev]

Tim, Thank you for your attention. I didn't solves this problem, yet.  The computacional resource have 16GB of RAM and I increased memory in the configuration file until 4GB, but the problem repeats.

At the moment I'm in holiday and when go back, I'll make more tests regarding the memory as you suggests, as well as the monitoring the others things in the operating system.  When I conclude the test I'll leaving a feedback here. Many thanks. Best regards, Regivaldo Costa Doe Sangue. Doe Vida.

  De: Tim Bruijnzeels <notifications@github.com>

Para: RIPE-NCC/rpki-validator rpki-validator@noreply.github.com Cc: rcosta regivaldocosta@yahoo.com.br; Author author@noreply.github.com Enviadas: Terça-feira, 9 de Agosto de 2016 12:33 Assunto: Re: [RIPE-NCC/rpki-validator] Application process dies after intensive requests. (#19)

Hi,We have a hunch that the problem may be related to the Linux Out-of-memory (OOM) killer. Could you check if you see suspicious messages in /var/log/messages?We recommend that you make sure that the JVM has at least 1.5GB of heap memory as configured by default. However, because Java has other memory pools (some of which are used fir each concurrent thread) the total memory usage is higher than this. If this exceeds your system memory, the process will be killed.You could try to lower the memory usage for the JVM in the configuration file, but then java itself may run out of memory or become slow.In your situation I would rather recommend that you give the JVM more than enough memory, possibly even more (2GB) to save it from having to do garbage collection when you subject it to high load - and you give it enough CPUs, and make sure that your OS still has memory to spare.How much memory the JVM will need exactly is fairly complicated to deduct. But if as a rule of fist you make sure that the OS has twice the memory that you configure in the validator config file you should be okay.Please let us know if this solved your problem. : Java7 -> https://plumbr.eu/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[timbru]

Hi,

We have found some issues with resource usage for the API and created a new development version. Can you try it out and let us know if it fixed the issue? Binary is here:

https://rrdp.ripe.net/certification/content/static/validator/rpki-validator-app-888-dist.tar.gz

[rclabs-dev]

Hi Tim, Great! Thank you for your feedback. I already downloaded and still today I will start the test and leave feedback. Many thanks. Best regards, Regivaldo Costa Doe Sangue. Doe Vida.

  De: Tim Bruijnzeels <notifications@github.com>

Para: RIPE-NCC/rpki-validator rpki-validator@noreply.github.com Cc: rcosta regivaldocosta@yahoo.com.br; Author author@noreply.github.com Enviadas: Terça-feira, 23 de Agosto de 2016 6:27 Assunto: Re: [RIPE-NCC/rpki-validator] Application process dies after intensive requests. (#19)

Hi,We have found some issues with resource usage for the API and created a new development version. Can you try it out and let us know if it fixed the issue? Binary is here:https://rrdp.ripe.net/certification/content/static/validator/rpki-validator-app-888-dist.tar.gz— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[rclabs-dev]

Dear Tim, I tested the new version of application in the same enviorment. As indicated by you, I also made a verification in the syslog and in fact was possible to identified some thing. So, through syslog became possible to see that the RPKI Validator always died because the system killed the process when occurred a lack memory (you can see in the screenshot where  the app was started with PID 4072 and in other screenshot we can view  which after +/- 3h the system killed this process by lack of memory). However, it was clear to me that there were improvements in application because before it, the system always killed the process between 100.000 and 200.000 processed records (or BGP UPDATE) and with the new version, I repeated the process and by three time the process died always above of 1.000.000 processed records. I will to change the computacional resource to ensure which the lack memory problem will not more occur. Your help was much important. So my thanks for their willingness to help. Best regards, Regivaldo Costa Doe Sangue. Doe Vida.

  De: Tim Bruijnzeels <notifications@github.com>

Para: RIPE-NCC/rpki-validator rpki-validator@noreply.github.com Cc: rcosta regivaldocosta@yahoo.com.br; Author author@noreply.github.com Enviadas: Terça-feira, 23 de Agosto de 2016 6:27 Assunto: Re: [RIPE-NCC/rpki-validator] Application process dies after intensive requests. (#19)

Hi,We have found some issues with resource usage for the API and created a new development version. Can you try it out and let us know if it fixed the issue? Binary is here:https://rrdp.ripe.net/certification/content/static/validator/rpki-validator-app-888-dist.tar.gz— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[timbru]

Hi,

If the OS is killing the process that means that the Java Virtual Machine is using more memory than is available on your system.

Adding more memory to your system really is the preferred solution because the application is needs quite a bit of memory to work properly. However, you can try to lower the JVM memory settings in the config file a bit - if you get OutOfMemory exceptions in the validator log you will know you have lowered it too much..

We plan to do major work on the validator in 2017 to lower its memory footprint, but that won't be available short term.

Cheers Tim