search

pulibrary / princeton_ansible

Ansible Roles and Playbooks for Princeton University Library
10 stars 2 forks source link

[loadtest] Cannot create sufficient load to stress Solr #5019

Closed maxkadel closed 1 month ago

maxkadel commented 2 months ago

Expected behavior

We can ramp up the number of threads and requests to Solr using JMeter to start hitting Solr-related errors, to see when we are genuinely stressing Solr.

Actual behavior

Before we start getting Solr errors, we get the JMeter error below:

Non HTTP response code: java.net.BindException/Non HTTP response message: Cannot assign requested address

See, for example, the errors on the test run on June 3, 2024

Steps to replicate

Follow the steps in the solr_load_testing README for testing reads.

Impact of this bug

We cannot truly stress the system, making our load tests less meaningful

Implementation notes

I am not actually sure whether the error stems from the loadtest machine or the Solr machine.

kayiwa commented 1 month ago

I bumped the memory and CPU (the error seems to suggest this is the problem) on the VM. Could you please test this @maxkadel

maxkadel commented 1 month ago

I tried testing this and am still seeing the same error (See today's test report).

~I'm going to try prefacing the command with the JVM_ARGS from @sandbergja's PR to see if they help (JVM_ARGS="-Xms2048m -Xmx2048m")~ did not impact this error

acozine commented 1 month ago

We got rid of the error by hitting an individual Solr VM instead of the load-balanced URL, after updating the firewall to allow the private .lib subnet to access the Solr port.