zloster
commented
5 years ago Actually the above problem seems to be bigger. I've checked the logs from the last tree tfb-status runs: two from Citrine and one from Azure(round 17 branch).
# Download and unzip the result.zip files
grep -r "unable to connect to" ./results/20180925161627/ > problems.txtThere is a lot of stuff with the following Cannot assign requested address:
./results/20180925161627/ffead-cpp-apache-mysql/query/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested address
./results/20180925161627/ffead-cpp-apache-mysql/json/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested address
./results/20180925161627/ffead-cpp-apache-mysql/json/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested address
./results/20180925161627/api_hour-mysql/fortune/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested address
./results/20180925161627/api_hour-mysql/fortune/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested address
./results/20180925161627/api_hour-mysql/update/raw.txt:unable to connect to 10.0.0.1:8080 Cannot assign requested addressIf the server application is not responding we should see only Connection refused:
./results/20180925161627/wizzardo-http/plaintext/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refused
./results/20180925161627/tornado-postgresql-raw/db/raw.txt:unable to connect to 10.0.0.1:8080 Connection refusedIn this particular case the JVM has died while running wizarrdo-http.
The servlet3.raw log from the same run (20180925) is nice illustration of the problem:
---------------------------------------------------------
Running Primer plaintext
wrk -H 'Host: 10.0.0.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 5 -c 8 --timeout 8 -t 8 http://10.0.0.1:8080/plaintext
---------------------------------------------------------
unable to connect to 10.0.0.1:8080 Cannot assign requested address
---------------------------------------------------------
Running Warmup plaintext
wrk -H 'Host: 10.0.0.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 512 --timeout 8 -t 28 http://10.0.0.1:8080/plaintext
---------------------------------------------------------
Running 15s test @ http://10.0.0.1:8080/plaintext
28 threads and 512 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.06ms 4.12ms 204.58ms 96.60%
Req/Sec 4.49k 540.59 10.67k 88.42%
Latency Distribution
50% 3.86ms
75% 5.64ms
90% 7.23ms
99% 8.79ms
1883257 requests in 15.10s, 232.02MB read
Requests/sec: 124729.70
Transfer/sec: 15.37MB
---------------------------------------------------------The primer doesn't produce result but the first warmup is OK?!
Another manifestation of the same problem are the logs of various frameworks that are complaining with the same Cannot assign requested address message when trying to talk to the database. I've used grep -r "Cannot assign requested address" ./results/20180925161627/ > problem-db-25.txt
msmith-techempower
nathantippy
NateBrady23
bhauer
michaelhixson
jsongte
joanhey
I've noticed that:
hexagon-undertow-mongodbhexagon-undertow-postgresqlare having problems. The following message is visible in the
raw.txt:Here are some example logs from
citrine:The
.dockerfilesare looking OK i.e. no obvious differences.