search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.92k stars 1.06k forks source link

[Platform] Getting "too many clients already" issue while connecting to ysqlsh #8416

Open VijiYB opened 3 years ago

VijiYB commented 3 years ago

Sometimes getting "too many clients already" health alert while connecting to ysqlsh for some nodes. Was running postgresqlkeyvalue sample apps on that universe.

Error executing command ['ssh', 'yugabyte@10.150.1.107', '-p', '54422', '-o', 'StrictHostKeyChecking no', '-o', 'ConnectTimeout=10', '-o', 'UserKnownHostsFile /dev/null', '-o', 'LogLevel ERROR', '-i', u'/opt/yugabyte/yugaware/data/keys/109e95b5-bf08-4a8f-a7fb-2d2866865e15/yb-gcp-config-key.pem', 'set -o pipefail; /home/yugabyte/tserver/bin/ysqlsh -h 10.150.1.107 -p 5433 -U yugabyte -c "\conninfo"']: ysqlsh: FATAL: sorry, too many clients already

VijiYB commented 3 years ago

Observed this problem in 2.7.1 (dev portal) @iSignal Please add your comments as you were debugging on this issue.

iSignal commented 3 years ago

Sample apps was running from 10.9.6.228 with

[centos@ip-10-9-6-228 ~]$ ps -ef | egrep sample
centos    38455  17588  0 18:34 pts/13   00:00:00 grep -E --color=auto sample
root      88370  88353  0 00:07 ?        00:11:01 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload SqlInserts --nodes 10.150.1.107:5433,172.151.27.97:5433,172.151.19.105:5433,172.151.17.99:5433
root      88537  88520 99 00:07 ?        1-17:46:05 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload CassandraKeyValue --nodes 10.150.1.107:9042,172.151.27.97:9042,172.151.19.105:9042,172.151.17.99:9042

causing many postgres processes (295) in "idle" status as seen at https://gist.github.com/iSignal/c6cce1239d6f64ca3c17005922d80aa8

Killing the sample apps processes reduced the number of postgres processes to <5.

There were repeated postgres errors of the form in the db postgres logs - https://gist.github.com/iSignal/2003f756551edf5ed7cd38c41f05799e

2021-05-12 12:29:11.518 UTC [3265] ERROR:  duplicate key value violates unique constraint "postgresqlkeyvalue_pkey"
2021-05-12 12:29:11.518 UTC [3265] STATEMENT:  INSERT INTO postgresqlkeyvalue (k, v) VALUES ($1, $2)

/mnt/d0/pg_data/ysql_pg.conf showed postgres max_connections setting at 300

@gurushant-yb : I saw that you made some changes to yb-sample-apps recently, is this something you have run into or have some idea about?

gurushant-yb commented 3 years ago

Sample apps was running from 10.9.6.228 with

[centos@ip-10-9-6-228 ~]$ ps -ef | egrep sample
centos    38455  17588  0 18:34 pts/13   00:00:00 grep -E --color=auto sample
root      88370  88353  0 00:07 ?        00:11:01 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload SqlInserts --nodes 10.150.1.107:5433,172.151.27.97:5433,172.151.19.105:5433,172.151.17.99:5433
root      88537  88520 99 00:07 ?        1-17:46:05 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload CassandraKeyValue --nodes 10.150.1.107:9042,172.151.27.97:9042,172.151.19.105:9042,172.151.17.99:9042

causing many postgres processes (295) in "idle" status as seen at https://gist.github.com/iSignal/c6cce1239d6f64ca3c17005922d80aa8

Killing the sample apps processes reduced the number of postgres processes to <5.

There were repeated postgres errors of the form in the db postgres logs - https://gist.github.com/iSignal/2003f756551edf5ed7cd38c41f05799e

2021-05-12 12:29:11.518 UTC [3265] ERROR:  duplicate key value violates unique constraint "postgresqlkeyvalue_pkey"
2021-05-12 12:29:11.518 UTC [3265] STATEMENT:  INSERT INTO postgresqlkeyvalue (k, v) VALUES ($1, $2)

/mnt/d0/pg_data/ysql_pg.conf showed postgres max_connections setting at 300

@gurushant-yb : I saw that you made some changes to yb-sample-apps recently, is this something you have run into or have some idea about?

@iSignal : I have observed 'unique constraint violation' , this happens when you have a --num_unique_keys < --num_writes

streddy-yb commented 3 years ago

hi @gurushant-yb - Can you add input validation & throw an error to handle --num_unique_keys < --num_writes case?

gurushant-yb commented 3 years ago

hi @gurushant-yb - Can you add input validation & throw an error to handle --num_unique_keys < --num_writes case?

Hi @streddy-yb @iSignal : Shall I set num_unique_keys = num_writes , by adding following condition instead of throwing an IllegalArgumentException and aborting the execution: cc: @kkg-yb 


 if(AppBase.appConfig.numKeysToWrite != AppBase.appConfig.numUniqueKeysToWrite){
      AppBase.appConfig.numUniqueKeysToWrite = AppBase.appConfig.numKeysToWrite;
    }