search

aerospike / aerospike-client-go

Aerospike Client Go
Apache License 2.0
429 stars 199 forks source link

Panic query_objects_executor #353

Closed un000 closed 3 years ago

un000 commented 3 years ago

Server version 5.6.0.5

{"level":"error","ts":"2021-06-01T21:46:35.295Z","caller":"grpc/update.go:18", "error":"error updating subscription data: error operating record: ResultCode: TIMEOUT, Iteration: 0, InDoubt: false, Node: <nil>: command execution timed out on client: See `Policy.Timeout`\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILAB2021-06-01T21:46:35.296113703Z LE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool2021-06-01T21:46:35.296113703Z is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no conne2021-06-01T21:46:35.296113703Z ctions were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available\nResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available"}
{"level":"info","ts":"2021-06-01T21:46:37.426Z","caller":"aerospike/logger.go:19","msg":"Node `BB94D643559A1A8 10.10.2.231:3000` refresh failed: `ResultCode: NO_AVAILABLE_CONNECTIONS_TO_NODE, Iteration: 0, InDoubt: false, Node: <nil>: connection pool is empty. This happens when either all connection are in-use already, or no connections were available`","version":"4ca13d95","system":"aerospike"}
{"level":"info","ts":"2021-06-01T21:46:37.426Z","caller":"aerospike/logger.go:19","msg":"Tend finished. Live node count changes from 3 to 2","version":"4ca13d95","system":"aerospike"}
{"level":"info","ts":"2021-06-01T21:46:37.426Z","caller":"aerospike/logger.go:19","msg":"Tending took 5.002756293s, while your requested ClientPolicy.TendInterval is 1s. Tends are slower than the interval, and may be falling behind the changes in the cluster.","version":"4ca13d95","system":"aerospike"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xa2a4af]

goroutine 4257322 [running]:
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).executeAt(0xc001cc8000, 0x12326c8, 0xc001cc8000, 0xc00ac8c1b0, 0xc00ac8c101, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.0.2/command.go:2041 +0x2ef
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).execute(0xc001cc8000, 0x12326c8, 0xc001cc8000, 0xc001cc8001, 0x405cc5, 0xd)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.0.2/command.go:1970 +0xda
github.com/aerospike/aerospike-client-go/v5.(*baseMultiCommand).execute(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.0.2/multi_command.go:366
github.com/aerospike/aerospike-client-go/v5.(*queryPartitionObjectsCommand).Execute(0xc001cc8000, 0xc00148e3c0, 0xc00993ed80)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.0.2/query_partitiopn_objects_command.go:56 +0x47
github.com/aerospike/aerospike-client-go/v5.(*Client).queryNodePartitionObjects(0xc0002d4280, 0xc00ac8c1b0, 0xc0014a41c0, 0xc00148e3c0, 0xc00993ed80, 0xc00148e280, 0x68117a, 0xc000903f01)

Operate command:

aerospike.PutOp(aero.NewBin("ls", time.Now().UTC().Unix()))

It happened when one node went down for upgrade 5.2 -> 5.6. It was a global panic so it couldn't be recovered. Also new chained errors are huge(1 line of the log above).

khaf commented 3 years ago

Fixed in v5.1.0

un000 commented 3 years ago

It's ok now. Thank you!