amphp / postgres

Async Postgres client for PHP based on Amp.
MIT License
96 stars 20 forks source link

PqHandle: another command is already in progress #41

Closed PNixx closed 3 years ago

PNixx commented 3 years ago

After 2-6 hours high load I try error:

another command is already in progress: /project/vendor/amphp/postgres/src/PqHandle.php:202

After this error, no request is answered. I need restart amphp-cluster for restore correct work.

Installed packages, channel pecl.php.net:
=========================================
Package Version State
ev      1.0.6   stable
pq      2.1.8   stable
raphf   2.0.1   stable
enumag commented 3 years ago

I ran into this issue as well. Fortunately it was in scenario where I could reproduce it reliably. So I added some dumps into PqHandle to see what was happening in the PqHandle instance that eventually failed.

I found out that the last operation prior to the failure was PqHandle::release(). So I checked $this->handle->busy boolean (IMPORTANT: don't confuse it with the $this->busy promise, that's a different variable) and sure enough it was still true when the release method was called.

So I tried to use Loop::defer() and when that didn't help even Loop::delay(1, ...) bus even in that callback it was still true.

From what I checked the SQL query that caused it seems to be "DISCARD ALL".

In the end the only thing I could do to avoid the crash was to consider the state invalid and add this into PqHandle::release():

if ($this->handle->busy) {
    $this->close();
}
kelunik commented 3 years ago

@enumag Do you have a reproduction script you can share?

enumag commented 3 years ago

I wish but it's not that easy. It happens in the middle of my data import.

enumag commented 3 years ago

I noticed this line in the PqHandle code and try to apply it to my case. So I copied this line into the PqHandle::release() method and it seems to have helped. According to log the case of $this->handle->busy being true in the release method still happened however the application no longer crashed.