search

pip-services3-go / pip-services3-postgres-go

Postresql components for Pip.Services in Golang
MIT License
0 stars 0 forks source link

Postgres driver fails when pummeled using simultaneous requests #11

Open harikayalamati opened 1 year ago

harikayalamati commented 1 year ago

Issue: When the postgres driver is pummeled using simultaneous requests, eventually it will fail and do not recover. This causes a service failure that isn't recoverable unless we restart the container/pod.

Happened when: In the test environment, a single pod of the deployed service became un-responsive and as such the dependent calls ended up in failure. What this presented as is that every other call to the deployed service via the load balancer would time out. When two calls are made to the deployed service, either the first call fails or the second call fails (hence all calls time out).

Suspicion: Issue might be with postgres persistence library for pip-services especially with how it's handling context or how it responds to multiple simultaneous requests.

Process to replicate in local: Download client-stars-go and service-stars-go zip files and unzip them. Open both projects in vscode. Read README.md in service-stars-go and build service image(command #4). Then read README.md in client-stars-go and follow instruction from "how to replicate issue?". Client tests start smooth but in few minutes they result in below error in the output terminal. Failed to call v1/stars.read_star_by_email_address method: Unknown communication problem on REST client Caused by: Post "http://localhost:9011/v1/stars/read_star_by_email_address?correlation_id=issue_replicate": context deadline exceeded (Client.Timeout exceeded while awaiting headers) error while reading star : Unknown communication problem on REST client. Restarting the client tests result in error from the beginning but restarting the service in docker and restarting the client tests clears the issue and error reappears in few minutes. If you have any issues replicating, please contact us.

Temporary fix in place: Replaced the postgres persistence with go sql driver and deployed the updated service to test environment. Observed that issue is fixed. This fix made our suspicion stronger.

I added the client and service projects in discord channel toolkit-go. image

spiritofnikopol commented 1 year ago

@harikayalamati we can't run your sample, it has private dependencies and we couldn't download it. Could you make more simple example without using private repos?

harikayalamati commented 1 year ago

@spiritofnikopol @levichevdmitry Apologies for the delay. I have removed the dependencies and attached in discord. Please use the latest projects. image