search

omgnetwork / elixir-omg

OMG-Network repository of Watcher and Watcher Info
https://omg.network
Apache License 2.0
211 stars 59 forks source link

DBConnection.ConnectionError: tcp connect (127.0.0.1:5432): connection refused - :econnrefused #1735

Open sentry-io[bot] opened 4 years ago

sentry-io[bot] commented 4 years ago

Sentry Issue: ELIXIR-OMG-6J

DBConnection.ConnectionError: tcp connect (127.0.0.1:5432): connection refused - :econnrefused

(DBConnection.ConnectionError) tcp connect (127.0.0.1:5432): connection refused - :econnrefused
boolafish commented 4 years ago

Some of my investigation: https://sentry.io/organizations/omisego/issues/1787235126/activity/?project=1477673&query=is%3Aunresolved

I am guessing when we restart watcher-info it will try to ask for Postgres while it is not ready yet (I assume Postgres got restart as well). Not sure what is the solution to this. We can ignore the sentry error or try to make our Genserver smarter (?) to detect Postgres is not ready in init?

boolafish commented 4 years ago

Another might be related Sentry issue: https://sentry.io/organizations/omisego/issues/1824732112/activity/?project=1477673&query=is%3Aunresolved

boolafish commented 4 years ago

Confirmed related to deployment (did one new deployment on stress for 1.0.4-pre.2 and the report immediately shows)

boolafish commented 4 years ago

From @arthurk , it can be caused by cloudsql-proxy gets restarted. (Some background, the stress deployment actually did not restart the pod for watcher-info nor the Postgres for the watcher-info as the image of stress was hardcoded.)

https://omgnetworkhq.slack.com/archives/CMX790Q5V/p1600073457072700?thread_ts=1600073144.072000&cid=CMX790Q5V