Failed to load resource: net::ERR_CONNECTION_TIMED_OUT to http://postgrest.openbalena.MY_DOMAIN:8000/device?limit=10&offset=0&order=id.asc

[Mark-Powers]

First, thank you so much for this project!

I'm having an issue that appears similar to #11, though I have a different deployment (no kubernetes), and the logs I am seeing are different.

I'm using compose for both openBalena and open-balena-admin on the same host.

5e93a039a28d   openbalena_haproxy                            "/docker-entrypoint.…"   6 days ago   Up 24 hours   0.0.0.0:80->80/tcp, :::80->80/tcp, 222/tcp, 3128/tcp, 5432/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 6379/tcp   openbalena_haproxy_1
0d40b7586d2f   balena/open-balena-vpn:v11.4.6                "/usr/bin/entry.sh"      6 days ago   Up 24 hours   80/tcp, 443/tcp, 3128/tcp                                                                                         openbalena_vpn_1
c41e1ec04f04   balena/open-balena-api:v0.209.2               "/usr/bin/entry.sh"      6 days ago   Up 24 hours   80/tcp                                                                                                            openbalena_api_1
6b9bf5b05597   balena/open-balena-registry:v2.25.3           "/usr/bin/entry.sh"      6 days ago   Up 24 hours   80/tcp                                                                                                            openbalena_registry_1
e984ed2ee800   openbalena_cert-provider                      "/entry.sh /usr/src/…"   6 days ago   Up 24 hours   80/tcp                                                                                                            openbalena_cert-provider_1
607aab4e127c   balena/open-balena-s3:v2.13.8                 "/usr/bin/entry.sh"      6 days ago   Up 24 hours   80/tcp                                                                                                            openbalena_s3_1
a59be8c26f16   redis:alpine                                  "docker-entrypoint.s…"   6 days ago   Up 24 hours   6379/tcp                                                                                                          openbalena_redis_1
76311bbeb70a   balena/open-balena-db:v5.1.0                  "/balena-entrypoint.…"   6 days ago   Up 24 hours   5432/tcp                                                                                                          openbalena_db_1
665438afae08   harmonidcaputo/open-balena-ui:latest          "bash start.sh"          6 days ago   Up 24 hours   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                                         openbalena-admin_ui_1
94c23598b55c   harmonidcaputo/open-balena-remote:latest      "bash start.sh"          6 days ago   Up 24 hours   0.0.0.0:10000-10009->10000-10009/tcp, :::10000-10009->10000-10009/tcp                                             openbalena-admin_remote_1
7180f27f3e05   harmonidcaputo/open-balena-postgrest:latest   "bash start.sh"          6 days ago   Up 24 hours   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                                                         openbalena-admin_postgrest_1

When I load the dashboard by going to admin.MY_DOMAIN:8080, I am able to login, but I get the error in the browser console:

GET http://postgrest.MY_DOMAIN:8000/device?limit=10&offset=0&order=id.asc net::ERR_CONNECTION_TIMED_OUT

In the postgrest container logs, I see

03/Aug/2022:21:11:05 +0000: Retrying listening for notifications on the pgrst channel..
03/Aug/2022:21:11:05 +0000: Attempting to connect to the database...
03/Aug/2022:21:11:05 +0000: Connection successful
03/Aug/2022:21:11:05 +0000: Config re-loaded
03/Aug/2022:21:11:05 +0000: Listening for notifications on the pgrst channel
03/Aug/2022:21:11:05 +0000: Schema cache loaded
03/Aug/2022:21:11:56 +0000: Retrying listening for notifications on the pgrst channel..
03/Aug/2022:21:11:56 +0000: Attempting to connect to the database...
03/Aug/2022:21:11:56 +0000: Connection successful
03/Aug/2022:21:11:56 +0000: Config re-loaded
03/Aug/2022:21:11:56 +0000: Listening for notifications on the pgrst channel
03/Aug/2022:21:11:56 +0000: Schema cache loaded

Is there another place I should be looking for debugging this? I see no logs in haproxy or the balena-db

From the host running my browser

$ nc -vz postgrest.MY_DOMAIN 8000
Connection to postgrest.MY_DOMAIN port 8000 [tcp/irdmi] succeeded!

$ curl -v http://postgrest.MY_DOMAIN:8000/device\?limit\=10\&offset\=0\&order\=id.asc
*   Trying 192.5.87.71...
* TCP_NODELAY set
* Connected to postgrest.MY_DOMAIN (192.5.87.71) port 8000 (#0)
> GET /device?limit=10&offset=0&order=id.asc HTTP/1.1
> Host: postgrest.MY_DOMAIN:8000
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 504 Gateway Timeout
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Date: Wed, 03 Aug 2022 21:26:02 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
< 
* Connection #0 to host postgrest.MY_DOMAIN left intact
Error occurred while trying to proxy: postgrest.MY_DOMAIN:8000/device?limit=10&offset=0&order=id.asc* Closing connection 0

[dcaputo-harmoni]

Can you ping postgrest.MY_DOMAIN from the computer you are browsing from? It looks like postgrest is not having a problem talking to the database, so the breakdown must be happening between your browser and the proxy, or the proxy and postgrest. You could also try adding some console.log statements to the postgrest-proxy.js file in the postgrest container to see what's happening there.

[Mark-Powers]

Thank you, I was able to ping. I added a console.log statement as you suggested, and restarted the container, and upon restart the dashboard started working. I'm not sure what else changed. I'll close this thread though, as the issue is resolved.