On starting prefect backend server getting the errors below
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
hasura_1 | {"type":"pg-client","timestamp":"2022-05-18T08:54:06.805+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
hasura_1 | {"type":"pg-client","timestamp":"2022-05-18T08:54:06.805+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(1)."}}
hasura_1 | {"type":"startup","timestamp":"2022-05-18T08:54:06.805+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"postgres\" (172.22.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 27811b58307b, Create extensions and initial settings
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 27811b58307b -> 72e2cd3e0469, Initial database tables migration
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 72e2cd3e0469 -> c4d792bdd05e, Add flow run idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c4d792bdd05e -> 3398e4807bfb, Add traversal functions
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3398e4807bfb -> b9086bd4b962, Create message table
graphql_1 | INFO [alembic.runtime.migration] Running upgrade b9086bd4b962 -> c1f317aa658c, Remove state_id foreign keys
graphql_1 | INFO [alembic.runtime.migration] Running upgrade c1f317aa658c -> 6611fd0ccc73, Simplify run state update triggers
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6611fd0ccc73 -> 70528cee0d2b, Add agent persistence
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 70528cee0d2b -> 9cb7539b7363, Add index on agent_id
postgres_1 | selecting default max_connections ... 100
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9cb7539b7363 -> e148cf9f1e5b, Add task run name
graphql_1 | INFO [alembic.runtime.migration] Running upgrade e148cf9f1e5b -> 850b76d44332, Add flow run config
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 850b76d44332 -> 24f10aeee83e, Add label column to flow runs
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 24f10aeee83e -> 3c87ad7e0b71, Add artifact api
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 3c87ad7e0b71 -> 57ac2cb01ac1, Add index for task run names
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 57ac2cb01ac1 -> 7ca57ea2fdff, Add run_config to flow runs and flow groups
postgres_1 | creating configuration files ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 7ca57ea2fdff -> 9116e81c6dc2, Add description to flow group table
postgres_1 | running bootstrap script ... ok
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 9116e81c6dc2 -> 459a61bedc9e, Improve run triggers to handle same-version states
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 459a61bedc9e -> a666a3f4e422, Add unique index for idempotency key
graphql_1 | INFO [alembic.runtime.migration] Running upgrade a666a3f4e422 -> ac5747fb571c, Add unique constraint on edge table for task IDs
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2022-05-18 08:54:06.567 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-05-18 08:54:06.608 UTC [49] LOG: database system was shut down at 2022-05-18 08:54:06 UTC
postgres_1 | 2022-05-18 08:54:06.622 UTC [48] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | 2022-05-18 08:54:07.041 UTC [48] LOG: received fast shutdown request
postgres_1 | waiting for server to shut down....2022-05-18 08:54:07.042 UTC [48] LOG: aborting any active transactions
postgres_1 | 2022-05-18 08:54:07.045 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
postgres_1 | 2022-05-18 08:54:07.054 UTC [50] LOG: shutting down
postgres_1 | 2022-05-18 08:54:07.068 UTC [48] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2022-05-18 08:54:07.166 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2022-05-18 08:54:07.166 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2022-05-18 08:54:07.175 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-05-18 08:54:07.213 UTC [76] LOG: database system was shut down at 2022-05-18 08:54:07 UTC
postgres_1 | 2022-05-18 08:54:07.234 UTC [1] LOG: database system is ready to accept connections
ui_1 | Missing the PREFECT_SERVER__BASE_URL environment variable. Using default
ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: using the "epoll" event method
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: nginx/1.20.2
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: OS: Linux 4.15.0-177-generic
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: getrlimit(RLIMIT_NOFILE): 1048576:1048576
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker processes
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker process 11
ui_1 | 2022/05/18 08:54:10 [notice] 10#10: start worker process 12
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.ZombieKiller", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 216, in run_once\n await self.reap_zombie_task_runs()\n File \"/prefect-server/src/prefect_server/services/towel/zombie_killer.py\", line 153, in reap_zombie_task_runs\n apply_schema=False,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n return await self.reschedule_flow_runs()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n limit=500,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
towel_1 | {"severity": "ERROR", "name": "prefect-server.HasuraClient", "message": "Encountered internal API exception: All connection attempts failed", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed"}
towel_1 | {"severity": "ERROR", "name": "prefect-server.Scheduler", "message": "Unexpected error: APIError('Unable to complete operation. An internal API error occurred.')", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 127, in try_connect\n stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)\n File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 1519, in connect_tcp\n local_addr=local_addr)\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 962, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.7/asyncio/base_events.py\", line 949, in create_connection\n await self.sock_connect(sock, address)\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 473, in sock_connect\n return await fut\n File \"/usr/local/lib/python3.7/asyncio/selector_events.py\", line 503, in _sock_connect_cb\n raise OSError(err, f'Connect call failed {address}')\nConnectionRefusedError: [Errno 111] Connect call failed ('172.22.0.3', 3000)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 8, in map_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/site-packages/anyio/_core/_sockets.py\", line 184, in connect_tcp\n raise OSError('All connection attempts failed') from cause\nOSError: All connection attempts failed\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 60, in map_httpcore_exceptions\n yield\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 253, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py\", line 237, in handle_async_request\n response = await connection.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 86, in handle_async_request\n raise exc\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 63, in handle_async_request\n stream = await self._connect(request)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_async/connection.py\", line 111, in _connect\n stream = await self._network_backend.connect_tcp(**kwargs)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/auto.py\", line 24, in connect_tcp\n host, port, timeout=timeout, local_address=local_address\n File \"/usr/local/lib/python3.7/site-packages/httpcore/backends/asyncio.py\", line 104, in connect_tcp\n local_host=local_address,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpcore/_exceptions.py\", line 12, in map_exceptions\n raise to_exc(exc)\nhttpcore.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 87, in reraise_as_api_error\n yield\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1833, in post\n extensions=extensions,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1506, in request\n return await self.send(request, auth=auth, follow_redirects=follow_redirects)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1597, in send\n history=[],\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1624, in _send_handling_auth\n history=history,\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1658, in _send_handling_redirects\n response = await self._send_single_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_client.py\", line 1695, in _send_single_request\n response = await transport.handle_async_request(request)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 353, in handle_async_request\n resp = await self._pool.handle_async_request(req)\n File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n self.gen.throw(type, value, traceback)\n File \"/usr/local/lib/python3.7/site-packages/httpx/_transports/default.py\", line 77, in map_httpcore_exceptions\n raise mapped_exc(message) from exc\nhttpx.ConnectError: All connection attempts failed\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/scheduler.py\", line 47, in run_once\n offset=500 * iterations,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 64, in execute\n timeout=30,\n File \"/usr/local/lib/python3.7/contextlib.py\", line 188, in __aexit__\n await self.gen.athrow(typ, value, traceback)\n File \"/prefect-server/src/prefect_server/utilities/exceptions.py\", line 93, in reraise_as_api_error\n raise APIError() from exc\nprefect_server.utilities.exceptions.APIError: Unable to complete operation. An internal API error occurred."}
graphql_1 |
graphql_1 | Could not upgrade the database!
graphql_1 | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f826cc9e590>: Failed to establish a new connection: [Errno 111] Connection refused'))
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | Applied Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
graphql_1 |
graphql_1 | Database upgraded!
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | {"severity": "INFO", "name": "prefect-server.GraphQL Server", "message": "Using uvicorn log level = 'debug'"}
graphql_1 | INFO: Started server process [8]
graphql_1 | INFO: Waiting for application startup.
graphql_1 | INFO: Application startup complete.
graphql_1 | INFO: Uvicorn running on http://0.0.0.0:4201 (Press CTRL+C to quit)
apollo_1 | Checking GraphQL service at http://graphql:4201/health ...
graphql_1 | INFO: 172.22.0.6:60896 - "GET /health HTTP/1.1" 200 OK
apollo_1 | {"status":"ok","version":"2022.04.14"}
apollo_1 | GraphQL service healthy!
apollo_1 |
apollo_1 | > @ serve /apollo
apollo_1 | > node dist/index.js
apollo_1 |
apollo_1 | Building schema...
graphql_1 | INFO: 172.22.0.6:60922 - "POST / HTTP/1.1" 200 OK
apollo_1 | Building schema complete!
apollo_1 | Server ready at http://0.0.0.0:4200 🚀 (version: 2022.04.14)
apollo_1 | Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"3b5b6298-f162-45d0-bf88-ea62e4eb573b","prefect_server_version":"2022.04.14","api_version":"0.2.0"}}
graphql_1 | INFO: 172.22.0.6:60944 - "POST / HTTP/1.1" 200 OK
graphql_1 | INFO: 172.22.0.6:60956 - "POST / HTTP/1.1" 200 OK
WELCOME TO
_____ _____ ______ ______ ______ _____ _______ _____ ______ _______ ________ _____
| __ \| __ \| ____| ____| ____/ ____|__ __| / ____| ____| __ \ \ / / ____| __ \
| |__) | |__) | |__ | |__ | |__ | | | | | (___ | |__ | |__) \ \ / /| |__ | |__) |
| ___/| _ /| __| | __| | __|| | | | \___ \| __| | _ / \ \/ / | __| | _ /
| | | | \ \| |____| | | |___| |____ | | ____) | |____| | \ \ \ / | |____| | \ \
|_| |_| \_\______|_| |______\_____| |_| |_____/|______|_| \_\ \/ |______|_| \_\
Visit http://localhost:8080 to get started, or check out the docs at https://docs.prefect.io
Description
Reproduction
Environment