field 'epochs' not found in type: 'query_root'

liv-io commented 1 year ago

Summary

Hello, I am unable to run cardano-graphql-background in the preview environment. I am running the whole stack as a "Podman Pod" (like Docker Compose) on a single virtual machine.

The versions are:

ubuntu 22.04 (virtual machine and containers)
podman 3.0.1
node v14.21.3
cardano-node 1.35.5
cardano-pgsql 14.6
cardano-db-sync 13.1.0.0
cardano-ogmios 5.6.0
cardano-graphql-engine 2.19.0
cardano-graphql-background 8.0.0

When I start cardano-graphql-background for the very first time after purging the cardano-pgsql database and cardano-db-sync data, I get:
```
node /usr/local/src/cardano-graphql/packages/api-cardano-db-hasura/dist/background.js

{"name":"background","hostname":"pod_cardano","pid":74,"level":30,"module":"Db","msg":"Initializing...","time":"2023-02-27T09:23:18.820Z","v":0}
{"name":"background","hostname":"pod_cardano","pid":74,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2023-02-27T09:23:18.840Z","v":0}
{"name":"background","hostname":"pod_cardano","pid":74,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2023-02-27T09:23:18.841Z","v":0}
{"name":"background","hostname":"pod_cardano","pid":74,"level":20,"module":"HasuraBackgroundClient","msg":"graphql-engine setup","time":"2023-02-27T09:23:31.217Z","v":0}
{"name":"background","hostname":"pod_cardano","pid":74,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 1 of 101, retrying...","time":"2023-02-27T09:23:31.261Z","v":0}
<...>
{"name":"background","hostname":"pod_cardano","pid":74,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 101 of 101, exiting","time":"2023-02-27T10:07:02.307Z","v":0}
{"name":"background","hostname":"pod_cardano","pid":74,"level":50,"msg":"field 'epochs' not found in type: 'query_root': {\"response\":{\"errors\":[{\"extensions\":{\"code\":\"validation-failed\",\"path\":\"$.selectionSet.epochs\"},\"message\":\"field 'epochs' not found in type: 'query_root'\"}],\"status\":200},\"request\":{\"query\":\"query {\\n            epochs (limit: 1, order_by: { number: desc }) {\\n                number\\n            }\\n        }\"}}","time":"2023-02-27T10:07:02.307Z","v":0}
```
- When I run it a second time, I get:
```
node /usr/local/src/cardano-graphql/packages/api-cardano-db-hasura/dist/background.js
```
{"name":"background","hostname":"pod_cardano","pid":113,"level":30,"module":"Db","msg":"Initializing...","time":"2023-02-27T10:10:30.456Z","v":0} {"name":"background","hostname":"pod_cardano","pid":113,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2023-02-27T10:10:30.467Z","v":0} {"name":"background","hostname":"pod_cardano","pid":113,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2023-02-27T10:10:30.468Z","v":0} {"name":"background","hostname":"pod_cardano","pid":113,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 1 of 10, retrying...","time":"2023-02-27T10:10:30.964Z","v":0} <...> {"name":"background","hostname":"pod_cardano","pid":113,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 10 of 10, exiting","time":"2023-02-27T10:13:58.759Z","v":0} {"name":"background","hostname":"pod_cardano","pid":113,"level":50,"msg":"Command failed: /usr/local/bin/cli-hasura --skip-update-check --project /usr/local/src/cardano-graphql/packages/api-cardano-db-hasura/hasura/project --endpoint http://localhost:8080 migrate --database-name default apply --down all\n{\"level\":\"warning\",\"msg\":\"[cli: v2.19.0] [server: v12345.0.0] version mismatch: older cli version might not be compatible with latest server apis, please update cli\",\"time\":\"2023-02-27T10:13:58Z\"}\n{\"level\":\"error\",\"msg\":\"skipping applying migrations on database 'default', encountered: \n{\n \\"code\\": \\"not-exists\\",\n \\"error\\": \\"source with name \\\\"default\\\\" does not exist\\",\n \\"path\\": \\"$\\"\n}\",\"time\":\"2023-02-27T10:13:58Z\"}\ntime=\"2023-02-27T10:13:58Z\" level=fatal msg=\"applying migrations failed on database(s): default\"\n","time":"2023-02-27T10:13:58.760Z","v":0}
```
- Interestingly, the `hasura-graphql-engine` health status changes from `OK` to `WARN` after the first attempt:
```
curl -i --silent http://localhost:8080/healthz HTTP/1.1 200 OK

OK
curl -i --silent http://localhost:8080/healthz HTTP/1.1 200 OK

WARN: inconsistent objects in schema

I would appreciate any inputs and help. Please let me know if you need more details of my setup.

Steps to reproduce the bug

Run the listed software versions against the preview network
Ensure cardano-node and cardano-db-sync are fully synchronized
Ensure cardano-ogmios and cardano-graphql-engine are running
Start cardano-graphql-background
Observe output

Actual Result

When running cardano-graphql-background on a newly and fully synchronized database, I get:
```
field 'epochs' not found in type: 'query_root
```

When running cardano-graphql-background a second time I get:

Command failed: /usr/local/bin/cli-hasura --skip-update-check --project /usr/local/src/cardano-graphql/packages/api-cardano-db-hasura/hasura/project --endpoint http://localhost:8080 migrate --database-name default apply --down all

Expected Result

I would expect cardano-graphql-background to run without throwing errors, especially on a new and clean database/db-sync.

Environment

Unfortunately, I fail at installing and running this CLI:

root@pod_cardano:/usr/local/src/cardano-graphql/node_modules# npx @cardano-graphql/cli system-info
sh: 1: cannot create buildcheck.gypi: Permission denied
internal/modules/cjs/loader.js:934
  throw err;
  ^

Error: Cannot find module '/root/.npm/_npx/1537/lib/node_modules/@cardano-graphql/cli/node_modules/ssh2/install.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:931:15)
    at Function.Module._load (internal/modules/cjs/loader.js:774:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! ssh2@1.11.0 install: `node install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the ssh2@1.11.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-02-27T10_49_13_748Z-debug.log
Install for [ '@cardano-graphql/cli@latest' ] failed with code 1

Platform

[X] Linux (Ubuntu)
[ ] Linux (Other)
[ ] macOS
[ ] Windows

Platform version

Ubuntu 22.04 LTS

Runtime

[X] Node.js
[ ] Docker

Runtime version

node: v14.21.3

liv-io commented 1 year ago

I thought I additionally share an excerpt of my other containers:

cardano-node

podman logs --follow pod_cardano-cardano-node

[pod_card:cardano.node.ChainDB:Notice:196] [2023-02-27 09:11:32.26 UTC] Chain extended, new tip: a0a73150fb370d58f684af4675c2b5055bb3f8089b12b3b9d11ca775032f6825 at slot 10833092
[pod_card:cardano.node.ChainDB:Notice:196] [2023-02-27 09:12:20.52 UTC] Chain extended, new tip: c4a80802aaccbbb194c0e0317668df4905b2e3c0f482e2938db87d01d7c05d36 at slot 10833140

cardano-pgsql

podman logs --follow pod_cardano-cardano-pgsql`

2023-02-27 09:03:49.497 UTC [9] LOG:  starting PostgreSQL 14.6 (Ubuntu 14.6-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, 64-bit
2023-02-27 09:03:49.498 UTC [9] LOG:  listening on IPv4 address "0.0.0.0", port 5432
<...>
2023-02-27 09:03:53.353 UTC [9] LOG:  database system is ready to accept connections

cardano-db-sync

podman logs --follow pod_cardano-cardano-db-sync

All good!                                                                                                                                                                                                  
[db-sync-node:Info:6] [2023-02-27 09:06:35.04 UTC] Version number: 13.1.0.0                                                                                                                                
<...>
[db-sync-node.Subscription:Notice:54] [2023-02-27 09:06:41.17 UTC] Identity Connection Attempt Start, destination LocalAddress "/var/local/cardano-node/db/node.socket"
[db-sync-node.Subscription:Notice:54] [2023-02-27 09:06:41.17 UTC] Identity Connection Attempt End, destination LocalAddress "/var/local/cardano-node/db/node.socket" outcome: ConnectSuccessLast
<...>
[db-sync-node:Info:66] [2023-02-27 09:11:32.45 UTC] Insert Babbage Block: epoch 125, slot 10833092, block 492104, hash a0a73150fb370d58f684af4675c2b5055bb3f8089b12b3b9d11ca775032f6825
[db-sync-node:Info:66] [2023-02-27 09:12:20.74 UTC] Insert Babbage Block: epoch 125, slot 10833140, block 492105, hash c4a80802aaccbbb194c0e0317668df4905b2e3c0f482e2938db87d01d7c05d36

cardano-ogmios

podman logs --follow pod_cardano-cardano-ogmios

{"severity":"Info","timestamp":"2023-02-27T09:11:32.272465547Z","thread":"17","message":{"Health":{"tag":"HealthTick","status":{"startTime":"2023-02-27T09:06:33.626451503Z","lastKnownTip":{"slot":10833092,"hash":"a0a73150fb370d58f684af4675c2b5055bb3f8089b12b3b9d11ca775032f6825","blockNo":492104},"lastTipUpdate":"2023-02-27T09:11:32.27151958Z","networkSynchronization":1.00000,"currentEra":"Babbage","metrics":{"activeConnections":0,"runtimeStats":{"cpuTime":633202081,"currentHeapSize":391,"gcCpuTime":350322814,"maxHeapSize":394},"sessionDurations":{"max":0,"mean":0,"min":0},"totalConnections":0,"totalMessages":0,"totalUnrouted":0},"connectionStatus":"connected","currentEpoch":125,"slotInEpoch":33092}}},"version":"v5.6.0"}
{"severity":"Info","timestamp":"2023-02-27T09:06:54.498292305Z","thread":"17","message":{"Health":{"tag":"HealthTick","status":{"startTime":"2023-02-27T09:06:33.626451503Z","lastKnownTip":{"slot":10832814,"hash":"bebb2ffc3a638f4102890ef1294aad03486fa73c5ee2afac8691afdb77bda9ce","blockNo":492091},"lastTipUpdate":"2023-02-27T09:06:54.498116003Z","networkSynchronization":1.00000,"currentEra":"Babbage","metrics":{"activeConnections":0,"runtimeStats":{"cpuTime":232617944,"currentHeapSize":390,"gcCpuTime":106704140,"maxHeapSize":390},"sessionDurations":{"max":0,"mean":0,"min":0},"totalConnections":0,"totalMessages":0,"totalUnrouted":0},"connectionStatus":"connected","currentEpoch":125,"slotInEpoch":32814}}},"version":"v5.6.0"}

curl --silent http://localhost:1337/metrics

# TYPE ogmios_active_connections gauge
ogmios_active_connections  0.0
# TYPE ogmios_connected gauge
ogmios_connected  1.0
<...>

cardano-graphql-engine

podman logs --follow pod_cardano-cardano-graphql-engine

{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":false,"enabled_apis":["config","pgdump","metadata","graphql"],"enabled_log_types":["startup","webhook-log","http-log","query-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":8080,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":true,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2023-02-27T09:03:55.255+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgresql://dbsync:...@localhost:5432/dbsync","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2023-02-27T09:03:55.255+0000","type":"startup"}

curl --silent http://localhost:8080/healthz

OK

liv-io commented 1 year ago

Alright, I might have found the solution to my issue. The cardano-graphql-engine container is supposed to have the following environment variable:

export HASURA_GRAPHQL_DATABASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}

It doesn't make much sense to me, as I've cleanly supplied all parameters on the command-line:

/usr/local/bin/graphql-engine --host localhost --port 5432 --user dbsync --password dbsync --dbname dbsync serve

I will report back, once I made further progress.

liv-io commented 1 year ago

:heavy_check_mark: Solved

After adding the HASURA_GRAPHQL_DATABASE_URL environment variable to the entrypoint script of the hasura-graphql-engine container, the cardano-graphql-background service started working.

cardano-foundation / cardano-graphql