cardano-graphql schema migrations failing for private testnet

[jonathangenlambda]

Summary

We have a private testnet up and running, with cardano-node, cardano-ogmios, postgres and cardano-db-sync successfully connected and everything works fine.

Now we want to bring in cardano-graphql but we seem not to get it working. Here is the docker-compose.yml (note that we have to use cardano-db-sync:master otherwise cardano-db-sync would fail to sync with the private testnet which bootstraps into alonzo from block 0):

version: "3.3"

services:
  testnet:
    build:
      context: .
    ports:
      - "3001:3001"
      - "12789:12789"
    volumes:
      - ${PWD}/shared/configs:/testnet/configs
      - ${PWD}/shared/sockets:/testnet/sockets
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

  cardano-ogmios:
      command: [
        "--host", "0.0.0.0",
        "--node-socket", "/shared/sockets/node-pool1.sock",
        "--node-config", "/shared/configs/configuration.yaml",
      ]
      image: cardanosolutions/ogmios:v5.2.0
      depends_on:
        - testnet
      logging:
        driver: "json-file"
        options:
          max-size: "400k"
          max-file: "20"
      ports:
        - ${OGMIOS_PORT:-1337}:1337
      restart: on-failure
      volumes:
        - ${PWD}/shared:/shared

  postgres:
    image: postgres:11.5-alpine
    shm_size: 1g 
    environment:
      - POSTGRES_LOGGING=true
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
    secrets:
      - postgres_password
      - postgres_user
      - postgres_db
    ports:
      - ${POSTGRES_PORT:-5432}:5432
    restart: on-failure
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

  cardano-db-sync:
    image: inputoutput/cardano-db-sync:master
    command: [
      "--config /shared/configs/db-sync-config.yaml",
      "--socket-path /shared/sockets/node-pool1.sock"
    ]
    environment:
      - EXTENDED=true
      - POSTGRES_HOST=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PORT=5432
      - RESTORE_RECREATE_DB=N
    depends_on:
      - testnet
      - postgres
    secrets:
      - postgres_password
      - postgres_user
      - postgres_db
    volumes:
      - ${PWD}/shared:/shared
      # NOTE: uncomment if you want to keep data
      # - ${PWD}/dbdata/sync/db-sync-data:/var/lib/cdbsync
    restart: on-failure
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

  hasura:
    image: inputoutput/cardano-graphql-hasura:master
    ports:
      - ${HASURA_PORT:-8090}:8080
    depends_on:
      - "postgres"
    restart: on-failure
    environment:
      - HASURA_GRAPHQL_ENABLE_CONSOLE=true
      - HASURA_GRAPHQL_CORS_DOMAIN=http://localhost:9695
    secrets:
      - postgres_db
      - postgres_password
      - postgres_user
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

secrets: 
  postgres_db:
    file: ./db/postgres_db
  postgres_password:
    file: ./db/postgres_password
  postgres_user:
    file: ./db/postgres_user

volumes:
  db-sync-data:
  postgres:
  prometheus-data:

When all services are up and running and the socket files of the private testnet nodes are exposed, we are firing up cardano-graphql with another docker-compose:

version: "3.3"

services:
  cardano-graphql:
    image: inputoutput/cardano-graphql:master
    environment:
      - CARDANO_NODE_CONFIG_PATH=/shared/configs/configuration-json.json
      - CARDANO_NODE_SOCKET_PATH=/shared/sockets/node-pool1.sock
      - HASURA_URI=http://hasura:8080
      - OGMIOS_HOST=cardano-ogmios
      - OGMIOS_PORT=1337
      - POSTGRES_HOST=postgres
      - POSTGRES_PORT=5432
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
      - ALLOW_INTROSPECTION=true
      - CACHE_ENABLED=true
      - LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY:-debug}
    expose:
      - ${API_PORT:-3100}
    ports:
      - ${API_PORT:-3100}:3100
    restart: on-failure
    secrets:
      - postgres_db
      - postgres_password
      - postgres_user
    volumes:
      - ${PWD}/shared:/shared
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"

secrets: 
  postgres_db:
    file: ./db/postgres_db
  postgres_password:
    file: ./db/postgres_password
  postgres_user:
    file: ./db/postgres_user

volumes:
  db-sync-data:
  postgres:
  prometheus-data:

When cardano-graphql starts we see the following output:

{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2022-05-16T17:34:47.695Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":30,"module":"Server","msg":"Initializing","time":"2022-05-16T17:34:47.696Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":30,"module":"HasuraClient","msg":"Initializing","time":"2022-05-16T17:34:47.811Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Help us improve Hasura! The cli collects anonymized usage stats which\\nallow us to keep improving Hasura at warp speed. To opt-out or read more,\\nvisit https://hasura.io/docs/1.0/graphql/manual/guides/telemetry.html\\n\",\"time\":\"2022-05-16T17:34:47Z\"}\n{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:50Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-05-16T17:34:50Z\"}\n","time":"2022-05-16T17:34:50.114Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:50Z\"}\n","time":"2022-05-16T17:34:50.250Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 1 of 10, retrying...","time":"2022-05-16T17:34:50.250Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:51Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-05-16T17:34:51Z\"}\n","time":"2022-05-16T17:34:51.353Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:51Z\"}\n","time":"2022-05-16T17:34:51.488Z","v":0}

This continues until Attempt 10 of 10 followed by a final error output:

{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":50,"msg":"Command failed: /usr/local/bin/hasura --skip-update-check --project /app/packages/api-cardano-db-hasura/hasura/project --endpoint http://hasura:8080 migrate apply --up all\ntime=\"2022-05-16T17:38:16Z\" level=fatal msg=\"apply failed: [postgres-error] query execution failed ($.args[0].args)\\r\\nFile: '1589369664961_init/up.sql'\\r\\n{\\n    \\\"check_metadata_consistency\\\": false,
...
FatalError: column pool.reward_addr does not exist\\r\\nHint: Perhaps you meant to reference the column \\\"pool.reward_addr_id\\\".\"\n","time":"2022-05-16T17:38:16.334Z","v":0}

Steps to reproduce the bug

No response

Actual Result

See summary description.

Expected Result

Expected is that cardano-graphql gets up and running without errors and that we can connect to it through http://localhost:3100/graphql.

Environment

Using environment as specified in the docker-compose in the Summary.

cardano-node --version: cardano-node 1.34.0 - linux-x86_64 - ghc-8.10 git rev c23d5d319cd3276575c6ac32458516232e8d2c48

cardano-db-sync --version: cardano-db-sync 12.0.1 - linux-x86_64 - ghc-8.10 git revision 6af53927161a3914598491de6c650da96f3c8c09

Platform

• [X] Linux (Ubuntu)
• [ ] Linux (Other)
• [ ] macOS
• [ ] Windows

Platform version

Ubuntu 22.04

Runtime

• [ ] Node.js
• [ ] Docker

Runtime version

No response

[rhyslbw]

This should work, but I wouldn't expect it to split across docker-compose files given you'd be creating two networks. Why not just add it to the same compose file?

[jonathangenlambda]

Added to the same compose file, now running into different issue:

{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":50,"err":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"172.18.0.6","port":5432},"msg":"connect ECONNREFUSED 172.18.0.6:5432","time":"2022-07-11T11:16:43.877Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 1","time":"2022-07-11T11:16:43.879Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 2","time":"2022-07-11T11:16:44.417Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 3","time":"2022-07-11T11:16:44.919Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2022-07-11T11:16:44.963Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":30,"module":"Server","msg":"Initializing","time":"2022-07-11T11:16:44.963Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":30,"module":"HasuraClient","msg":"Initializing","time":"2022-07-11T11:16:45.088Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Help us improve Hasura! The cli collects anonymized usage stats which\\nallow us to keep improving Hasura at warp speed. To opt-out or read more,\\nvisit https://hasura.io/docs/1.0/graphql/manual/guides/telemetry.html\\n\",\"time\":\"2022-07-11T11:16:45Z\"}\n","time":"2022-07-11T11:16:46.344Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 1 of 10, retrying...","time":"2022-07-11T11:16:46.344Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.501Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"migrations applied\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.720Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Clearing metadata...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"Metadata cleared\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.857Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying metadata...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"Installing plugin cli-ext...\",\"time\":\"2022-07-11T11:16:48Z\"}\n{\"level\":\"info\",\"msg\":\"plugin installed\",\"name\":\"cli-ext\",\"time\":\"2022-07-11T11:16:52Z\"}\n{\"level\":\"info\",\"msg\":\"Metadata applied\",\"time\":\"2022-07-11T11:16:52Z\"}\n","time":"2022-07-11T11:16:52.646Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"graphql-engine setup","time":"2022-07-11T11:16:52.883Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:52.888Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 1 of 101, retrying...","time":"2022-07-11T11:16:52.888Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:53.894Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 2 of 101, retrying...","time":"2022-07-11T11:16:53.894Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:54.950Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 3 of 101, retrying...","time":"2022-07-11T11:16:54.950Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:56.057Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 4 of 101, retrying...","time":"2022-07-11T11:16:56.057Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:57.220Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 5 of 101, retrying...","time":"2022-07-11T11:16:57.220Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:58.441Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 6 of 101, retrying...","time":"2022-07-11T11:16:58.441Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:59.723Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 7 of 101, retrying...","time":"2022-07-11T11:16:59.723Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:01.066Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 8 of 101, retrying...","time":"2022-07-11T11:17:01.066Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:02.479Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 9 of 101, retrying...","time":"2022-07-11T11:17:02.479Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:03.962Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 10 of 101, retrying...","time":"2022-07-11T11:17:03.962Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:05.518Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 11 of 101, retrying...","time":"2022-07-11T11:17:05.518Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:07.151Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 12 of 101, retrying...","time":"2022-07-11T11:17:07.151Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:08.866Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 13 of 101, retrying...","time":"2022-07-11T11:17:08.866Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:10.666Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 14 of 101, retrying...","time":"2022-07-11T11:17:10.666Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:12.556Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 15 of 101, retrying...","time":"2022-07-11T11:17:12.556Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:14.540Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 16 of 101, retrying...","time":"2022-07-11T11:17:14.540Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:16.625Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 17 of 101, retrying...","time":"2022-07-11T11:17:16.625Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"DB sync state has reached minimum progress","time":"2022-07-11T11:17:18.814Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:18.820Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":50,"msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:18.820Z","v":0}

[jonathangenlambda]

It seems that graphql just "gives up" after Attempt 17 of 101, instead of keep trying. The solution was to really start hasura and graphql "later", that is after the testnet has started to produce a few blocks (~50). We start hasura and graphql in a separate docker-compose file, which works just fine.

In this process we also have switched to cardano-node/cli version 1.35 and bumped the versions of the services accordingly:

• cardano-wallet:2022.7.1
• cardano-ogmios:v5.5.2
• postgres:14.4-alpine
• cardano-db-sync:13.0.1
• cardano-graphql-hasura:master (6.2.0 does not work, failing schema migrations)
• cardano-graphql:master (same)

Note that in our private testnet we are bootstrapping into Babbage from epoch 0, as defined in configuration.yml:

TestShelleyHardForkAtEpoch: 0
TestAllegraHardForkAtEpoch: 0
TestMaryHardForkAtEpoch: 0
TestAlonzoHardForkAtEpoch: 0
TestBabbageHardForkAtEpoch: 0