cli-migrations-v3: delay in starting hasura container in docker-compose setup

[skanel]

here is my docker-compose.yml, I was tried for a few days to find a solution to this, but I can not.

version: "3.6"
services:
  postgres:
    image: postgres:12
    restart: always
    volumes:
      - ./db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
  graphql-engine:
    image: hasura/graphql-engine:v2.0.7.cli-migrations-v3
    env_file:
      - ./hasura-server/.env
    volumes:
      - ./hasura-server/migrations:/hasura-migrations
      - ./hasura-server/metadata:/hasura-metadata
    ports:
      - "8080:8080"
    depends_on:
      - "postgres"
    restart: always
    environment:
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## this env var can be used to add the above postgres database to Hasura as a data source. this can be removed/updated based on your needs
      PG_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      ## uncomment next line to set an admin secret
      HASURA_GRAPHQL_ADMIN_SECRET: xxxxxx

      HASURA_GRAPHQL_JWT_SECRET: '{"type":"HS384", "key":"3EK6FD+o0+c7tzBNVfjpMkNDi2yARAAKzQlk8O2IKoxQu4nF7EdAh8s3TwpHwrdWT6R"}'
volumes:
  db_data:

here is my config.yaml

version: 3
endpoint: http://localhost:8080/
admin_secret: "xxxxxx"
api_paths:
  v1_query: v1/query
  v2_query: v2/query
  v1_metadata: v1/metadata
  graphql: v1/graphql
  config: v1alpha1/config
  pg_dump: v1alpha1/pg_dump
  version: v1/version
metadata_directory: metadata
migrations_directory: migrations
seeds_directory: seeds
actions:
  kind: synchronous
  handler_webhook_baseurl: http://localhost:3000
  codegen:
    framework: ""
    output_dir: ""

when I run docker-compose up -d it works fine. but when run cd hasura-server && hasura console i got this error output

FATA[0001] version check: failed to get version from server: failed making version api call: Get "http://localhost:8080/v1/version": EOF

anyone, please help?

[praveenweb]

@skanel - It looks like the graphql-engine server is unable to establish a connection with Postgres with the above configuration. Can you try adding the port mapping to expose postgres service on 5432?

So your docker-compose will look something like:

version: "3.6"
services:
  postgres:
    image: postgres:12
    restart: always
    volumes:
      - ./db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
    ports:
     - "5432:5432"
  graphql-engine:
    image: hasura/graphql-engine:v2.0.7.cli-migrations-v3
    ....

I will check with the team to see why the cli-migrations image in particular seems to have this issue.

[scriptonist]

@skanel what probably might be happening is, cli-migrations image starts an intermediate HGE instance, PG might not be ready when this happens, intermediate HGE will not retry the connection and the container waits till a timeout to restart the container, and the restart should fix this.

Can you try waiting ~30 seconds after you do a docker-compose up and also make sure docker-compose ps show status of containers as "UP" before you execute hasura console command? (to verify the hypothesis)

This probably might need a fix to add connection retries to intermediate HGE container that the cli-migrations image starts.

[skanel]

1. docker-compose ps

   
           Name                             Command               State                    Ports                  
   ----------------------------------------------------------------------------------------------------------------------
   posgraphql-engine_graphql-engine_1   docker-entrypoint.sh graph ...   Up      0.0.0.0:8080->8080/tcp,:::8080->8080/tcp
   posgraphql-engine_postgres_1         docker-entrypoint.sh postgres    Up      5432/tcp 
   ` ` ` 

2. hasura console

FATA[0001] version check: failed to get the version from server: failed to make version API call: Get "http://localhost:8080/v1/version": EOF 

I have waited more ~30 seconds, please help

[scriptonist]

@skanel Please share output of docker-compose logs

[skanel]

thanks for your help, here is the log

log

[scriptonist]

@skanel from the logs your Postgres container is not ready and hence Hasura is not able to connect to it.

So when CLI makes an API call to hasura instance it returns nothing (EOF) and hence the error failed to make version API call: Get "http://localhost:8080/v1/version": EOF

[skanel]

but docker shows that the container is UP.

could you help suggest the solution of change or what I should change, for me the configuration looks good, user, password, host , right?

[atb00ker]

@skanel

Container gets in the UP state before everything is ready. I usually open http://localhost:8080 on browser and wait for it to load before starting CLI! :smile:

[BenoitRanque]

Can you try with the environment variable HASURA_GRAPHQL_DATABASE_URL instead of PG_DATABASE_URL? As it stands, your project will not have a database connected when the container first comes online, it will have to wait for the metadata to be applied first. This could be the issue, but is a shot in the dark.

[clausMeko]

If it still is an issue: I use HASURA_GRAPHQL_DATABASE_URL as @BenoitRanque mentioned.

I wait for hasura to come up like this:

func hasuraOnline(port string) error {
    r, err := http.Get(fmt.Sprintf("http://localhost:%s/healthz", port))
    if err != nil || r.StatusCode != http.StatusOK {
        time.Sleep(100 * time.Millisecond)
        _ = hasuraOnline(port)
    }
    return nil
}

30 seconds should be fine - from my experience: If that is dependent of the migration scripts (I have ~270kB migrations)

Working DockerFile:

version: '3.6'
services:

  postgres:
    image: postgres:12
    restart: always
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
    ports:
      - "${POSTGRES_PORT}:5432"

  graphql-engine:
    image: hasura/graphql-engine:v2.0.9.cli-migrations-v3
    depends_on:
      - "postgres"
    restart: always
    ports:
      - "${HASURA_PORT}:8080"
    environment:
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## this env var can be used to add the above postgres database to Hasura as a data source. this can be removed/updated based on your needs
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      #      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      HASURA_GRAPHQL_EXPERIMENTAL_FEATURES: inherited_roles
      ## uncomment next line to set an admin secret
      # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
      HASURA_GRAPHQL_MIGRATIONS_DIR: /hasura-migrations/migrations
        # custom env variables
      # note: each value is considered to be a string. URLs must not be quoted (they will be passed double-quoted otherwise)
      HASURA_GRAPHQL_ACTION_RENDER_ENDPOINT: http://road_to_nowhere:3000/render
      HASURA_GRAPHQL_ACTION_ABOUT_ENDPOINT: http://road_to_nowhere:3000/about
    volumes:
      - ./migrations:/hasura-migrations
      - ./metadata:/hasura-metadata
volumes:
  db_data: