search

hasura / graphql-engine

Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
https://hasura.io
Apache License 2.0
31.09k stars 2.77k forks source link

Migrations and metadata tutorial provides incorrect information #7539

Open Gozala opened 3 years ago

Gozala commented 3 years ago

Version Information

Server Version: v2.0.9 CLI Version (for CLI related issue): v2.0.8

Environment

Running hasura in a docker container based as per documentation https://hasura.io/docs/latest/graphql/core/getting-started/docker-simple.html#step-1-get-the-docker-compose-file

What is the expected behaviour?

I expect that following steps in migration tutorial will provision a database with a desired schema and graphql-engine.

Keywords

How to reproduce the issue?

  1. Start hasura in docker container as per https://hasura.io/docs/latest/graphql/core/getting-started/docker-simple.html#step-1-get-the-docker-compose-file

  2. Run following set of commands

hasura init hasura
cd hasura
hasura migrate create init --sql-from-file ../database_v2/artifacts/provision.sql --database-name default
#> WARN database default is not connected to hasura  
#> INFO Migrations files created                      name=init version=1631292489116
hasura migrate apply --database-name default
#> FATA[0001] database not found: error determining database kind for default, check if database exists on hasura

Any possible solutions?

Update documentation so all the steps are included and can be used to get started.

Can you identify the location in the source code where the problem exists?

I do not know the code base to be able to tell. As far as I can tell problem is that image starts without any database source and all my attempts to identify a way to start image with connected database source have failed. Here is the related issue #7538

If the bug is confirmed, would you be willing to submit a PR?

At the moment I do not know how to fix this, assuming I do understand how and have necessary skills I would.

BenoitRanque commented 3 years ago

This happens because you are trying to create a migration based on the current state of a database that is not yet connected to hasura, or perhaps you connected it under a diferent name? You could solve this by adding the database in the console, which is what the tutorial suggests.

Once the database is added, the connection will be recorded in the metadata. Bringing up the project would then require that this medata be applied before any operations involving the database (eg. migrations or seeds) be performed.

So typically, bringing up a project that is configured with metadata, migrations and seeds looks like this:

docker compose up -d
hasura metadata apply # at this point, the medata will be inconsistent, since migrations have not been applied yet
hasura migrate apply --database-name default #or omit the flag, and pick from the list
hasura seeds apply --database-name default #or omit the flag, and pick from the list
hasura metadata reload # hasura will check again and metadata should now be consistent with database information.

Alternatively, you could connect the database automatically as default using the environment variable HASURA_GRAPHQL_DATABASE_URL like so:

version: '3.6'
services:
  postgres:
    image: postgres:12
    restart: always
    volumes:
    - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
  graphql-engine:
    image: hasura/graphql-engine:v2.0.9
    ports:
    - "8080:8080"
    depends_on:
    - "postgres"
    restart: always
    environment:
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## this env var can be used to add the above postgres database to Hasura as a data source. this can be removed/updated based on your needs
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
      ## uncomment next line to set an admin secret
      # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
volumes:
  db_data:

This is legacy behavior from when hasura only supported one database, but is still perfectly fine.

Gozala commented 3 years ago

Alternatively, you could connect the database automatically as default using the environment variable HASURA_GRAPHQL_DATABASE_URL like so:

In fact I have had that set, but I think I did so after I've discovered this env veriable somewhere in the docs, but I tihnk it was not picked up because I've started docker image was cached or something.

After I have removed docker volume image etc... then default connection did appear, however I've burned a day on trying to get hasura image to start and use metadata / migrations without having to resort to UI interactions.

I think there some low hanging fruits here to make whole experience a lot smoother and confusing, here are few suggesitions:

  1. Myself and other person on a team both missunderstood quoted section from under adding connection in the docs:

    The docker-compose file has an additional env var PG_DATABASE_URL which points to the created metadata database. You can use this env var to connect the same database as a data source and continue.

    Which sounded like if you do have PG_DATABASE_URL set it will be used as data source connection, while what it actually, I think means you can copy value from there and paste it in the UI.

  2. Mentioning HASURA_GRAPHQL_DATABASE_URL in the quickstart with docker doc as an alternative to UI interaction would also have helped a lot.

  3. Adding a commented out HASURA_GRAPHQL_DATABASE_URL env variable, explaining that uncommenting it would add it to the default source connection would also make it a lot easier to discover. https://github.com/hasura/graphql-engine/blob/7c35ffb36561214390d0d545d418746f29a29ba4/install-manifests/docker-compose/docker-compose.yaml#L17-L28

  4. Adding non legacy alternative that does not involve UI interaction e.g. hasura source add default postgres://... or something would make a lot of sense as well.

  5. Make adding connections part of hasura init so you do not end up with graphql-engine without any connections.

There may be better alternatives. But so far my experience have been:

  1. Trying hasura cloud, feels nice and easy.
  2. Trying adopt it on actual project that uses version control and CI is confusing and frustrating.

Hope this feedback provides some useful insights that can help improve overall experience.

Gozala commented 3 years ago

I think this line in docker-compose.yml also seems to suggest that PG_DATABASE_URL would add it as a default data source. If it says something else then wording is really confusing

https://github.com/hasura/graphql-engine/blob/7c35ffb36561214390d0d545d418746f29a29ba4/install-manifests/docker-compose/docker-compose.yaml#L20-L21

BenoitRanque commented 3 years ago

The wording is correct, the load bearing word being can

this env var can be used to add the above postgres database to Hasura as a data source.

But I do agree we need better documentation on how this all works. Please bear in mind hasura's ability to handle multiple data sources is a fairly new feature, and the docs do need some improvements in that regard to make things clearer.

Importantly, PG_DATABASE_URL isn't a "special" variable to the hasura server, and only matters if you use it to establish a connection. You could name it literally anything else. Which is why it is missing from the server config reference