Error updating Hasura GraphQL engine on Heroku

[realph]

I've followed the guide from the docs step by step, and ran git push heroku master. But now I'm getting an Application Error when trying to run my app:

This is what my logs look like.

2019-07-07T20:40:31.189138+00:00 heroku[web.1]: State changed from starting to crashed
2019-07-07T20:40:31.173620+00:00 heroku[web.1]: Process exited with status 1
2019-07-07T20:40:31.021574+00:00 app[web.1]: {"timestamp":"2019-07-07T20:40:31.021+0000","level":"info","type":"startup","detail":{"kind":"serve_options","info":{"enable_allowlist":false,"auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enable_telemetry":true,"enable_console":true,"auth_hook":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":true,"port":13719}}}
2019-07-07T20:40:31.115492+00:00 app[web.1]: {"internal":{"statement":"select\n  json_build_object(\n    'tables', tables.items,\n    'relations', relations.items,\n    'permissions', permissions.items,\n    'query_templates', query_templates.items,\n    'event_triggers', event_triggers.items,\n    'remote_schemas', remote_schemas.items,\n    'functions', functions.items,\n    'foreign_keys', foreign_keys.items,\n    'allowlist_collections', allowlist.item\n  )\nfrom\n  (\n    select\n      coalesce(json_agg(\n        json_build_object(\n          'table',\n          json_build_object(\n            'name', ht.table_name,\n            'schema', ht.table_schema\n          ),\n          'system_defined', ht.is_system_defined,\n          'info', tables.info\n        )\n      ), '[]') as items\n    from\n      hdb_catalog.hdb_table as ht\n      left outer join (\n        select\n          table_schema,\n          table_name,\n          json_build_object(\n            'name',\n            json_build_object(\n              'schema', table_schema,\n              'name', table_name\n            ),\n            'columns', columns,\n            'primary_key_columns', primary_key_columns,\n            'constraints', constraints,\n            'view_info', view_info\n          ) as info\n        from\n          hdb_catalog.hdb_table_info_agg\n      ) as tables on (\n        tables.table_schema = ht.table_schema\n        and tables.table_name = ht.table_name\n      )\n  ) as tables,\n  (\n    select\n      coalesce(\n        json_agg(\n          json_build_object(\n            'table',\n            json_build_object(\n              'schema', table_schema,\n              'name', table_name\n            ),\n            'rel_name', rel_name,\n            'rel_type', rel_type,\n            'def', rel_def :: json,\n            'comment', comment\n          )\n        ),\n        '[]'\n      ) as items\n    from\n      hdb_catalog.hdb_relationship\n  ) as relations,\n  (\n    select\n      coalesce(\n        json_agg(\n          json_build_object(\n            'table',\n            json_build_object(\n              'schema', table_schema,\n              'name', table_name\n            ),\n            'role', role_name,\n            'perm_type', perm_type,\n            'def', perm_def :: json,\n            'comment', comment\n          )\n        ),\n        '[]'\n      ) as items\n    from\n      hdb_catalog.hdb_permission\n  ) as permissions,\n  (\n    select\n      coalesce(\n        json_agg(\n          json_build_object(\n            'name', template_name,\n            'def', template_defn :: json\n          )\n        ),\n        '[]'\n      ) as items\n    from\n      hdb_catalog.hdb_query_template\n  ) as query_templates,\n  (\n    select\n      coalesce(\n        json_agg(\n          json_build_object(\n            'table',\n            json_build_object(\n              'schema', schema_name,\n              'name', table_name\n            ),\n            'name', name,\n            'def', configuration :: json\n          )\n        ),\n        '[]'\n      ) as items\n    from\n      hdb_catalog.event_triggers\n  ) as event_triggers,\n  (\n    select\n      coalesce(\n        json_agg(\n          json_build_object(\n            'name',\n            name,\n            'definition', definition :: json,\n            'comment', comment\n          )\n        ),\n        '[]'\n      ) as items\n    from\n      hdb_catalog.remote_schemas\n  ) as remote_schemas,\n  (\n    select\n      coalesce(json_agg(q.info), '[]') as items\n    from\n        (\n        select\n          json_build_object(\n            'function',\n            json_build_object(\n              'schema', hf.function_schema,\n              'name', hf.function_name\n            ),\n            'info', function_info\n          ) as info\n        from\n          hdb_catalog.hdb_function hf\n        left outer join\n            hdb_catalog.hdb_function_info_agg hf_agg on\n            ( hf_agg.function_name = hf.function_name\n              and hf_agg.function_schema = hf.function_schema\n            )\n      ) as q\n   ) as functions,\n  (\n    select\n      coalesce(json_agg(foreign_key.info), '[]') as items\n    from\n      (\n        select\n          json_build_object(\n            'table',\n            json_build_object(\n              'schema', f.table_schema,\n              'name', f.table_name\n            ),\n            'ref_table',\n            json_build_object(\n              'schema', f.ref_table_table_schema,\n              'name', f.ref_table\n            ),\n            'constraint', f.constraint_name,\n            'column_mapping', f.column_mapping\n          ) as info\n        from\n         hdb_catalog.hdb_foreign_key_constraint f\n         left outer join hdb_catalog.hdb_table ht\n         on ( ht.table_schema = f.table_schema\n              and ht.table_name = f.table_name\n            )\n      ) as foreign_key\n  ) as foreign_keys,\n  (\n    select\n      coalesce(json_agg(hqc.collection_defn), '[]') as item\n    from hdb_catalog.hdb_allowlist ha\n    left outer join\n         hdb_catalog.hdb_query_collection hqc\n         on (hqc.collection_name = ha.collection_name)\n  ) as allowlist\n","prepared":true,"error":{"exec_status":"FatalError","hint":null,"message":"relation \"hdb_catalog.hdb_table_info_agg\" does not exist","status_code":"42P01","description":null},"arguments":[]},"path":"$","error":"postgres query error","code":"unexpected"}

I'm not sure what to do next? Any help would be appreciated.

[0x777]

@realph Hi, what's your current version? And you are trying to update to the latest version, beta.2?

[realph]

@0x777 I'm not sure what my current version before it broke was. I think it was beta.1 and yep, I'm trying to update to beta.2.

[realph]

Tried with hasura/graphql-engine:v1.0.0-beta.3 also. But still the same error.

[thfrei]

Same error here. I guess there are some migration scripts missing? I tried updateing from hasura/graphql-engine:v1.0.0-alpha33 tohasura/graphql-engine:v1.0.0-beta.3`

[thfrei]

Using hasura/graphql-engine:v1.0.0-alpha45 works. I guess there might be some breaking changes from alpha to beta regarding postgres-tables.

How can I check if the migration script executed? Found the table here: https://github.com/hasura/graphql-engine/blob/204cd3514b89ae5fa9be7cf66dfa1dd4fb752e46/server/src-rsr/migrate_from_13_to_14.sql

[thfrei]

Since it seems to be a mgirations issue, I tried the migrations container as described here: https://docs.hasura.io/1.0/graphql/manual/migrations/auto-apply-migrations.html

So I tried this version: FROM hasura/graphql-engine:v1.0.0-beta.3.cli-migrations

FROM hasura/graphql-engine:v1.0.0-beta.3.cli-migrations

# Enable the console
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=false

# Change $DATABASE_URL to your heroku postgres URL if you're not using
# the primary postgres instance in your app
CMD graphql-engine \
    --database-url $DATABASE_URL \
    serve \
    --server-port $PORT

## Comment the command above and use the command below to
## enable an access-key and an auth-hook
## Recommended that you set the access-key as a environment variable in heroku
#CMD graphql-engine \
#    --database-url $DATABASE_URL \
#    serve \
#    --server-port $PORT \
#    --access-key XXXXX \
#    --auth-hook https://myapp.com/hasura-webhook
#
# Console can be enable/disabled by the env var HASURA_GRAPHQL_ENABLE_CONSOLE

However now the build fails:

2019-07-23T09:45:56.008522+00:00 app[web.1]: Fatal Error : Invalid options. Expecting all database connection params (host, port, user, dbname, password) or database-url (HASURA_GRAPHQL_DATABASE_URL)
2019-07-23T09:45:56.022868+00:00 app[web.1]: {"timestamp":"2019-07-23T09:45:56.008+0000","level":"info","type":"startup","detail":{"kind":"server_configuration","info":{"live_query_options":{"fallback_options":{"refetch_delay":1000},"multiplexed_options":{"batch_size":100,"refetch_delay":1000}},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","enabled_log_types":["unstructured","jwk-refresh-log","ws-server","http-log","schema-sync-thread","websocket-log","startup","webhook-log","pg-client","telemetry-log","metadata","event-trigger"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":false,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":true,"port":8080}}}
2019-07-23T09:46:24.419227+00:00 heroku[web.1]: State changed from starting to crashed
2

[brandonmp]

@thfrei finding same problem, basically same dockerfile as you. Removing the .cli-migrations and using the plain Hasura image works just fine.

this dockerfile uses beta.6, but beta.4 fails as well

FROM hasura/graphql-engine:v1.0.0-beta.6.cli-migrations

VOLUME ./migrations/migrations:/hasura-migrations

ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true

CMD graphql-engine \
    --database-url $DATABASE_URL \
    serve \
    --server-port $PORT 

[mikebobadilla]

Yea same thing happened to me. I had to reset the database and then push the new hasura engine. I didn't have much data in there so it wasn't a big deal.

[brandonmp]

@mikebobadilla nice. i'm working w/ a fresh database each deploy (heroku review apps), so unfortunately i don't think that's the fix in my case

[thfrei]

@brandonmp Thanks, so it seems to work with beta6?

As to the problem: I did fresh installation also, since i could not debug it otherwise. I guess we can close it, since this issue onnly applies to old installations and seems to be fixed in beta6?

[brandonmp]

@thfrei the non-migrations beta6 image spins up fine, but the migrations docker doesn't work.

I think it has to do with the fact that the migrations container relies on an entrypoint, which doesn't seem to jive with Heroku review apps.

I've given up that avenue and instead am trying to execute migrations via a release script, but I don't know Heroku well enough to make this work

I have an open SO question (with a 500 point bounty if anyone's interested), so hoping i can get this sorted out.

[jkopb]

In case anyone else is experiencing issues - I just had a similar issue when upgrading from beta6 to beta9, giving this error:

Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch

Apparently, the version migration takes too long (at least with my database) to be performed as part of the heroku deployment. The solution was to simply run a local hasura docker container on my computer, which could perform the migrations without worrying about a timeout. After this, beta9 deploys without any issues.

[lexi-lambda]

I’m a bit late to this discussion, but I think the issue described in this thread—namely, SQL errors when migrating from certain versions—was fixed in v1.0.0-beta.8, so I am inclined to close it. Please feel free to follow up if you find otherwise, and we can investigate further.

@jbuverud, the issue you’re describing sounds like something different: it sounds like the migrations do eventually work, but they’re too slow. If you wouldn’t mind, it would be very helpful if you could open a new issue on the hasura/graphql-engine repo—if you can share some information about your schema, we can figure out what’s causing the problem.

[Emiliano-Bucci]

@lexi-lambda Hi! I write here because it seems the only related ticket. I've recently updated to graphql-engine to v2.8.3 and i've started to get the following error:

Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch

I've also tried to install a previous version but without luck.

I have an hasura instance deployed in Heroku with the free tier; now, though, is down and it seems that the issues started after a heroku maintenance procedure.

Is there any help you can give me? I really don't know what to do :/ Thanks!