search

hasura / graphql-engine

Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
https://hasura.io
Apache License 2.0
31.13k stars 2.76k forks source link

Matadata apply timing out #8565

Open simonphughes opened 2 years ago

simonphughes commented 2 years ago

Version Information

Server Version: 2.0.1 CLI Version (for CLI related issue): 2.7.0

Environment

AWS Cloud Hosted - PostgreSQL running on Aurora Serverless, hasura running in 2 containers on ECS fargate, frontend by Load Balancer. All based on cloudformation script provided.

What is the expected behaviour?

Migrations should run quickly without timing out.

Keywords

504 gateway timeout migrations

What is the current behaviour?

Just started to hit some issues with running migrations as we move our configuration from dev through to test and production. Running our migration of metadata we are now hitting issues where the load balancer is timing out and returning a http 504 error. The command thats causing issues is hasura metadata apply --endpoint https://myendpoint.com/ --admin-secret mypassword

Have tried increasing the timeout on the load balancer to over 30 minutes and still not able to push any metadata changes.

How to reproduce the issue?

  1. As above, with a reasonably complex platform consisting of 50 related tables with a suite of permissions on them for multiple roles.
  2. Try to apply metadata chnages

Screenshots or Screencast

time="2022-06-08T12:24:30+01:00" level=fatal msg="error applying metadata \n\r\n504 Gateway Time-out\r\n\r\n

504 Gateway Time-out

\r\n\r\n\r\n"

Please provide any traces or logs that could help here.

Happy to provide any logs you can suggest.

Any possible solutions?

Unknown. At this stage I have a production system is use that I can no longer apply updates to.

Can you identify the location in the source code where the problem exists?

No

If the bug is confirmed, would you be willing to submit a PR?

Happy to help.

efstathiosntonas commented 3 months ago

@simonphughes sorry for digging this from the grave, have you found a solution?