Request failed with status code 504 starting MetadataFetchClient

[leslieonline27]

GraphQL v5.1.0

Platform

• [x] Linux / Ubuntu
• [ ] Linux / Other
• [ ] macOS
• [ ] Windows

Platform version:

Runtime

• [ ] Node.js
• [x] Docker

Runtime version:

Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:01 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:13 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps to reproduce the bug

Run docker-compose up -d and allow everything to sync to the tip. Run docker-compose down and then docker-compose up -d again. GraphQL takes a while to start and this is in the log:

{"name":"cardano-graphql","hostname":"8af28a781f1f","pid":1,"level":30,"module":"CardanoNodeClient","msg":"Initialized","time":"2021-09-08T20:40:37.057Z","v":0}
{"name":"cardano-graphql","hostname":"8af28a781f1f","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2021-09-08T20:40:37.057Z","v":0}
{"name":"cardano-graphql","hostname":"8af28a781f1f","pid":1,"level":50,"msg":"Request failed with status code 504","time":"2021-09-08T20:41:37.568Z","v":0}

GraphQL will not respond to requests on 3100

What is the expected behavior?

All components start successfully and GraphQL responds to requests

[CyberCyclone]

How long did you leave it after starting up? If it was mainnet, depending on your hardware it can take about 10 minutes for cardano-node / ogmios to start up. That needs to have finished starting up before everything comes online.

[leslieonline27]

Yep - left the entire stack for at least an hour before checking on it again. db-sync is up and running ingesting blocks and so is ogmios, but graphql isn't happy

[CyberCyclone]

Just double checking, was this mainnet or a different net? Mainnet is on port 3100, other nets might be on a different port.

[leslieonline27]

Mainnet

[CyberCyclone]

Was this a completely fresh setup or is it an upgrade from a previous cardano-graphql version?

I've just restarted my 5.1.0 and it started fine. Could you please trying bringing it down again and try bringing it up? I'm wondering if it glitched when trying to connect to the metadata server.

[cardistack]

same here. Starts fine wait for a few days to sync then graphql wont listen on 3100

{"name":"cardano-graphql","hostname":"f136361575bc","pid":1,"level":30,"module":"CardanoNodeClient","msg":"Initialized","time":"2021-09-10T08:26:43.076Z","v":0}

{"name":"cardano-graphql","hostname":"f136361575bc","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2021-09-10T08:26:43.077Z","v":0}

{"name":"cardano-graphql","hostname":"f136361575bc","pid":1,"level":50,"msg":"Request failed with status code 504","time":"2021-09-10T08:27:43.231Z","v":0}

I have just deleted the whole postgres volume in docker . Its now started ok . However thats what happens and then after a few days stops

{"name":"cardano-graphql","hostname":"a617837d1034","pid":1,"level":30,"module":"Server","msg":"DB sync progress: 1.82 %","time":"2021-09-10T10:47:01.932Z","v":0}

{"name":"cardano-graphql","hostname":"a617837d1034","pid":1,"level":30,"module":"Server","msg":"DB sync progress: 1.82 %","time":"2021-09-10T10:47:06.949Z","v":0}

{"name":"cardano-graphql","hostname":"a617837d1034","pid":1,"level":30,"module":"Server","msg":"DB sync progress: 1.82 %","time":"2021-09-10T10:47:11.945Z","v":0}

{"name":"cardano-graphql","hostname":"a617837d1034","pid":1,"level":30,"module":"Server","msg":"DB sync progress: 1.82 %","time":"2021-09-10T10:47:16.938Z","v":0}

{"name":"cardano-graphql","hostname":"a617837d1034","pid":1,"level":30,"module":"Server","msg":"DB sync progress: 1.82 %","time":"2021-09-10T10:47:21.930Z","v":0}

[rhyslbw]

The metadata service has been a bit flaky over the past couple of days, which will cause issues on cardano-graphql startup as the service needs to be healthy at the time of booting. The responsible team are aware and are making changes to address things. Is it still happening today?

[leslieonline27]

I've restarted the stack a bunch of times and the error has eventually gone away - seems like the metadata service is the likely culprit so if it's being addressed this issue can probably be closed