"Getting the most recent point with a new asset" endless loop

Environment

I'm using official cardano-graphql:5.1.0 and cardano-graphql-hasura:5.1.0 docker images.

{
  os: {
    arch: 'x64',
    cpus: { count: 4, model: 'AMD EPYC 7282 16-Core Processor' },
    release: '5.4.0-80-generic',
    totalMem: 8344616960,
    type: 'Linux'
  },
  version: '5.1.0'
}

Platform

[x] Linux / Ubuntu
[ ] Linux / Other
[ ] macOS
[ ] Windows

Platform version:

Runtime

[ ] Node.js
[x] Docker

Runtime version:

Node:
```
$ node -v
v14.17.4
```

Docker (well containerd from Kubernetes):

OS-IMAGE                        KERNEL-VERSION     CONTAINER-RUNTIME
Ubuntu 20.04.1 LTS 31403879ea   5.4.0-80-generic   containerd://1.5.4

Steps to reproduce the bug

It goes into this loop while booting and never brings the API up.

What is the expected behavior?

That it boots :)

Additional details

After enabling debug output, I found that it goes into an endless loop with this:

{"name":"cardano-graphql","hostname":"cardano-graphql-75f8d6f79f-tjlnf","pid":1,"level":20,"msg":"Getting the most recent point with a new asset: Attempt 1 of 1001, retrying...","time":"2021-09-13T21:54:49.162Z","v":0}

I've checked the query that it's performed for that task and realised it returns empty results in this deployment:

root@cardano-node-0:/# curl 'http://hasura-headless:8080/v1/graphql' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' -H 'DNT: 1' - --data-binary '{"query":"query {\n            assets (\n                limit: 1\n                offset: 1\n                order_by: { firstAppearedInBlock: { slotNo: desc }}\n            ) {\n                firstAppearedInBlock {\n                    hash\n                    slotNo\n                }\n            }\n        }"}' --compressed

{"data":{"assets":[{"firstAppearedInBlock":null}]}}

While it works in other ones I have:

root@cardano-node-0:/# curl 'http://hasura-headless:8080/v1/graphql' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary '{"query":"query {\n            assets (\n                limit: 1\n                offset: 1\n                order_by: { firstAppearedInBlock: { slotNo: desc }}\n            ) {\n                firstAppearedInBlock {\n                    hash\n                    slotNo\n                }\n            }\n        }"}' --compressed

{"data":{"assets":[{"firstAppearedInBlock":{"hash":"\\x7d101f5d5c73e1e0827a1a1283086def2e58e1d5c3beb1e7ddaa27dd756dd353","slotNo":37251447}}]}}

I've tried restarting db-sync, hasura and ogmios also, just to see if there was some problem with any schema/table/view that would recover on restart, but no luck :( It seems db might have been corrupted because of some graphql abrupt restart (I had some), but wanted to report the issue just in case.

cardano-foundation / cardano-graphql