[Morse] Morse CLI & Data Corruption Investigations

Olshansk commented 2 months ago

Objective

Investigate, identify the root cause & fix potential data corruption (and other) issues in Morse.

Origin Document

Various issues spanning pokt nodes, pokt archival, CLI endpoints, etc...

Goals

Identify the root cause(s)
Understand how far these issues expand
Retain happy POKT users

Deliverables

[ ] Use this GitHub Issue to track ongoing parallel efforts & investigations across different individuals
[ ] Submit PRs & fixes to tackle each of them one by one (if applicable)
[ ] Find an explanation if the issue is not in the source code.

Olshansk commented 2 months ago

Issues with `pocket query supported-network`

What's going on?

A previously supported endpoint no longer malfunctions

Why?

Uknown but could range from:

Pruning issues
Data corruption issues
Issues with the latest release
Unsynched nodes
Etc...

Quick Triage

I haven't done an investigation into this, but have tried the endpoints and found the following.

This provides an empty response:

pocket query supported-networks  130000 --remoteCLIURL ${POCKET_ENDPOINT}

So does this:

curl -v -s -X POST ${POCKET_ENDPOINT}/v1/query/supportedchains

Current workaround

A workaround is:

pocket query params  130000 --remoteCLIURL ${POCKET_ENDPOINT} | tail -n +2 | jq '.pocket_params[] | select(.param_key == "pocketcore/SupportedBlockchains") | .param_value'

[\"0001\",\"0003\",\"0004\",\"0005\",\"0006\",\"0009\",\"000A\",\"000B\",\"000C\",\"000F\",\"0010\",\"0012\",\"0021\",\"0022\",\"0024\",\"0025\",\"0026\",\"0027\",\"0028\",\"0040\",\"0044\",\"0046\",\"0047\",\"0048\",\"0049\",\"0050\",\"0051\",\"0052\",\"0053\",\"0054\",\"0056\",\"0057\",\"0058\",\"0059\",\"0060\",\"0061\",\"0063\",\"0065\",\"0066\",\"0067\",\"0068\",\"0069\",\"0070\",\"0071\",\"0072\",\"0074\",\"0075\",\"0076\",\"0077\",\"0078\",\"0079\",\"0080\",\"0081\",\"0082\",\"0083\",\"0084\",\"03DF\",\"A001\",\"A003\",\"A053\",\"A085\",\"A086\",\"A087\",\"A088\",\"A089\",\"A0CA\",\"B021\",\"B081\",\"C006\"]"

Olshansk commented 2 months ago

@msmania As the current maintainer of this codebase, could you look into this?

msmania commented 2 months ago

Both pocket query supported-networks and /v1/query/supportedchains work well on my end. What was your POCKET_ENDPOINT?

Olshansk commented 2 months ago

Closing this out. Lots of offline discussions with @msmania, @kutoft, @adshmh and others leading us to determine that this issue was related to response unmarshalling on the Gateway side.

Thanks to everyone involved!

pokt-network / pocket-core