[Argus Node] 500 errors for image fetching

[ignazio-bovo]

When fetching image 1554542 which is stored on storage nodes: https://storage.freakstatic.com/storage/api/v1/files/1454542 https://adovrn-joystream.yyagi.cloud/storage/api/v1/files/1454542

We are getting 500 errors on all servers

[ignazio-bovo]

The issue is that for that asset the QN object size is different from the size fetched by Argus https://github.com/Joystream/joystream/blob/da2f47b00bd85ee39dd20c6f64403d53056820ba/distributor-node/src/services/content/ContentService.ts#L266-L267

[mnaamani]

Can we determine which bag this object is in so we can check its size according to the runtime?

[mnaamani]

Query Node reports:

  "data": {
    "storageDataObjects": [
      {
        "id": "1454542",
        "storageBagId": "dynamic:channel:29895",
        "size": "20022",
        "ipfsHash": "gWBKXpt5W4pEtPyyk73rnbkb7zyT1kdD2F7vvn1BFpyB73"
      }
    ]
  }
}

Which matches with runtime:

So at least we know the expected size is correct.

Fetching the file and checking hash:

yarn storage-node util:multihash -f ~/tmp/1454542.jpg 
yarn run v1.22.19
$ /Users/mokhtar/joystream/joystream/node_modules/.bin/storage-node util:multihash -f /Users/mokhtar/tmp/1454542
2024-01-09 10:57:02:572 info: Hashing /Users/mokhtar/tmp/1454542 ....
2024-01-09 10:57:02:572 info: Hash: gWCT6Z3stvybLos6f3EzJKtkSM57D6eXiCtCcT5toNUzMQ

renamed the file to 1454542.jpg and it opens as a valid image file.

Files from both sources are the same but do not match runtime/QN.

So there are two problems..

• How did the wrong file make it into the uploads folder (without manual intervention)
• If a node had a corrupt/wrong file.. how did the other nodes sync an invalid object?

[mnaamani]

On further investigation it was found that this particular file was manually added by operator onto multiple nodes and it did not match the object created in the runtime.

[kdembler]

@ignazio-bovo given that we have found the root of the problem, I think we can close this?