search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Object Upload fails but still set as accepted on chain #5097

Open yasiryagi opened 6 months ago

yasiryagi commented 6 months ago

Issue:

Uploads that has failed , but still marked as accepted on chain

Evidence:

dataObjectId: 2100687

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "2100687"
      }
    ]
  }
}
Internal Server Error: Upload aborted
    at error (/joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:123:20)
    at /joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:16:26
    at Immediate._onImmediate (/joystream/node_modules/multer/lib/make-middleware.js:53:37)
    at process.processImmediate (node:internal/timers:478:21)

dataObjectId: 2112736

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "2112736"
      }
    ]
  }
}
uncaughtException: Upload aborted
Internal Server Error: Upload aborted
    at error (/joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:123:20)
    at /joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:16:26
    at Immediate._onImmediate (/joystream/node_modules/multer/lib/make-middleware.js:53:37)
    at process.processImmediate (node:internal/timers:478:21)

Error: Unexpected end of JSON input
    at new ApolloError (/joystream/node_modules/@apollo/client/errors/errors.cjs:34:28)
    at /joystream/node_modules/@apollo/client/core/core.cjs:1654:19
    at both (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:986:53)
    at /joystream/node_modules/@apollo/client/utilities/utilities.cjs:979:72
    at new Promise (<anonymous>)
    at Object.then (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:979:24)
    at Object.error (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:987:49)
    at notifySubscription (/joystream/node_modules/zen-observable/lib/Observable.js:140:18)
    at onNotify (/joystream/node_modules/zen-observable/lib/Observable.js:179:3)
    at SubscriptionObserver.error (/joystream/node_modules/zen-observable/lib/Observable.js:240:7)

dataObjectId: 1713397

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "1713397"
      }
    ]
  }
}
uncaughtException: Timeout
Error
    at new module.exports.TimeoutError (/joystream/node_modules/promise-timeout/index.js:44:16)
    at module.exports.timeout (/joystream/node_modules/promise-timeout/index.js:20:15)
    at extrinsicWrapper (/joystream/storage-node/lib/services/runtime/extrinsics.js:207:45)
    at acceptPendingDataObjects (/joystream/storage-node/lib/services/runtime/extrinsics.js:115:18)
    at uploadFile (/joystream/storage-node/lib/services/webApi/controllers/filesApi.js:99:57)

dataObjectId: 2112361

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "2112361"
      }
    ]
  }
}
Error: Server is not accepting uploads into this bucket
    at validateUploadFileParams (/joystream/storage-node/lib/services/webApi/app.js:199:15)
    at app.use.OpenApiValidator.middleware.apiSpec (/joystream/storage-node/lib/services/webApi/app.js:59:13)
    at Layer.handle [as handle_request] (/joystream/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/joystream/node_modules/express/lib/router/index.js:317:13)
    at /joystream/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/joystream/node_modules/express/lib/router/index.js:335:12)
    at next (/joystream/node_modules/express/lib/router/index.js:275:10)
    at /joystream/storage-node/lib/services/webApi/app.js:54:9
    at Layer.handle [as handle_request] (/joystream/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/joystream/node_modules/express/lib/router/index.js:317:13)
ignazio-bovo commented 3 months ago

I am not fully convinced that these error refers to the above objectId:

Background on upload file flow

1️⃣ Middleware chain

Currently on uploads we rely on the following middlewares chain:

  1. verifyUploadFileParams middleware
  2. OpenApiValidator.middleware with multer for uploads, into tempDirectory If 1 throws then 2 is not called.

    2️⃣ Route handler

    Once file upload has succesfully executed, the route handleruploadFile is caled and basically moves the file to the pendingDirectory after verifying the hash file

    3️⃣ Accepting object service

    Once every 6s (i.e. every target block production rate) the AcceptPendingObject service is called which processes objects in the pendingDirectory and then issues the accepting transaction for each of these. Only after the tx is successfully issued the objects are either moved into the uploadDir (from where they are actually served).

The errors describes appears to be Middleware errors (except for object 1713397) for which Express returns an error and the following steps 2️⃣ and 3️⃣ are not pursued.
Objects 1713397 encounters a timeout error in the acceptPendingDataObjects transaction.

ignazio-bovo commented 3 months ago

I am investigating on the first object and:

{
    "data": {
        "storageDataObjectById": {
            "isAccepted": true,
            "storageBag": {
                "id": "dynamic:channel:42488",
                "storageBuckets": [
                    {
                        "id": "10-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage2.sieemmastorage.com/storage/"
                            },
                            "id": "10"
                        }
                    },
                    {
                        "id": "1-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage.mrbovo.xyz/storage/"
                            },
                            "id": "1"
                        }
                    },
                    {
                        "id": "2-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage.joystream.8k.pm/storage/"
                            },
                            "id": "2"
                        }
                    },
                    {
                        "id": "3-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://joystream.name/storage/"
                            },
                            "id": "3"
                        }
                    }
                ]
            },
            "acceptingBucketId": "12"
        }
    }
}
ignazio-bovo commented 3 months ago

I think this is worth a task on his own and not be included in the s3 epic as it might be require more time and investigation than needed