search

OriginTrail / ot-node

OriginTrail Decentralized Knowledge Graph network node
https://origintrail.io
Apache License 2.0
207 stars 75 forks source link

Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') #3394

Open haroldboom opened 3 weeks ago

haroldboom commented 3 weeks ago

Issue description

Freshy rebuilt node, new paranets and node is in a reboot loop

Expected behavior

Normal startup and run

Actual behavior

Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Storing new session stream for remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf with operation id: cce218e5-a8a6-4663-a7a5-af591e8e63cc Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Receiving message from QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /get/1.0.0, messageType: PROTOCOL_INIT; Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] USERLVL: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) level-change Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Checking if assertion 0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320 exists for state 0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320, on blockchain: base:84532, contract: 0xb8B904c73D2fB4D8c173298A51c27Fab70222c32, and tokenId: 5683591 Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Checking if assertion exists in pending storage at path: /root/ot-node/data/pending_storage_cache/public/base:84532:0xb8b904c73d2fb4d8c173298a51c27fab70222c32:5683591/0xe388cf288822784ae94b2e5cb481ce6c6862c4d958392c1f9fd0d0921beed320 Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Node connected to QmNpDvED1ud6N8HpH2KUG1X6v2Vcp3h4YsFYbki4SWxHTQ, updating sharding table last seen and last dialed. Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Sending response from QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv to QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf: protocol: /get/1.0.0, messageType: NACK; Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Session found remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, operation id: cce218e5-a8a6-4663-a7a5-af591e8e63cc Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Removed session for remotePeerId: QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, operationId: cce218e5-a8a6-4663-a7a5-af591e8e63cc. Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] TRACE: Storing new session stream for remotePeerId: QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 with operation id: e6cfb4e6-2397-456f-8ec0-b05272c43b76 Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: Receiving message from QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /store/1.0.0, messageType: PROTOCOL_REQUEST; Nov 05 01:07:48 curated2 node[697867]: [2024-11-05 01:07:48] DEBUG: {Operation ID: cce218e5-a8a6-4663-a7a5-af591e8e63cc} [v1_0_0HandleGetInitCommand] (Command ID: ea436511-3be2-4efd-bc6f-d58230093af5) Reading operation id: e6cfb4e6-2397-456f-8ec0-b05272c43b76 cached data from file Nov 05 01:07:48 curated2 node[697867]: Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') Nov 05 01:07:48 curated2 node[697867]: at PublishController.v1_0_0HandleRequest (file:///root/ot-node/8.1.1+beta.3/src/controllers/rpc/publish-rpc-controller.js:33:45) Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Main process exited, code=exited, status=1/FAILURE Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Failed with result 'exit-code'. Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Consumed 10.861s CPU time. Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Scheduled restart job, restart counter is at 6. Nov 05 01:07:48 curated2 systemd[1]: Stopped OriginTrail V8 Node. Nov 05 01:07:48 curated2 systemd[1]: otnode.service: Consumed 10.861s CPU time.

Steps to reproduce the problem

  1. Only seems to be on 1 of my nodes, all identical hardware and publishing scripts, paranets etc. I tried fully removing ot-node folder and also the mysql databases and starting over but somehow it keeps crashing with the same error

Specifications

Error logs

Nov 05 01:05:34 curated2 node[697766]: [2024-11-05 01:05:34] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmbKh9rxcfEeysLu3DbwDfAVBRNPtjMkk1ngqs2jEPEUqf, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to Qmbe7mDtqgaXVgMjUT6BgeAUQK7ybtFeue7Sw21NYYRVwF, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmVikRKmkqrxEg2q8uXza3uLtRE5WLNEMCfX7WgrCBVAUw, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to QmemEAoebqoXnv8ZqRjn25WtuBkH45MDU1gFyFARbudNhy, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Node connected to Qmcx6RG1bA4yaZz2tDyDfep3h8xDYDyEgX6dmnNCRkVAnU, updating sharding table last seen and last dialed. Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] TRACE: Storing new session stream for remotePeerId: QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 with operation id: c2679a89-ff15-4886-ae74-c46ebdd2fd1b Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] DEBUG: Receiving message from QmaG78PiYa97z6ArM7uYZkSXaX1SCGWpwBGPspDq1xgm71 to QmPs2wjFAtb4uRuPZyKF2dt2gjG7vBdRu1bdbv2zW66iSv: protocol: /store/1.0.0, messageType: PROTOCOL_REQUEST; Nov 05 01:05:35 curated2 node[697766]: [2024-11-05 01:05:35] DEBUG: {Operation ID: 74543a8d-b2a5-4d1f-899b-a24719f51710} [findNodesCommand] (Command ID: 2989ce18-299b-4ed5-8924-c22b315fa3e5) Reading operation id: c2679a89-ff15-4886-ae74-c46ebdd2fd1b cached data from file Nov 05 01:05:35 curated2 node[697766]: Something went really wrong! OT-node shutting down... TypeError: Cannot read properties of undefined (reading 'assertionId') Nov 05 01:05:35 curated2 node[697766]: at PublishController.v1_0_0HandleRequest (file:///root/ot-node/8.1.1+beta.3/src/controllers/rpc/publish-rpc-controller.js:33:45) Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Main process exited, code=exited, status=1/FAILURE Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Failed with result 'exit-code'. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Consumed 10.178s CPU time. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Scheduled restart job, restart counter is at 4. Nov 05 01:05:35 curated2 systemd[1]: Stopped OriginTrail V8 Node. Nov 05 01:05:35 curated2 systemd[1]: otnode.service: Consumed 10.178s CPU time.

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to tech@origin-trail.com.

botnumberseven commented 3 weeks ago

Do all your nodes use the same RPC?

haroldboom commented 3 weeks ago

Do all your nodes use the same RPC?

Split between 2 different RPCs