search

p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
70 stars 5 forks source link

Node crashes when receiving blob #562

Closed adzialocha closed 1 year ago

adzialocha commented 1 year ago

Steps to reproduce:

  1. Start two nodes
  2. Send a blob to one of them
  3. Restart one node
[2023-09-12T08:46:11Z INFO  aquadoggo::manager] Start materializer service
[2023-09-12T08:46:11Z INFO  aquadoggo::materializer::worker] Register reduce worker with pool size 16
[2023-09-12T08:46:11Z INFO  aquadoggo::materializer::worker] Register dependency worker with pool size 16
[2023-09-12T08:46:11Z INFO  aquadoggo::materializer::worker] Register schema worker with pool size 16
[2023-09-12T08:46:11Z INFO  aquadoggo::materializer::worker] Register blob worker with pool size 16
[2023-09-12T08:46:11Z INFO  aquadoggo::materializer::worker] Register garbage_collection worker with pool size 16
[2023-09-12T08:46:11Z DEBUG aquadoggo::materializer::service] Dispatch 0 pending tasks from last runtime
[2023-09-12T08:46:11Z DEBUG aquadoggo::materializer::service] Materialiser service is ready
[2023-09-12T08:46:11Z INFO  aquadoggo::manager] Start http service
[2023-09-12T08:46:11Z DEBUG aquadoggo::graphql::schema] Subscribing GraphQL manager to schema provider
[2023-09-12T08:46:11Z DEBUG aquadoggo::graphql::schema] Finished building initial GraphQL schema
HTTP port 2020 was already taken, try random port instead ..
Go to http://0.0.0.0:35569/graphql to use GraphQL playground
[2023-09-12T08:46:11Z DEBUG aquadoggo::http::service] HTTP service is ready
[2023-09-12T08:46:11Z INFO  aquadoggo::manager] Start network service
Peer id: 12D3KooWDPDvhGBK249kVkak3BGmfXuQvo6VSDErqc2BH2gqGFHG
[2023-09-12T08:46:11Z INFO  aquadoggo::network::service] Networking service initializing...
[2023-09-12T08:46:11Z DEBUG aquadoggo::network::behaviour] mDNS network behaviour enabled
[2023-09-12T08:46:11Z DEBUG aquadoggo::network::behaviour] Relay client network behaviour enabled
QUIC port 2022 was already taken, try random port instead ..
[2023-09-12T08:46:11Z INFO  aquadoggo::network::service] Network service ready!
[2023-09-12T08:46:11Z INFO  aquadoggo::manager] Start replication service
Node is listening on 0.0.0.0:47501
[2023-09-12T08:46:11Z DEBUG aquadoggo::network::service] mDNS discovered a new peer: 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq
[2023-09-12T08:46:11Z INFO  aquadoggo::replication::service] Established connection with peer: 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (1)
[2023-09-12T08:46:11Z INFO  aquadoggo::replication::service] Established connection with peer: 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (2)
[2023-09-12T08:46:16Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (1)
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51/-> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51/->
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce/-> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce/->
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Created <Document 16bd51>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33/-> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33/->
[2023-09-12T08:46:16Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (1)
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4/-> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4/->
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Created <Document 20abd4>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Created <Document 6b7a33>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 2 tasks
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Duplicate materializer dependency task already in progress, setting re-queue flag for task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> and not adding this task to the queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33>
[2023-09-12T08:46:16Z INFO  aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Created <Document d457ce>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 2 tasks
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Materializer dependency task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> not sent to queue as a task for this document has already been re-queued.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51>
[2023-09-12T08:46:16Z INFO  aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 2 tasks
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Materializer dependency task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> not sent to queue as a task for this document has already been re-queued.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce>
[2023-09-12T08:46:16Z INFO  aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4
[2023-09-12T08:46:16Z ERROR aquadoggo::materializer::worker] Critical error in worker blob with task <QueueItem 0 w. <TaskInput -/002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33>>: Could not write blob file @ /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4: Some pieces missing for the requested blob
[2023-09-12T08:46:16Z INFO  aquadoggo::manager] Received shutdown signal
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33
[2023-09-12T08:46:16Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (1)
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002083088e6d6d9f94f86d54c836bc8343a8305ad3698a3ec6ebcd3a0895aa6b7a33
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 1 tasks
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4> to the task queue.
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4>
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51
[2023-09-12T08:46:16Z INFO  aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4
[2023-09-12T08:46:16Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (2)
[2023-09-12T08:46:16Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWApRV2vny2MQb6Psbc5JxHw8ynJe3tXHfBVPNNcDRieMq (1)
[2023-09-12T08:46:16Z ERROR aquadoggo::materializer::worker] Critical error in worker blob with task <QueueItem 1 w. <TaskInput -/0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51>>: Could not write blob file @ /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4: background task failed
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020725a1aeb32236b41e3d24e89f9ec1ec253ba588a2d350d40e47629b23616bd51
[2023-09-12T08:46:16Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce
[2023-09-12T08:46:16Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 4 w. <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-09-12T08:46:16Z ERROR aquadoggo::materializer::worker] Critical error in worker blob with task <QueueItem 2 w. <TaskInput -/00207d38b972a8eb1407b169bf4388a77763bc7b19428f6fecef9fdcd46970d457ce>>: Could not write blob file @ /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4: background task failed
[2023-09-12T08:46:16Z ERROR aquadoggo::materializer::worker] Critical error in worker blob with task <QueueItem 3 w. <TaskInput -/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4>>: Could not write blob file @ /tmp/.tmpJA3c6E/0020f56b7856b3e383fd969ffd47375ab43d14cad30ebaa6cf5340cc1500ee20abd4: background task failed