p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Re-run tasks for partially materialized blobs #618

Closed sandreae closed 2 months ago

sandreae commented 2 months ago

Before materializing a blob to the file system the blob task checks if a file already exists at the expected path. If there is one then it aborts the task without completing. This check is there as blob tasks can be triggered multiple times and we only want it to complete once.

This is fine except for in the case where a node is shutdown or crashes while a blob task is running. In this case the file is only partially written to disc and even when the task is picked up again on next restart the blob task will not be retried because of the previously described check.

This PR fixes this issue by introducing a check which ensures the file existing at the blob path is the expected length. If not the task continues to materialize the blob at that path as it assumes it didn't complete last time.

closes: #617

📋 Checklist

sandreae commented 2 months ago

Some clippy errors unrelated to this PR....

adzialocha commented 2 months ago

Some clippy errors unrelated to this PR....

Probably a new Rust version!