Before materializing a blob to the file system the blob task checks if a file already exists at the expected path. If there is one then it aborts the task without completing. This check is there as blob tasks can be triggered multiple times and we only want it to complete once.
This is fine except for in the case where a node is shutdown or crashes while a blob task is running. In this case the file is only partially written to disc and even when the task is picked up again on next restart the blob task will not be retried because of the previously described check.
This PR fixes this issue by introducing a check which ensures the file existing at the blob path is the expected length. If not the task continues to materialize the blob at that path as it assumes it didn't complete last time.
closes: #617
📋 Checklist
[x] Add tests that cover your changes
[x] Add this PR to the Unreleased section in CHANGELOG.md
Before materializing a blob to the file system the blob task checks if a file already exists at the expected path. If there is one then it aborts the task without completing. This check is there as blob tasks can be triggered multiple times and we only want it to complete once.
This is fine except for in the case where a node is shutdown or crashes while a blob task is running. In this case the file is only partially written to disc and even when the task is picked up again on next restart the blob task will not be retried because of the previously described check.
This PR fixes this issue by introducing a check which ensures the file existing at the blob path is the expected length. If not the task continues to materialize the blob at that path as it assumes it didn't complete last time.
closes: #617
📋 Checklist
CHANGELOG.md