search

p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Materialised blobs sometimes corrupted / missing bytes #586

Closed sandreae closed 10 months ago

sandreae commented 10 months ago

Sometimes blobs are materialised with missing bytes. Noticed mostly when uploading larger files.

sandreae commented 10 months ago

I wonder if it's connected to https://github.com/p2panda/aquadoggo/issues as we use the pagination API to retrieve blob pieces.

adzialocha commented 10 months ago

Hmn, looking at the pictures I doubt that, the data is there, its just corrupted? If it would be the pagination issues we would have whole parts missing?

sandreae commented 10 months ago

Not sure about the images, but I've observed materialised blobs (uploaded successfully using send-to-node) having a smaller file size (roughly 40mb upload losing a couple of mb). I assume because it's missing pieces. I'll try and write a test for this though to support the theory.

adzialocha commented 10 months ago

Yeah, with large blobs it's possible as pagination will play a role then.

sandreae commented 10 months ago

Aha! It indeed wasn't the pagination, but the fact that write isn't guaranteed to successfully write all bytes from a buffer to a file :roll_eyes:

Fixed here: https://github.com/p2panda/aquadoggo/pull/587