search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

[Colossus] Serve objects from pending folder #5060

Open mnaamani opened 9 months ago

mnaamani commented 9 months ago

Currently on request for an object that is still being processed in the pending folder, we return a 404. We were concerned with what would happen if file was being streamed to the response, while the AcceptPendingObjects service tries to move/copy the file from pending folder to the uploads folder.

If we find that it is safe and we are okay with the resulting behaviour, and the potential latency between uploading a new object and being able to fetch it from the same node is not acceptable, then implement this change.

kdembler commented 9 months ago

If an operator runs out of balance, they could build up a stack of uploaded objects that don't get accepted. If a node like that disappears for some reason, currently all those objects would get lost since other operators won't sync until object is moved to main uploads/.

kdembler commented 9 months ago

This is possibly blocked by the fact that storage nodes will currently not synch objects that haven't been accepted on-chain