search

banyancomputer / banyan-cli

Banyan's client, BanyanFS+CAR encrypted filesystem packer and metadata handler.
https://banyan.computer/
MIT License
25 stars 2 forks source link

feat: locations request #258

Closed amiller68 closed 1 year ago

amiller68 commented 1 year ago

Description

Adds block location request and test. Implementation in core services PRd in https://github.com/banyancomputer/banyan-core/pull/132

linear[bot] commented 1 year ago
ENG-378 Add Block Tracking to Core Service

We're getting ready to have multiple potential storage provider hosting any individual blocks. Tomb clients are going to need to be able to locate where specific blocks are stored. No client changes should be required for this. When the staging service receives an upload, we're currently indexing all blocks and storing them in the database. Once the upload is complete we send a status message that just includes the size of the upload. We need to upload the list of blocks from the staging service to the meta data service as part of that status message. On the metadata server side we need to update the request to receive the block location and associate with both the specific metadata that it was uploaded for, as well as which storage provider reported in with the block report. Be careful with uniqueness constraints as we need to support blocks living at multiple storage providers and the same block CID existing for multiple metadata instances. This should also expose a block location API for tomb-like clients that allows querying for a list of block IDs, returning a mapping between storage host's and which relevant blocks are stored there. This should also return a list of blocks we don't know about or where they are stored so clients can perform extra handling. This API should require authentication.