search

application-research / outercore-eng-kb

Official Knowledge base repo of Estuary
https://estuary.tech
5 stars 0 forks source link

Edge nodes for Upload and Retrieval #22

Closed alvin-reyes closed 1 year ago

alvin-reyes commented 1 year ago

Proposal: Edge Upload and Retrieve

Contributors @alvin-reyes
Status Draft
Revision  

The idea is to create node instance that does the following:

This will allow us to redirect uploads to different servers instead of putting them directly on the shuttles.

Development HL guide:

With whypfs-core, we can build several microservices that we see fit to scale estuary.

image

alvin-reyes commented 1 year ago

Bargev2 can be extended to have this.

alvin-reyes commented 1 year ago

if we end up compiling whypfs-core to WASM (https://github.com/application-research/whypfs-core/issues/1), then we can put these edge upload nodes on browser extensions and *theoretically on mobile phones (performance might not be the best though).

snissn commented 1 year ago

How similar / different is this to seaweedfs + whypfs?

On Sat, Jan 7, 2023 at 6:29 AM Alvin Reyes @.***> wrote:

Proposal: Edge Upload and Retrieve Contributors @alvin-reyes https://github.com/alvin-reyes Status Draft Revision

The idea is to create node instance that does the following:

  • node to accept uploads and queue them to estuary for pinning
  • node to retrieve CIDs and serve it as a gateway

This will allow us to redirect uploads to different servers instead of putting them directly on the shuttles. Development HL guide:

With whypfs-core, we can build several microservices that we see fit to scale estuary.

  • We can use whypfs-core to create the node and introduce upload endpoints and pass the CIDs to the estuary api node. Estuary will take care of pulling the CID from the upload nodes.
  • We can use whypfs-core and create a gateway in top of to serve the CIDs from estuary or any peers. This is similar to whypfs-gateway.
  • These nodes needs to be peered for discoverability.
  • This is a single node with "mode" parameter. Either upload and retrieve, upload only or retrieve only.

[image: image] https://user-images.githubusercontent.com/4479171/211160727-594c6866-a98e-4e32-b7d1-e690d5bf37c2.png

  • will add more details later.

— Reply to this email directly, view it on GitHub https://github.com/application-research/outercore-eng-kb/issues/22, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPDYYM3DKQPGOEMTSAOTUDWRGKU7ANCNFSM6AAAAAATUBBJTA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

alvin-reyes commented 1 year ago

I have yet to check whypfs+seaweedfs. Do we have docs for it? If it covers this then we can merge the idea there OR discard this. The intention is to decouple the http upload process from the api nodes by using small edge nodes that can be scaled on it’s own.

alvin-reyes commented 1 year ago

This is now complete https://github.com/application-research/edge-ur/edit/main/README.md