figure out how to automatically spin up nodes in a distributed systems architecture

[mdo6180]

• [x] Create a class called Anacostia Server.

How Anacostia Servers work:

1. Server 1 spins up a copy of its subgraph.
2. Server 1 sends a request to Server 2 to spin up a copy of Server 2's subgraph.
3. Server 2 spins up a copy of its subgraph (subgraph is defined by the user) by spinning up all the nodes in the subgraph.
4. Server 2 sends an ACK to Server 1 when its subgraph is set up.
5. Server 2's subgraph starts listening for signals from Server 1's subgraph (signals are sent via RPC)
6. Server 1's subgraph starts executing.

Note: for now, assume that a DAG can only be split into two subgraphs on server 1 and server 2. however, keep in mind that in the future, we may allow for DAGs like the following:

server1, server2, and server3 all spin up one copy of a subgraph:

server1 -> server2 -> server3

server3 spins up two copies of its subgraph, one copy to serve server1, the other subgraph to serve server2; i.e., each machine runs a subgraph as a service. (subgraph as a service - SaaS)

server2 
       \
        server3
       /
server1

server1's subgraph connects to the subgraphs of server2 and server3

         server2 
        /
server1 
        \
         server3

• Each server runs a subgraph as a service.

• Node config is statically defined. I.e., server1 knows what nodes are available on server2 because user of server1 will tell it in the config.

• Nodes are globally addressable. Any node can send signals to any node across another machine.

• There are now three types of subgraphs:

  • Root subgraphs (must contain a metadata store)
  • Resource subgraphs (input must a metadata store node)
  • Action subgraphs (input must be either resource nodes or action nodes)
  • Only root subgraphs can initiate a pipeline run

• [x] #38

• [x] #31

  • [x] Create a connector node.
  • This node is used specifically to send the user information such as the IP address of the Anacostia Server hosting the non-root subgraph.
  • User procures the connector node by pip installing a python module.
  • User adds connector node to current root pipeline.
  • User's Anacostia Server is responsible for setting up CORS and other things (we will define more of these "things" later) to enable for root subgraph Anacostia Server to connect to non-root subgraph Anacostia Server.

• [x] #32

• [x] #33

  • [ ] Enable anacostia server of root subgraph to send an RPC to anacostia server of non-root subgraph.
  • [ ] Make sure that every time the non-root anacostia server spins up a copy, it sends the ID of the copy back to the root Anacostia Server as an ACK.
  • [ ] Root Anacostia Server registers the ID of the copy in its system.
  • [ ] Try using a session token to accomplish the two tasks above.
  • [ ] Remove requirement in pipeline.py that checks if a graph is valid iff it has a metadata store node.

• [ ] #34

  • [ ] Try using cross-origin resource sharing (CORS) in FastAPI to allow UI to receive html snippets from other Anacostia Servers.
  • [x] Handle naming conflicts (i.e., what happens if two nodes across two different environments have the same name?).

• [ ] #35

  • [ ] root subgraph Anacostia Server sends an RPC to non-root subgraph Anacostia Server to shut down its copy.

• [ ] #36

  • [ ] Just use http endpoints to send files from one Anacostia Server to another.

• [ ] #37

  • [ ] Since we are allowed only 6 SSE connections per tab in a web browser, figure out a way to use CORS to enable nodes in other Anacostia Servers to send messages through the same SSE endpoint.

[mdo6180]

@Veryyes