Create testing environment

[mdo6180]

• [ ] Build GraphService
  • [ ] Rename Pipeline class to Graph
  • [ ] Rename Webserver class to GraphServer
    • GraphServer handles all web traffic for a Graph. This includes:
    • Handling all endpoints for each node managed by the Graph.
    • Each Graph is assigned a "GraphID"
    • In FastAPI terminology, the GraphServer is the "main app" that sub apps get mounted into (each node has its own sub-app).
  • GraphService is bound to a specific ip address and port
  • GraphServer uses the Graph class to spin up a copy of the subgraph.
  • GraphService spins up an instance of GraphServer and assigns an endpoint prefix to the GraphServer.
  • The instance of the GraphServer is then mounted into GraphService.
  • There are two types of GraphService:
    • RootGraphService serves a graph that has a metadata store.
    • LeafGraphService serves a graph that does not have a metadata store.

Test outline:

1. Use shell script to call docker compose and call the python file that creates dummy files for testing and puts the dummy files into the repositories mounted into the container running RootGraphService
2. Docker compose spins up two containers
   • One container spins up a RootGraphService and the other container spins up a LeafGraphService.
   • RootGraphService contains metadata store node, monitoring resource node, and action node
   • LeafGraphService contains an action node.
3. Test creates a few files, puts those files into the monitoring resource node, and checks to see if the action node on the RootGraphService and the LeafGraphService are both triggered.

Testing logging configurations

• [x] Configure logger for both containers to log to the same log file.
  • [x] Enable GraphService to accept loggers
• [x] Log file lives inside testing folder.
• [x] Create two subfolders inside the testing folder to hold data from each GraphService.
• [x] Enable containers to use pip install -e command to install Anacostia inside the container.
• [x] Mount folder to each container's filesystem.
• [x] Mount a folder to each container's filesystem for input artifacts (folder is called input_artifacts).
  • This is where data will be deposited to be picked up by the monitoring resource node.
• [x] Mount a folder to each container's filesystem for output artifacts (folder is called output_artifacts).
  • This is where newly created pipeline artifacts will be deposited.

[mdo6180]

• [ ] Create healthcheck in docker compose to know when root and leaf graph services are up and running (see 1 and 2)
  • [ ] Install curl or wget into the container first. (python:3.12.4-slim don't come with curl preinstalled)

[mdo6180]

Created healthcheck by implementing healthcheck as python file. Python file call os._exit(status=0) to ensure script exits with status 0. Docker compose looks for exit status 0 from a shell command as passing condition for healthcheck.

[mdo6180]

i'm going to stop using docker and docker compose because docker consumes too much memory (i literally ran out of ram to build the containers).

thus, i have decided to make the switch to bash scripts to set up the tests.

[mdo6180]

the following code snippet is an example of how to assign IP addresses and ports to FastAPI servers using bash.

#!/bin/bash

# Define IP addresses and ports
IP1="192.168.100.1"
IP2="192.168.100.2"
PORT1="8001"
PORT2="8002"

# Create network namespaces
sudo ip netns add ns1
sudo ip netns add ns2

# Create veth pairs
sudo ip link add veth0 type veth peer name veth1
sudo ip link add veth2 type veth peer name veth3

# Assign one end of each veth pair to the respective namespace
sudo ip link set veth1 netns ns1
sudo ip link set veth3 netns ns2

# Assign IP addresses to the veth interfaces
sudo ip addr add $IP1/24 dev veth0
sudo ip netns exec ns1 ip addr add $IP1/24 dev veth1
sudo ip addr add $IP2/24 dev veth2
sudo ip netns exec ns2 ip addr add $IP2/24 dev veth3

# Bring up the interfaces
sudo ip link set veth0 up
sudo ip link set veth2 up
sudo ip netns exec ns1 ip link set veth1 up
sudo ip netns exec ns2 ip link set veth3 up

# Start FastAPI servers
# Run app1 in the first namespace
sudo ip netns exec ns1 uvicorn app1:app --host $IP1 --port $PORT1 &

# Run app2 in the second namespace
sudo ip netns exec ns2 uvicorn app2:app --host $IP2 --port $PORT2 &

# Output the completion message
echo "FastAPI servers are running at:"
echo "http://$IP1:$PORT1 (in namespace ns1)"
echo "http://$IP2:$PORT2 (in namespace ns2)"

[mdo6180]

• [ ] Create healthcheck in docker compose to know when root and leaf graph services are up and running (see 1 and 2)

  • [ ] Install curl or wget into the container first. (python:3.12.4-slim don't come with curl preinstalled)

no longer pursuing docker approach.