Data flow diagram Pipeline flow diagram
More information regarding harvesting can be found here
More information regarding distribution can be found here
kubectl apply -f charts/harvest-output-pvc. This is the PVC where all the harvesters will pipe their output to and where the latest_provider and distributors will read from.
Manually create the following folders on the PVC:
mkdir -p
Based on a sources.yaml file, the harvest.py script will generate cronjobs that will pipe their output to a persistent volume.
The latest_provider service will upon a GET request to /apis expose the sum of all the .json files in the mentioned persistent volume claim.
The harvester frontend does a GET /apis to the latest_provider service and presents the result.