At the moment, we're using a simple limiting method to parallelise our building of Falco probes. We should migrate this to use job queues where workers pull configurations off of a queue.
An extra nice thing to think about is turning the building process into a pipeline, e.g.:
fetchSources(kernelPackageName):
push kernelPackage to falcoVersionMatrix queue
falcoVersionMatrix(kernelPackage):
push to isUploaded queue
isUploaded(falcoVersion, kernelPackage):
if not uploaded, push kernelPackage to buildProbe queue
buildProbe(falcoVersion, kernelPackage):
if built, push to uploadProbe queue
uploadProbeQueue(falcoVersion, kernelPackage, builtProbe):
upload to repository
where each of the above functions have multiple workers executing them.
At the moment, we're using a simple limiting method to parallelise our building of Falco probes. We should migrate this to use job queues where workers pull configurations off of a queue.
(see https://www.opsdash.com/blog/job-queues-in-go.html) for examples/guidance.
An extra nice thing to think about is turning the building process into a pipeline, e.g.:
where each of the above functions have multiple workers executing them.