Open evankanderson opened 2 months ago
(In fact, I don't see where we ever clean up entries in the ingestCache...)
Okay, it looks like the ingestCache is created per-EntityEvent in evalEntityEvent
: https://github.com/stacklok/minder/blob/main/internal/engine/executor.go#L193, and presumably only lasts for the duration of handling the event.
Unfortunately, there isn't a good hook in that method or in ingestCache
for calling "destructors" on the cached items, so we'd need to plumb that in as well.
I also thought sometime back what'll happen if someone tries to ddos the server by registering a lot of repos that are large in size (e.g. k8s clones). Most likely it will timeout as we have max limit for processing each request, but still multiple timeouts one after the other 🤷🏼♂️
Don was poking around and discovered that some Git repos may be 900MB just to shallow clone (think: they check in a copy of the JVM).
This isn't a reasonable amount to store in memory, but it's totally reasonable to put into tmpfs using a Kubernetes
EmptyDir
volume.Unfortunately, we don't have a really great way to clean these up afterwards at the moment --
Git.Clone()
in https://github.com/stacklok/minder/blob/main/internal/providers/git/git.go#L71 builds thememfs.New()
and returns it as part of the*git.Repository
, which is called byGit.Ingest()
in https://github.com/stacklok/minder/blob/main/internal/engine/ingester/git/git.go#L89, which packages the repository up into anengine/interfaces.Result
and possibly stores it in the RuleTypeEngine'sr.ingestCache
inRuleTypeEngine.Eval()
here: https://github.com/stacklok/minder/blob/main/internal/engine/rule_type_engine.go#L160