jasondellaluce
commented
1 year ago /milestone 0.11.0
/assign
I think this is a crucial step we need to face in order to achieve adequate API stability and code quality inside libsinsp, which is one of the biggest stretch goals we have as a community for this year. I volunteer to personally work on this as I conducted plenty of research in the past and have some ideas I think it would be worth proposing. This is the right place for having conversation about the topic. I've set a milestone for traceability.
FedeDP
Description
In the tech stack of Falco and its libraries, the whole architecture is event-driven and mostly functionally stateless, with the only exception being libsinsp. Among the many responsibilities and features of libsinsp, the most important is for certain state management. Libsinsp consumes every event collected by drivers and plugins and maintains an internal eventually-consistent cache representing the machine's state, which includes all the threads, file descriptors, containers, infrastructure info, etc.
Whoever worked on the libs codebase for a good amount of time is likely aware that the way state management is currently implemented is sticky, strongly coupled, and quite fragile. This is one of the biggest parts of technical debt that the libraries suffer from, it prevents us from cleaning up the API interfaces for all the consumers, and makes it increasingly harder to introduce new features.
Feature
This issue aims to track the effort of introducing a standard and long sustainable way for defining stateful components inside libsinsp and to make them accessible from other actors. Nonetheless, the highest priority is to not introduce performance regressions in any of the hot paths in which the current stateful components are involved. Additionally, we should avoid introduce excessive breaking changes for libraries consumers (ideally, we should not introduce any breaking change at all). The end goal is to walk the first steps toward a stable API for libsinsp itself, as the state management components are the biggest blockers for that.
Implications for the plugin API
The core purpose of the plugin API is to allow the libraries to be extended. Given the domain complexity that state management inevitably introduces, the effort of bringing stateful capabilities in the plugin system has been delayed since it launch. As such, the goal that I foresee for this initiative is to introduce a solution that would functionally work for both libsinsp and the plugin API, with adequate levels of simplifications.
For more context, this has been requested and discussed multiple times over the past year. Relevant links are: