syalioune
commented
2 years ago Thanks for the detailed design. Few remarks below :
Every time a component is created or updated in the UI via the componentResource or a new BOM is uploaded resulting in a BOMUploadProcessingTask, the proposal is to produce components as events on a Kafka topic (maybe the same topic being used for vulnerabilityAnalysis?)
I would recommend using the same topic for two reasons :
- The event model is the same. It does not make sense to have the exact same event in different topics. The whole benefit of Kafka is to be able to plug different consumer to process the same event 😉
- All of the conditions that trigger the RepositoryMetaAnalyzer are the same that trigger Vuln Analyzer. It would make sense for the API server to publish only one component event for all analyzers.
For each analyser type, we should get the supported repository and then get all the repositories corresponding to the supported type into a Global KTable to be used for this analysis. This KTable would use the repositoryType as the key and the list of supported repositories as the value. A repository might also need a username and password. These are expected to be in the environment variables at the start of the application.
I believe a database will be more fit for this requirement (storing repository configuration and sensitive data). The data ownership is blurry as the repositories configuration is stored in both API Server and Quarkus application whereas it should be the responsibility of the application containing the analyzer. Moreover, by using environment variables for credentials, you prevent them to be updated by the frontend.
Offloading API Server Database does not necessarily imply that other modules should not have their own databases and expose APIs for API server to consume.
We can have something like in the picture below using an appropriate DB for each module, with clear responsibilities and separation of concerns. Analyzer module (current Quarkus app) can have its database and exposing a subset of API Server rest api allowing update and fetching from Frontend.
If all DB choices end up being SQL, this allow flexibility as :
- A module can never need a DB
- All modules can target the same DB and schema
- Modules can target the same DB and different schemas (one for each module or one per use-case [write-heavy vs read-only])
- Modules can target different DBs altogether (with different sizing)
To be discussed :smiley:
nscuro
mehab
Current Implementation
Currently, the Dependency Track API server has a RepositoryMetaAnalyzer class that analyses a given component based on its purl type to get a meta model of the given component. If the meta model has a latestVersion field that is not null then a metaComponent is created with the latest information from the model and then synchronised with the database. This RepositoryMetaAnalyzer is actively listening for RepositoryMetaEvent.
RepositoryMetaEvent : This is an event type in DT API server that can be triggered from 3 different places:
In the RepositoryMetaAnalyzer, there are two workflows:
Drawbacks of this approach While the portfolio analysis can be seen as a really heavy task from the description above itself that is consuming a lot of resources of the API server. The events containing individual components would also lead to making DT API server resources busy, in the sense that, consider a large SBOM containing 10000 components uploaded resulting in an event for the BOMUploadProcessingTask. Then a repositoryMetaEvent is triggered for every component in the SBOM.
Proposal
We move out the RepositoryMetaAnalyzer to the external Quarkus application that leverages Kafka and we have used it already to offload the VulnerabilityAnalysis from DT Api server. This is the initial proposed design.
Analysing each component
Every time a component is created or updated in the UI via the componentResource or a new BOM is uploaded resulting in a BOMUploadProcessingTask, the proposal is to produce components as events on a Kafka topic (maybe the same topic being used for vulnerabilityAnalysis?). If component does not have a purl associated with it, then we do not need to send this on the kafka topic since in the current implementation, if the model retrieved after analysis has the latestVersion field as null (this would be the case when the component does not have a purl) then the meta analysis has no impact for this.
For each analyser type, we should get the supported repository and then get all the repositories corresponding to the supported type into a Global KTable to be used for this analysis. This KTable would use the repositoryType as the key and the list of supported repositories as the value. A repository might also need a username and password. These are expected to be in the environment variables at the start of the application.
These events contain the component UUID and purl at a minimum. The incoming events can be mapped to different meta analyser consumers by a primary meta analyser consumer based on the type of Purl. Once the analysis is done by the corresponding meta analyser implementation like Maven or Golang, then the fields of a metaComponent object are updated with the meta Model. This marks the completion of the work by the meta analysers. The output containing the metaComponent can be sent back to the DT API server to save it to the database.
RepositoryMetaEvent from Task Scheduler When the repository meta event is triggered by the task scheduler, we are essentially doing a portfolio analysis. Get all the components of all the active projects in DT API server and then each of these components can be sent to the same event as above and be analysed.