search

trustification / trustify

Apache License 2.0
8 stars 15 forks source link

Bad CSAF ingestion performance #456

Open ctron opened 1 week ago

ctron commented 1 week ago

I noticed a massive drop in performance when running the CSAF import process. I don't have numbers yet, but I'll investigate.

In the SQL logs I noticed at of a "select then insert" patters again.

ctron commented 1 week ago

I tried to go back a bit in the git history to see if that's a recent change. Doesn't look like that's the case.

I started to work with the test data file cve-2023-33201.json. Which takes around 470ms to ingest. It looks like the majority of time goes into ingesting the vulnerabilities. Parsing, inserting, and committing the main entry takes around 70ms.

ctron commented 1 week ago

The majority of time (~390ms) goes into ingest_product_statuses.

ctron commented 1 week ago

I changed the title as I cannot provide any numbers. I just have the feeling it got much slower since some time ago.