search

aboutcode-org / vulnerablecode

A free and open vulnerabilities database and the packages they impact. And the tools to aggregate and correlate these vulnerabilities. Sponsored by NLnet https://nlnet.nl/project/vulnerabilitydatabase/ for https://www.aboutcode.org/ Chat at https://gitter.im/aboutcode-org/vulnerablecode Docs at https://vulnerablecode.readthedocs.org/
https://public.vulnerablecode.io
Apache License 2.0
520 stars 190 forks source link

Process unstructured data sources #251

Open pombredanne opened 4 years ago

pombredanne commented 4 years ago

These contain valuable data nuggets among an ocean of junk and we need to be able to find the good things there.

Some sources are:

We can either automate it all, but that's going to be super difficult, or rather start to craft a curation queue and parse as much as we can to make it easy to curate by humans

... and progressively also improve some mini AI and classification to help further automate the work.

AyanSinhaMahapatra commented 1 year ago

A reference: https://hal.science/hal-03430826/document

ThePhilosopher4097 commented 1 year ago

Interested in the Project Idea...

ThePhilosopher4097 commented 1 year ago

Interested in the Project Idea... I think, processing of changelogs, reflogs of commits and mailing list data can be a automated

TG1999 commented 7 months ago

Please also check: https://github.com/cve-search/git-vuln-finder

TG1999 commented 7 months ago

https://github.com/pyupio/changelogs

ykodwani01 commented 6 months ago

I guess the process of change logs of Apache mailing list can be automated using OpenAI' API or other open source LLMs, where we scrape the data using Selenium, feed into LLM, get the output as json format and then update the database accordingly. What is your view on that @pombredanne . #218 Can also be implemented.

Suraj209211 commented 6 months ago

Automating the extraction of valuable information from Apache mailing list changelogs using OpenAI’s API and other tools is a great initiative and I think for the unstructured data we can focus primarily onto the Dataset for feature Engineering and classified into diverse group

Model Training: Fine-tune the selected model on a prepared dataset of CVEs in code. This will help the model learn to identify vulnerabilities in the unstructured data..... As well as we can use LoRA for the model to train

Vulnerability Detection: Use the trained model to parse through the unstructured data and identify potential vulnerabilities. This could involve using NLP techniques to understand the vulnerability descriptions and infer the vulnerable package name and versions.

Most Important Parameter to be checked is this Text Classification: This involves categorizing text into predefined groups. [In vulnerability detection, this could be used to classify descriptions as either indicating a vulnerability or not

Information Extraction: This is the process of automatically extracting structured information from unstructured text data.

@pombredanne @AyanSinhaMahapatra