search

aboutcode-org / vulnerablecode

A free and open vulnerabilities database and the packages they impact. And the tools to aggregate and correlate these vulnerabilities. Sponsored by NLnet https://nlnet.nl/project/vulnerabilitydatabase/ for https://www.aboutcode.org/ Chat at https://gitter.im/aboutcode-org/vulnerablecode Docs at https://vulnerablecode.readthedocs.org/
https://public.vulnerablecode.io
Apache License 2.0
543 stars 201 forks source link

added openjdk-importer to pipeline #1635

Open harmonicfunc opened 3 weeks ago

harmonicfunc commented 3 weeks ago

1496

revision to previous openjdk-importer with new pr

harmonicfunc commented 3 weeks ago

@keshav-space @pombredanne @ambuj-1211 pls review this pr

Rishi-source commented 3 weeks ago

Hi @harmonicfunc , When you add a importer pipeline you have to certainly make test_importer with some test data in the test directory. you can read the tutorial_add_importer_pipeline in docs/source directory. As well as you can also read the contributing guidelines https://aboutcode.readthedocs.io/en/latest/contributing.html . You will also find the slack channel link there which you can join.

harmonicfunc commented 3 weeks ago

the logs for importer run are here: (venv) (base) alok@alok-IdeaPad-5-15ITL05:~/GSOC/vulnerablecode$ ./manage.py import openjdk_importer Importing data using openjdk_importer INFO 2024-11-04 10:36:13.527 Pipeline [OpenJDKImporterPipeline] starting INFO 2024-11-04 10:36:13.527 Step [collect_and_store_advisories] starting INFO 2024-11-04 10:36:24.174 Collecting 23 advisories INFO 2024-11-04 10:40:27.413 Progress: 13% (3/23) ETA: 1628 seconds (27.1 minutes) INFO 2024-11-04 10:40:27.415 Progress: 26% (6/23) ETA: 692 seconds (11.5 minutes) INFO 2024-11-04 10:40:27.434 Progress: 39% (9/23) ETA: 380 seconds (6.3 minutes) INFO 2024-11-04 10:40:27.445 Progress: 52% (12/23) ETA: 225 seconds (3.8 minutes) INFO 2024-11-04 10:40:27.471 Progress: 65% (15/23) ETA: 131 seconds (2.2 minutes) INFO 2024-11-04 10:40:27.482 Progress: 78% (18/23) ETA: 69 seconds (1.1 minutes) INFO 2024-11-04 10:40:27.492 Progress: 91% (21/23) ETA: 24 seconds INFO 2024-11-04 10:40:27.517 Progress: 104% (24/23) ETA: -9 seconds INFO 2024-11-04 10:40:27.529 Progress: 117% (27/23) ETA: -35 seconds INFO 2024-11-04 10:40:27.541 Progress: 130% (30/23) ETA: -56 seconds INFO 2024-11-04 10:40:27.558 Progress: 143% (33/23) ETA: -73 seconds INFO 2024-11-04 10:40:27.578 Progress: 156% (36/23) ETA: -87 seconds INFO 2024-11-04 10:40:27.636 Progress: 169% (39/23) ETA: -99 seconds INFO 2024-11-04 10:40:27.653 Progress: 182% (42/23) ETA: -110 seconds INFO 2024-11-04 10:40:27.682 Progress: 195% (45/23) ETA: -119 seconds INFO 2024-11-04 10:40:27.693 Progress: 208% (48/23) ETA: -126 seconds INFO 2024-11-04 10:40:27.709 Progress: 221% (51/23) ETA: -133 seconds INFO 2024-11-04 10:40:27.727 Progress: 234% (54/23) ETA: -139 seconds INFO 2024-11-04 10:40:27.740 Progress: 247% (57/23) ETA: -145 seconds INFO 2024-11-04 10:40:27.757 Progress: 260% (60/23) ETA: -150 seconds INFO 2024-11-04 10:40:27.769 Progress: 273% (63/23) ETA: -154 seconds INFO 2024-11-04 10:40:27.790 Progress: 286% (66/23) ETA: -158 seconds INFO 2024-11-04 10:40:27.802 Progress: 300% (69/23) ETA: -162 seconds INFO 2024-11-04 10:40:27.864 Progress: 313% (72/23) ETA: -166 seconds INFO 2024-11-04 10:40:27.873 Progress: 326% (75/23) ETA: -169 seconds INFO 2024-11-04 10:40:27.883 Progress: 339% (78/23) ETA: -172 seconds INFO 2024-11-04 10:40:27.894 Progress: 352% (81/23) ETA: -174 seconds INFO 2024-11-04 10:40:27.905 Progress: 365% (84/23) ETA: -177 seconds INFO 2024-11-04 10:40:27.927 Progress: 378% (87/23) ETA: -179 seconds INFO 2024-11-04 10:40:27.939 Progress: 391% (90/23) ETA: -181 seconds INFO 2024-11-04 10:40:27.950 Progress: 404% (93/23) ETA: -183 seconds INFO 2024-11-04 10:40:27.959 Progress: 417% (96/23) ETA: -185 seconds INFO 2024-11-04 10:40:27.975 Progress: 430% (99/23) ETA: -187 seconds INFO 2024-11-04 10:40:27.989 Progress: 443% (102/23) ETA: -189 seconds INFO 2024-11-04 10:40:28.014 Progress: 456% (105/23) ETA: -190 seconds INFO 2024-11-04 10:40:28.032 Progress: 469% (108/23) ETA: -192 seconds INFO 2024-11-04 10:40:28.045 Progress: 482% (111/23) ETA: -193 seconds INFO 2024-11-04 10:40:28.071 Progress: 495% (114/23) ETA: -195 seconds INFO 2024-11-04 10:40:28.085 Progress: 508% (117/23) ETA: -196 seconds INFO 2024-11-04 10:40:28.098 Progress: 521% (120/23) ETA: -197 seconds INFO 2024-11-04 10:40:28.109 Progress: 534% (123/23) ETA: -198 seconds INFO 2024-11-04 10:40:28.134 Progress: 547% (126/23) ETA: -199 seconds INFO 2024-11-04 10:40:28.144 Progress: 560% (129/23) ETA: -200 seconds INFO 2024-11-04 10:40:28.156 Progress: 573% (132/23) ETA: -201 seconds INFO 2024-11-04 10:40:28.166 Progress: 586% (135/23) ETA: -202 seconds INFO 2024-11-04 10:40:28.176 Progress: 600% (138/23) ETA: -203 seconds INFO 2024-11-04 10:40:28.196 Progress: 613% (141/23) ETA: -204 seconds INFO 2024-11-04 10:40:28.207 Progress: 626% (144/23) ETA: -205 seconds INFO 2024-11-04 10:40:28.229 Progress: 639% (147/23) ETA: -206 seconds INFO 2024-11-04 10:40:28.243 Progress: 652% (150/23) ETA: -207 seconds INFO 2024-11-04 10:40:28.254 Progress: 665% (153/23) ETA: -207 seconds INFO 2024-11-04 10:40:28.264 Progress: 678% (156/23) ETA: -208 seconds INFO 2024-11-04 10:40:28.274 Progress: 691% (159/23) ETA: -209 seconds INFO 2024-11-04 10:40:28.284 Progress: 704% (162/23) ETA: -209 seconds INFO 2024-11-04 10:40:28.303 Progress: 717% (165/23) ETA: -210 seconds INFO 2024-11-04 10:40:28.316 Progress: 730% (168/23) ETA: -211 seconds INFO 2024-11-04 10:40:29.350 Progress: 743% (171/23) ETA: -212 seconds INFO 2024-11-04 10:40:29.370 Progress: 756% (174/23) ETA: -213 seconds INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds INFO 2024-11-04 10:40:29.440 Successfully collected 178 advisories INFO 2024-11-04 10:40:29.440 Step [collect_and_store_advisories] completed in 256 seconds (4.3 minutes) INFO 2024-11-04 10:40:29.440 Step [import_new_advisories] starting INFO 2024-11-04 10:40:29.442 Importing 184 new advisories INFO 2024-11-04 10:40:29.948 Progress: 10% (19/184) ETA: 5 seconds INFO 2024-11-04 10:40:30.468 Progress: 20% (37/184) ETA: 4 seconds INFO 2024-11-04 10:40:31.043 Progress: 30% (56/184) ETA: 4 seconds INFO 2024-11-04 10:40:31.621 Progress: 40% (74/184) ETA: 3 seconds INFO 2024-11-04 10:40:32.198 Progress: 50% (92/184) ETA: 3 seconds INFO 2024-11-04 10:40:32.909 Progress: 60% (111/184) ETA: 2 seconds INFO 2024-11-04 10:40:33.523 Progress: 70% (129/184) ETA: 2 seconds INFO 2024-11-04 10:40:34.087 Progress: 80% (148/184) ETA: 1 seconds INFO 2024-11-04 10:40:34.654 Progress: 90% (166/184) ETA: 1 seconds INFO 2024-11-04 10:40:35.278 Progress: 100% (184/184) INFO 2024-11-04 10:40:35.308 Successfully imported 184 new advisories INFO 2024-11-04 10:40:35.308 Step [import_new_advisories] completed in 6 seconds INFO 2024-11-04 10:40:35.308 Pipeline completed in 262 seconds (4.4 minutes)

pls review the pr!!

pombredanne commented 3 weeks ago

The progress logging looks weird:

 INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds

769%? Do you know where this is from?

harmonicfunc commented 3 weeks ago

The progress logging looks weird:

 INFO 2024-11-04 10:40:29.427 Progress: 769% (177/23) ETA: -213 seconds

769%? Do you know where this is from?

yeah thats because i took the advisory count from the main page of url:https://openjdk.org/groups/vulnerability/advisories/ , but that might be wrong