search

CloudSecurityAlliance / gsd-tools

Global Security Database Tools
https://gsd.id
Apache License 2.0
41 stars 20 forks source link

Add importer for ruby-advisory-db #200

Closed joshbuker closed 1 year ago

joshbuker commented 1 year ago

Opening this up for review / discussion while I put on the finishing touches, but this is functional (sans affected range events, which I couldn't figure out how to translate from the ruby-advisory-db format).

This imports ruby-advisory-db into their respective namespace, and if the osvSchema is undefined, will populate it with the converted values from ruby-advisory-db. Eventually, this should incrementally update the osvSchema, if it's partially pre-defined.

Can see the first run of the script at: https://github.com/cloudsecurityalliance/gsd-database/pull/2471

postmodern commented 1 year ago

Note that most of the advisories in ruby-advisory-db are either manually added once they have a CVE or are synced from GHSA using a script. If you're already importing advisories from NVD or GHSA, you already have all of the data which ruby-advisory-db has.

kurtseifried commented 1 year ago

One note: because we have namespaces we can easily import multiple instances of the data, some advantages are: different formats that existing tools can use, and a better sense of what is covered/is missing (e.g. this CVE about ruby? does ruby know? Yes, their data is in there), which will potentially help with coordination and notifications.

joshbuker commented 1 year ago

@postmodern If you'd like me to extract the OSV converter for more generic usage, please let me know (or feel free to snag it yourself). It doesn't translate the affected versions, though, just due to difficulty mentally mapping the two.

jasnow commented 1 year ago

Increases MAX_FILES_PER_COMMIT over 609/try 1000 (find ruby-advisory-db/gems -name "CVE*" |wc) imports more.

joshbuker commented 1 year ago

@jasnow

Increases MAX_FILES_PER_COMMIT over 609/try 1000 (find ruby-advisory-db/gems -name "CVE*" |wc) imports more.

I'll just remove the limiter entirely, it was mostly for testing purposes / avoiding infinite loops, and technically it's limiting by the rubysec count not the actually modified files count right now.

If it does get added back in, should be based on the modified files count instead.

jasnow commented 1 year ago

One of the tricks I used to debug today's issue was to enhance the "Invalid" error message and then do analysis on the log (2>&1 |tee) file.