terriko
commented
2 years ago I like the line of thinking here, but there's huge chasms of bad ideas that happen when you talk about building an abstraction layer. So I'm glad you opened a discussion thread for this!
Some thoughts off the top of my head:
-
We honestly don't want to be in the business of writing parsers at all if we don't have to. We already try to use external well tested libraries for xml, json, and hopefully will be doing so for sbom as more mature tools become available. I can see writing our own parsers to conform to a bit of an API, but will it be a pain if we have to maintain wrapper layers for external libraries to conform to that same API?
-
What exactly are you talking about as parsers in this context? We probably need a name that reflects what we're parsing, because "parser" could mean just about anything. Are we talking component list parsers only? Are we talking about things like rpms and gzip files and jar files as well?
-
We already have abstraction layers for checkers, input and output. You're mostly talking about stuff that falls into the input category, I think, which handles the .csv and triage files. Are those fundamentally different than parsing requirements.txt? I think probably not -- it would be nice for the user if
cve-bin-tool -i requirements.txtworked just as well ascve-bin-tool -i dependencies.csv. Should we be glomming the SBOMs in as well so it also works through-i? -
Do we think it's likely that people will want to specify multiple component lists and have them in the same report? How should that work? (right now I think if you generate each into a .csv report or somesuch you can then merge them)
XDRAGON2002
While going through #1526 I was thinking about how this could lead to support for multiple parsers, but as the current structure stands, it is a bit too random (not inefficient, just feels scattered). The parsers that are currently supported, all take a similar input (a file format that needs to be parsed) and output similarly as well (product info, triage data) that is then passed to the cve_scanner in cli.py.
The parsers act as black boxes with respect to each other and other components, but these sets of black boxes also have common input format, output format and most importantly the same thought process (implementation could differ a bit). Hence following the Object Orientated Paradigm I feel we can structure this based on overriding the class methods.
This would look something like having a parsers directory and to add a new parser to the tool, one could simply add a new file, inherit from the parent parser class, override the class with appropriate parameters and that's about it! This is very (if not completely) similar to the system currently in use for adding checkers (add file to checkers directory, inherit from parent checker class, override the data members and you're good to go).
It would lead to a general system for adding new parsers and support for new package file formats, all the while abstracting away the implementation details for the black box approach even more and also reduce complexity of code (which would allow even more people to work on these parsers, again similar to checkers), also would allow a better method to write tests for these modules.
We could also look into creating a wrapper around all these parsers after we restructure appropriately (something like a parse() function) that simply takes in the input file, and passes it to the appropriate parser depending on file properties (requirements.txt,*.json,package.json etc) this would improve abstraction further and also cleanup the codebase a bit (currently parsers are being supplied their inputs directly from cli.py).
This thread is for just an overview of the idea, would love to figure out the intricacies of this over time with proper discussion, approval and help from the community.