I'm writing this issue to explain why we need a database in the project (Following our recent discussions in the regular meeting concerning this rfcs: https://github.com/NodeSecure/rfcs/issues/3).
Key points
Expanding our field of research
Having our own database would allow us to drastically expand our field of research. The idea is not new and for JS-X-Ray I had started the star-analytica project which aimed to reduce false positives and provide new ideas for SAST analysis (by storing JS-X-Ray results for each tarball/package).
Having our own replica of the registry would allow us to quickly launch new analyses without having to depend on third party APIs (which as we will see are often limited).
Beyond the technical aspects, this will also allow us to learn more and complete our expertise.
Do not suffer from ratelimits anymore
Almost all API are limited in usage and this handicaps us enormously in providing complete and relevant analyses through the scanner (For example for IANA socket API is limited to 30req/s). Even the NPM API is limited without the implementation of a token (which makes our tools less accessible).
Some project like preview require a database to properly work as online tool. It is not conceivable to run scanner in loop on a hosted back-end (this would lead to untenable infrastructure costs).
Also the user experience is poor because the analysis times will be several seconds minimum.
New features
Having a database will allow us to implement new functionalities within our graphical tools such as the CLI or Preview.
An example of this are Snyk advisor charts:
We could also implement checks related to OpenSSF standards (like Open source insights).
An income opportunity
This could become in the very long term an opportunity to have financial support from users or companies in exchange for tokens to exploit the API without limit.
Hello 👋,
I'm writing this issue to explain why we need a database in the project (Following our recent discussions in the regular meeting concerning this rfcs: https://github.com/NodeSecure/rfcs/issues/3).
Key points
Expanding our field of research
Having our own database would allow us to drastically expand our field of research. The idea is not new and for JS-X-Ray I had started the star-analytica project which aimed to reduce false positives and provide new ideas for SAST analysis (by storing JS-X-Ray results for each tarball/package).
Having our own replica of the registry would allow us to quickly launch new analyses without having to depend on third party APIs (which as we will see are often limited).
Another example is the need to compare different vulnerability strategies for the Vulnera project. See https://github.com/NodeSecure/vulnera/issues/29.
We can apply this logic to many of our projects;
Beyond the technical aspects, this will also allow us to learn more and complete our expertise.
Do not suffer from ratelimits anymore
Almost all API are limited in usage and this handicaps us enormously in providing complete and relevant analyses through the scanner (For example for IANA socket API is limited to 30req/s). Even the NPM API is limited without the implementation of a token (which makes our tools less accessible).
Some project like preview require a database to properly work as online tool. It is not conceivable to run scanner in loop on a hosted back-end (this would lead to untenable infrastructure costs).
Also the user experience is poor because the analysis times will be several seconds minimum.
New features
Having a database will allow us to implement new functionalities within our graphical tools such as the CLI or Preview.
An example of this are Snyk advisor charts:
We could also implement checks related to OpenSSF standards (like Open source insights).
An income opportunity
This could become in the very long term an opportunity to have financial support from users or companies in exchange for tokens to exploit the API without limit.