search

prowler-cloud / prowler

Prowler is an Open Source Security tool for AWS, Azure, GCP and Kubernetes to do security assessments, audits, incident response, compliance, continuous monitoring, hardening and forensics readiness. Includes CIS, NIST 800, NIST CSF, CISA, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, Well-Architected Security, ENS and more
https://prowler.com
Apache License 2.0
10.81k stars 1.54k forks source link

CloudQuery Collaboration? #1881

Closed yevgenypats closed 1 year ago

yevgenypats commented 1 year ago

Hi folks!

I've been following this project closely as this is prob the best suite of open source checks for AWS and Azure :)

I had an interesting idea for collaboration:

1) For the data extraction phase use CloudQuery plugins - https://www.cloudquery.io/docs/plugins/sources/overview 2) For the policies use SQL and we will be happy to help here and transfer the current policies that we have to this repo. 2.1 ) The current that we have are for AWS, Azure, GCP and K8S

This can be a win win as this will give us a way to focus on the data extraction phase as we are mostly a data company and for you to focus solely on security rules and logic and expand to other cloud providers and checks (without writing the ingestion part).

Will be happy also to brainstorm more here or on our discord or via email yp at cloudquery.io

toniblyx commented 1 year ago

Hello @yevgenypats! Thanks for your kind words and suggestions. I respond to your ideas below:

  1. For example, for AWS, we do data extraction using boto3, it is not SQL but we do that for each AWS service pretty easy and, at the end of the day, we get what we need to show results in multiple formats (to query again as well).
  2. In terms of 'policies' or 'checks' we do that too using the gathered information (in memory) and we get the results with checks. That is fast enough for us. With results we give users text results, csv, json, adff for security hub, html and more formats coming.

In Prowler v3 we have a new way to implement security frameworks that make them more accurate and better to get comprehensive reports. In the nutshell, we have separated infra checks from compliance requirements. Unlike most of the tools out there, we don't map 1:1 a check with compliance requirement but multiple checks with them when needed. That is a game changer in v3 in my opinion (and also the speed of scan from the unique source of truth that is the CSP API).

Also we do non CSP API checks, like secrets scanning in multiple places of the infra, Shodan integration and more to come. I don't know if that is even possible with SQL queries.

With that said, what other collaboration would work?

yevgenypats commented 1 year ago

@toniblyx Thanks for the detailed response!

Yeah I know you use boto right now, but basically the suggestion is to save you from developing all this code here: https://github.com/prowler-cloud/prowler/tree/master/prowler (which is a lot to develop and maintain).

If you use SQL for the queries then you can output in any format out of the box as depending on the destination that you use, most destinations like sqlite or postgres support export to csv, json and anything else.

I know it's a big architecture change but thought maybe it will be interesting. Totally understand if not as it might too big.

Re other collaboration, I think that's the only one that comes to mind as of this moment :)

toniblyx commented 1 year ago

Considering the important milestone we have achieved recently with v3, another foundational change is probably not appropriate at this point. We will think about other ways to integrate if possible. Thanks for your comments.