feat: compute GCP resources parallelly

[peb-peb]

Description

• Parallelized GCP resource computation using the ProcessPoolExecutor().
• Improves tool runtime from ~160s to ~100s.
• Moved functions to utils.py file

TODOs

• [ ] identify the crawlers which take the most amount of time.
• [ ] identify any room for improvement in performance.
• [x] refactor the code and maintain standards
• [ ] add appropriate documentation for above changes

@mshudrak @ZetaTwo

[peb-peb]

I've drawn a draft plan for how the completed refactoring for scanner loop would look like. This allows us to use python's context managers for handling files. It also allows us to separate them into individual blocks of code.

What are your opinions on this @mshudrak @ZetaTwo ?

class GCPScanner:
  """
  TODO: description for GCP Scanner Class
  """

  def __init__(self) -> None:
    pass

  def crawl_loop(self):
    pass

  def is_set(self):
    pass

  def save_results(self):
    pass

  def iam_client_for_credentials(self, credentials):
    pass

  def compute_client_for_credentials(self, credentials):
    pass

  def gke_client_for_credentials(self, credentials):
    pass

  # could be separated as a separate class
  # Since, it is a complete set of different Operations
  def get_sas_for_impersonation(self, iam_policy):
    pass

  # separate this from main loop
  # Since, this need to be called prior to crawling
  def get_project_list(self):
    pass

  # context managers
  def __enter__(self):
    pass

  def __exit__(self):
    pass

[peb-peb]

Comment on -p flag. PTAL.

Working on it.

[mshudrak]

@peb-peb are you sure this step is absolutely necessary right now? Do you think it need to be implemented before we go for parallelization?

[peb-peb]

Nope. This refactorization could be done once we are done with the parallelization.

So, should we first complete the parallelization, then proceed with this?

[mxmssh]

Yes, I think it makes sense to work on parallelization first

On Sun, Jul 16, 2023, 3:56 PM Harsh @.***> wrote:

Nope. This refactorization could be done once we are done with the parallelization.

So, should we first complete the parallelization, then proceed with this?

— Reply to this email directly, view it on GitHub https://github.com/google/gcp_scanner/pull/245#issuecomment-1637209521, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUFDYXJNZSQHZR77H2GWQDXQRWQLANCNFSM6AAAAAA2K47OB4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[peb-peb]

@mshudrak Added the suggested changes. PTAL.

usage: python3 scanner.py -o <output_directory> -k <sa_keys_directory> -l INFO --cpu-count <any number> Example: python3 scanner.py -o outputs/ -k sa_keys/ -l INFO --cpu-count 4

[peb-peb]

Also, while deciding upon the flag for CLI, the -p flag is already used for --project, so that couldn't be used. So, instead of --parallel, I decided to go with --cpu-count or -cc.

Any other suggestions are also welcome! :) @sudiptob2 @mshudrak @ZetaTwo

[peb-peb]

Closing as this PR would continue it's discussion in #265.