Error while running GCP scan for resources not used by the project

[sidyatish]

Description Getting multiple errors while running scoutsuite scan for GCP. The errors are for the resources that we have not enabled by us in GCP. Command : scout gcp -s <service_account_path.json>

Error format

ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including KMS in the execution
ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including Stackdriver Logging in the execution
ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including Kubernetes Engine in the execution
ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including DNS in the execution
ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including Cloud SQL in the execution
ERROR base.py L152: Could not fetch the state of services for project <PROJECT_NAME>, including MEMORYSTORE in the execution

Debug logs Getting similar error for all the above mentioned resources

ERROR base.py L152: Could not fetch the state of services for project <PROJECT>, including DNS in the execution
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/ScoutSuite/providers/gcp/facade/base.py", line 152, in is_api_enabled
    services_response = await GCPFacadeUtils.get_all('services', request, services)
  File "/usr/local/lib/python3.9/site-packages/ScoutSuite/providers/gcp/facade/utils.py", line 17, in get_all
    await GCPFacadeUtils._get_all(resources, resource_key, request, resources_group)
  File "/usr/local/lib/python3.9/site-packages/ScoutSuite/providers/gcp/facade/utils.py", line 8, in _get_all
    response = request.execute()
  File "/usr/local/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 429 when requesting https://serviceusage.googleapis.com/v1/projects/<PROJECT_NAME>/services?alt=json&pageToken=<TOKEN> returned "The metric quota for serviceusage.googleapis.com/list_available_requests is exhausted.
Help Token: <SOME_TOKEN>". Details: "[{'@type': 'type.googleapis.com/google.rpc.PreconditionFailure', 'violations': [{'type': 'googleapis.com', 'subject': '?error_code=900021&metric=serviceusage.googleapis.com/list_available_requests'}]}, {'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'API_SHARED_QUOTA_EXHAUSTED', 'domain': 'serviceusage.googleapis.com', 'metadata': {'metric': 'serviceusage.googleapis.com/list_available_requests'}}]">

But we checked our quotas are still available and this is happening only with scout suite.

[ckuethe]

Also seeing this. I've turned down my scan intensity by using --max-workers 2 --max-rate 1, excluding services that don't exist in my project, and sleeping a couple of minutes between scanning each project.

[chargraves85]

I cannot find any documentation showing how to skip services that are not existing or enabled in project. Please let us know how you did that @ckuethe. Thank you.

[ckuethe]

You can use scout gcp --list-services to list the service types scout knows about, and something like scout gcp --skip bigquery ... to skip service checks.