search

GoogleCloudPlatform / security-response-automation

Take automated actions against threats and vulnerabilities.
Apache License 2.0
208 stars 52 forks source link

getting "googleapi: got HTTP response code 404 with body: Not Found" error #204

Open deepakdimri-ce opened 3 years ago

deepakdimri-ce commented 3 years ago

i have CloseBucket Cloud Function deployed as per the instructions from the repo and its getting triggered from SCC notification as well but when its comes to action its throwing "googleapi: got HTTP response code 404 with body: Not Found" error. what could be the reason for this error?

onetwopunch commented 3 years ago

Could you please paste the error message showing which API call responded with a 404 and what resource was not found?

deepakdimri-ce commented 3 years ago

Below is series of events from Cloud Logging. This is all what i see in the logs every-time i create a test bucket with public access enabled.


  "textPayload": "Function error: googleapi: got HTTP response code 404 with body: Not Found\n",
  "insertId": "000000-5a80908c-51bb-4230-bdff-5e0b1076c5be",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165Z",
  "severity": "ERROR",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
{
  "textPayload": "Function execution took 21 ms, finished with status: 'error'",
  "insertId": "000000-48665883-aaca-4132-9f6c-05a133854b86",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket",
      "region": "us-central1"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165885367Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
https://console.cloud.google.com/logs/query;cursorTimestamp=2021-04-11T03:39:01.165Z;query=resource.type%20%3D%20%22cloud_function%22%0Aresource.labels.function_name%20%3D%20%22CloseBucket%22%0Aresource.labels.region%20%3D%20%22us-central1%22%0A%20severity%3E%3DDEFAULT%0Atimestamp%3D%222021-04-11T03:39:01.165885367Z%22%0AinsertId%3D%22000000-48665883-aaca-4132-9f6c-05a133854b86%22;timeRange=2021-04-11T03:27:11.682Z%2F2021-04-11T05:27:11.682214Z?project=contextawaredemo```
daniel-cit commented 3 years ago

@onetwopunch @deepakdimri-ce Just a detail that may help the investigation.

The minimal deployment for the security response automation is the Router cloud function + any other remediation cloud function.

Only the Router function knows how to read SCC findings.

The router function is the one that reads the finding and generates a struct with only the project name + the bucket name that is used by the CloseBucket function.

The SCC notification must send messages to the Router Cloud Function.

The error may be related to trying to get the bucket policy from an bucket name that is the empty string "" because the ClouseBucket cloud function was not able to get the bucket name from the finding, because only the Router knows how to read the finding.

deepakdimri-ce commented 3 years ago

thanks @daniel-cit - i have got Router cloud function deployed, with scc notifications going to Router function. but now i see Router function reporting "finding already remediated" message and not doing any thing. i am testing it with having a test GCS bucket wide open for public access. there are no logs showing up in CloseBucket function. any idea? i am still trying to investigate


  "textPayload": "Function execution took 1386 ms, finished with status: 'ok'",
  "insertId": "000000-1ae451f4-8cfa-4130-a84d-8bf01809a83c",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:21.119026453Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kpgx1qubq"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/7a629f3a9e0db83ad0700c3febdb85cf",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "Function execution started",
  "insertId": "000000-25733616-6443-4dfc-972d-67a02dd830f8",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-13T04:41:22.584244984Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "2021/04/13 04:41:22 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated",
  "insertId": "000000-a39d5a68-b346-41ed-bdde-3efde1e8a8f4",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "region": "us-central1",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:22.589Z",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}```
deepakdimri-ce commented 3 years ago

this is my sra.yaml file looks like replaced with dummy org and folder ids:


kind: Remediation
metadata:
  name: router
spec:
  parameters:
    sha:
      public_bucket_acl:
        - action: close_bucket
          target:
            - organizations/999999999/folders/888888888/projects/contextawaredemo
          excludes:
            - organizations/999999999/folders/88888888/projects/forresterdemo
          properties:
            dry_run: false```
daniel-cit commented 3 years ago

Hi @deepakdimri-ce , The "finding already remediated" is a known issue https://github.com/GoogleCloudPlatform/security-response-automation/issues/197

You can try to force the execution by going to the Console and deleting the security mark "sra-remediated-event-time" in the original Security Command Center finding.

Regarding the yaml file, you should use this file as an initial version https://github.com/GoogleCloudPlatform/security-response-automation/wiki/Sample-configuration.

A trick point is that It is an yaml file and preserving the indentation is importante.

deepakdimri-ce commented 3 years ago

thanks @daniel-cit for helping me with this.

I have tried deleting the mark against the finding in SCC but then it just comes back again on the console. I also tried with the sample yaml file but no luck either. It ends up with same error textPayload: "2021/04/14 07:50:47 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated"

is there any other pointers you could offer? i am kinda stuck with this.. and i need it working for a demo to my customers.

daniel-cit commented 3 years ago

Hi @deepakdimri-ce until https://github.com/GoogleCloudPlatform/security-response-automation/issues/197 is fixed what you could do as a temporary workaround would be to comment the code that is marking the finding as remediated, something like this:

Go to the function that mark the finding as remediated https://github.com/GoogleCloudPlatform/security-response-automation/blob/d115061c1f855d75c64edf7ded8f7303518b9e3b/cloudfunctions/router/router.go#L176

And comment its body from:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
    m := map[string]string{"sra-remediated-event-time": eventTime}
    if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
        return err
    }
    return nil
}

to:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
    // m := map[string]string{"sra-remediated-event-time": eventTime}
    // if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
    //  return err
    // }
    return nil
}

This would prevent then security mark to return after you delete it in the console BUT this will make the finding be reprocessed by any change on it not related to the original cause if the finding satisfies you notification filter, like adding a security mark to the finding.

deepakdimri-ce commented 3 years ago

hi @daniel-cit - thanks again. i tried commenting the code as you suggested and now i see its not marking the findings. But it still does not do anything. router logs ends up with below messages and there is nothing interesting showing up in closebucket function logs.


{
  "textPayload": "2021/04/14 17:19:36 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:329: got rule \"public_bucket_acl\" with 1 automations",
  "insertId": "000000-6ef59f11-7aeb-4df1-8bbc-e6077c712d32",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:36.849Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}

{
  "textPayload": "2021/04/14 17:19:37 /workspace/serverless_function_source_code/clients/logger.go:60: failed to publish: \"failed to check if project \\\"\\\" is within the target or is excluded: failed to get project ancestry path: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>\\n<html lang=en>\\n  <meta charset=utf-8>\\n  <meta name=viewport content=\\\"initial-scale=1, minimum-scale=1, width=device-width\\\">\\n  <title>Error 404 (Not Found)!!1</title>\\n  <style>\\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\\n  </style>\\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\\n  <p><b>404.</b> <ins>That’s an error.</ins>\\n  <p>The requested URL <code>/v1/projects/:getAncestry?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>\\n\"",
  "insertId": "000000-424459ec-08f9-4eca-99c3-3cdd92b89b11",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:37.153Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}```
daniel-cit commented 3 years ago

Searching form part of the error message in your log failed to check if project we find this code:

func publish(ctx context.Context, services *Services, action, topic, projectID string, target, exclude []string, values interface{}) error {
    ok, err := services.Resource.CheckMatches(ctx, projectID, target, exclude)
    if err != nil {
        return errors.Wrapf(err, "failed to check if project %q is within the target or is excluded", projectID)
    }

The projectID subistituition in the message is \\\"\\\" which indicates that the value for the projectID is the empty string.

Since the code was able to hit the publish method it means it was able to find the correct automation and the issue looks like to be related to the reading the project from the finding.

Looking at real findings in the Security Command Center we can see that there used to be a field in the finding for the ProjectId: SourceProperties.ProjectId = "PROJECT_ID" but current findings have removed the field. the field existed at least up to january 2021.

Now the only place to find the project id is in the ResourcePath property

SourceProperties.ResourcePath = [ "projects/PROJECT_ID/", "folders/0000000000000/", "organizations/0000000000000/" ]

It looks like to be a general change over all the SHA findings.

Because of this the SHA findings remediations will not work until the parsing of the findings is fixed.

deepakdimri-ce commented 3 years ago

Yeah i see ResourcePath has PROJECT_ID but then we should get this fixed in the code to reflect what is being sent by the SCC. it's weird that we have this code base which does not auto remediate the findings as expected from it?

tomscript commented 3 years ago

thanks @deepakdimri-ce for reporting this and @daniel-cit for identifying the cause. really appreciate it

tomscript commented 3 years ago

@deepakdimri-ce can u give https://github.com/GoogleCloudPlatform/security-response-automation/pull/206 a test? it should work, but i have not actually tested it myself other than just locally

deepakdimri-ce commented 3 years ago

Thanks @tomscript for your help - i will test it later today and will let you know. In the meanwhile i have got this from SHA engineering team when asked for example finding event: Please use resource.project_display_name instead. SCC Platform adds the project information based on the resource name provided by the finding provider and is expected to be there even for non SHA findings. Here's an example: https://bit.googleplex.com/#/raghuagarwal/5790160849469440

Example:

{
  "notificationConfigName": "organizations/123456789011/notificationConfigs/test-notifications-ga-2",
  "finding": {
    "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375",
    "parent": "organizations/123456789011/sources/3873364276576983545",
    "resourceName": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "state": "ACTIVE",
    "category": "PRIVATE_GOOGLE_ACCESS_DISABLED",
    "externalUri": "https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936",
    "sourceProperties": {
      "Recommendation": "Go to https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936, click \"Edit\", and enable Private Google Access. Then, remove public (external) IPs from any VM instances whose only external traffic is to Google APIs by following the steps at: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#unassign_ip",
      "ReactivationCount": 0,
      "ExceptionInstructions": "Add the security mark \"allow_private_google_access_disabled\" to the asset with a value of \"true\" to prevent this finding from being activated again.",
      "Explanation": "Private Google Access enables VM instances with only internal (private) IP addresses to reach the public IP addresses of Google APIs and services. Learn more at: https://cloud.google.com/vpc/docs/configure-private-google-access",
      "ScannerName": "SUBNETWORK_SCANNER",
      "ResourcePath": [
        "projects/rd-project-725348--936/",
        "organizations/123456789011/"
      ],
      "compliance_standards": {
        "cis": [
          {
            "ids": [
              "3.8"
            ]
          }
        ]
      }
    },
    "securityMarks": {
      "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375/securityMarks"
    },
    "eventTime": "2021-04-15T14:18:18.029106Z",
    "createTime": "2020-10-22T09:39:08.327Z",
    "severity": "LOW",
    "canonicalName": "projects/555555555/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375"
  },
  "resource": {
    "name": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "project": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "projectDisplayName": "rd-project-725348--936",
    "parent": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "parentDisplayName": "rd-project-725348--936"
  }
}
tomscript commented 3 years ago

ah oops ok, so my change had resource under finding where this sample has it before it. np. ill adjust, let me fix it before u test

tomscript commented 3 years ago

ok the pr has been updated to reflect this sample (thanks for grabbing that!)

deepakdimri-ce commented 3 years ago

hi @tomscript - many thanks for working on this. I was able to test the closebucket function. the remediation is happening within a second - Its like i open the bucket for public access and with in that very second it removes the permissions.

tomscript commented 3 years ago

awesome to hear :) i'll update the other sha findings on monday, but you should be good to use the current pr for your demo so that shouldn't block you further. thanks for reporting :)

deepakdimri-ce commented 3 years ago

Yes, I am all set for now - thank you for the great support @tomscript