After implementing argocd-k8s-auth seeing Connection closed early in gke audit logs

[rwong2888]

As per, 9786

We see the same GKE Audit logs after migrating to argocd-k8s-auth and the message shows Connection closed early.

cc @toVersus

[toVersus]

I looked into this and found that those audit logs are normal and not harmful. I think we can close this issue.

At first, those logs are generated here and intended to record a long running request (i.e. watch API). I’m not a person in GKE team, so the only thing I can do is guessing from the sample Audit policy here. The default level is set to Metadata, and the kube-apiserver log request metadata without request and response body. As you know, the Argo CD application controller is calling watch API for available resources, so Connection closed early messages are logged. I’m not sure why those logs are not written when using argocd-manager service account in kube-system namespace, but I doubt that the requests authenticating with GCP service account are targeted.

I found other controller in a similar situation. The following is the audit logs for watch API called by GKE internal controllers with GCP internal service account:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "authenticationInfo": {
      "principalEmail": "service-********************@container-engine-robot.iam.gserviceaccount.com"
    },
    "authorizationInfo": [
      {
        "granted": true,
        "permission": "io.k8s.apiextensions.v1.customresourcedefinitions.watch",
        "resource": "apiextensions.k8s.io/v1/customresourcedefinitions"
      }
    ],
    "methodName": "io.k8s.apiextensions.v1.customresourcedefinitions.watch",
    "requestMetadata": {
      "callerIp": "xx.xx.xx.xx",
      "callerSuppliedUserAgent": "GoogleGKEGatewayController"
    },
    "resourceName": "apiextensions.k8s.io/v1/customresourcedefinitions",
    "serviceName": "k8s.io",
    "status": {
      "code": 0,
      "message": "Connection closed early"
    }
  },
  "resource": {
    "type": "k8s_cluster"
  },
  "labels": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "access granted by IAM permissions."
  },
  "logName": "projects/gcp-xxxxxxx/logs/cloudaudit.googleapis.com%2Factivity"
}

[rwong2888]

Thanks for the detailed analysis @toVersus ! Closing the issue