GoogleCloudPlatform / functions-framework-dotnet

FaaS (Function as a service) framework for writing portable .NET functions
Apache License 2.0
189 stars 41 forks source link

Error deploying with cloudbuild #176

Closed dpinart-uxland closed 3 years ago

dpinart-uxland commented 3 years ago

Hi I'm trying to deploy a cloud function using a cloudbuild trigger.

Apparently it seems it is working fine until the pipe terminates with a criptic error:

Step #1: Already have image (with digest): gcr.io/cloud-builders/gcloud
Step #1: Created .gcloudignore file. See `gcloud topic gcloudignore` for details.
Step #1: Deploying function (may take a while - up to 2 minutes)...
Step #1: ..
Step #1: For Cloud Build Stackdriver Logs, visit: https://console.cloud.google.com/logs/viewer?project=ghe-po-dev-10112020&advancedFilter=resource.type%3Dbuild%0Aresource.labels.build_id%3D2666ff1b-80ca-4317-8d4f-90f68bde0805%0AlogName%3Dprojects%2Fghe-po-dev-10112020%2Flogs%2Fcloudbuild
Step #1: ...................................................................................................................................................................failed.
Step #1: ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. Error message: Error: please examine your function logs to see the error cause: https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gcloud" failed: step exited with non-zero status: 1

Visiting the build logs I see:

2020-12-01T04:29:49.624201487Z Finished Step #6 - "tag" I 
  undefined
2020-12-01T04:29:49.624336747Z PUSH I 
  undefined
2020-12-01T04:29:49.624366565Z DONE I 
  undefined
2020-12-01T04:29:50.274090508Z Step #5 - "exporter": Unable to delete previous cache image: DELETE https://eu.gcr.io/v2/ghe-po-dev-10112020/gcf/europe-west3/a2b7dc19-2e88-45d0-9a42-231f65c0bf4a/cache/manifests/sha256:03cbce912ef1a8a658f73c660ab9c539d67188622f00b15c4f15b89b884f0e10: NAME_UNKNOWN: Failed to compute blob liveness for manifest: 'sha256:03cbce912ef1a8a658f73c660ab9c539d67188622f00b15c4f15b89b884f0e10' I 
  undefined

The function appears in the project function list marked in red. Going into the function logs I get an "A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found in '/workspace/bin/'.",

Any clue?

Below there are last log entries

 {
    "textPayload": "A fatal error was encountered. The library 'libhostpolicy.so' required to execute the application was not found in '/workspace/bin/'.",
    "insertId": "000000-ff0499d5-09fd-4893-a5bf-502def4b9c42",
    "resource": {
      "type": "cloud_function",
      "labels": {
        "project_id": "ghe-po-dev-10112020",
        "function_name": "invoice-parser",
        "region": "europe-west3"
      }
    },
    "timestamp": "2020-12-01T04:09:45.884Z",
    "labels": {
      "execution_id": ""
    },
    "logName": "projects/ghe-po-dev-10112020/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
    "receiveTimestamp": "2020-12-01T04:09:47.390566038Z"
  },
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "authenticationInfo": {
        "principalEmail": "276759109732@cloudbuild.gserviceaccount.com",
        "serviceAccountDelegationInfo": [
          {
            "firstPartyPrincipal": {
              "principalEmail": "cloud-build-argo-foreman@prod.google.com"
            }
          }
        ]
      },
      "requestMetadata": {
        "callerIp": "35.231.155.140",
        "callerSuppliedUserAgent": "google-cloud-sdk gcloud/318.0.0 command/gcloud.functions.deploy invocation-id/34cfc92f3b0b4b9f8f52d59734ee0b4c environment/GCE environment-version/None interactive/False from-script/True python/3.5.2 term/ (Linux 5.4.0-1029-gcp),gzip(gfe),gzip(gfe)",
        "requestAttributes": {
          "time": "2020-12-01T04:05:13.368791Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "cloudfunctions.googleapis.com",
      "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
      "authorizationInfo": [
        {
          "resource": "projects/ghe-po-dev-10112020/locations/europe-west3/functions/invoice-parser",
          "permission": "cloudfunctions.functions.update",
          "granted": true,
          "resourceAttributes": {}
        }
      ],
      "resourceName": "projects/ghe-po-dev-10112020/locations/europe-west3/functions/invoice-parser",
      "request": {
        "updateMask": "entryPoint,eventTrigger,httpsTrigger,runtime,sourceUploadUrl",
        "@type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest",
        "function": {
          "eventTrigger": {
            "eventType": "google.pubsub.topic.publish",
            "resource": "projects/ghe-po-dev-10112020/topics/po-invoice-scanned"
          },
          "ingressSettings": "ALLOW_ALL",
          "timeout": "60s",
          "availableMemoryMb": 256,
          "status": "OFFLINE",
          "updateTime": "2020-11-30T08:25:18.833Z",
          "serviceAccountEmail": "ghe-po-dev-10112020@appspot.gserviceaccount.com",
          "labels": {
            "deployment-tool": "cli-gcloud"
          },
          "versionId": "4",
          "entryPoint": "Galenicum.POInvoiceParser.FileParsedHandler",
          "runtime": "dotnet3",
          "name": "projects/ghe-po-dev-10112020/locations/europe-west3/functions/invoice-parser",
          "buildId": "ecfd0a16-647d-468c-b1f0-837f9488070a",
          "sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-europe-west3-3dfa026b-1dc3-454e-94c2-b60331b0a220/00fe9a16-f90d-4a3e-af6a-859976beef5f.zip?GoogleAccessId=service-493242935790@gcf-admin-robot.iam.gserviceaccount.com&Expires=1606797310&Signature=tUFy3lEUXtJlirdc4cbNWGgPPtoCDk8rnYWScBHLFm07qnUSgBpH17%2BrtM%2BHu6t09VjW6GY7JUJ9hGG8EQVQGiuQOIUP2dIlIQqNTIgFdbxbJJ3ySK2WVKJhGJ7F2NGdHfq4L0aoopTaNjbooNwvr7qETl10Q2siNZfc4esM%2F6gnkKTW6cOJbkyD3aoC%2BE60NvGano8Ec1JgLKCd%2F9IoohfUP%2BwSsh5GIRcFDQ98vKDsUZ3VVWyoRF1JsXTXYPof0Ylrcee2ZiN1VIe5sww7kxWTEWOpG1MAFRZkpZfb1Q73N6wQohi8bidb3Rf81T9MkCMiYALjU1P6m8DZ676MjQ%3D%3D"
        }
      },
      "resourceLocation": {
        "currentLocations": [
          "europe-west3"
        ]
      }
    },
    "insertId": "1yrfxtse1n15t",
    "resource": {
      "type": "cloud_function",
      "labels": {
        "function_name": "invoice-parser",
        "project_id": "ghe-po-dev-10112020",
        "region": "europe-west3"
      }
    },
    "timestamp": "2020-12-01T04:05:13.115242Z",
    "severity": "NOTICE",
    "logName": "projects/ghe-po-dev-10112020/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "operations/Z2hlLXBvLWRldi0xMDExMjAyMC9ldXJvcGUtd2VzdDMvaW52b2ljZS1wYXJzZXIvRjlBUTI3SGpGUjQ",
      "producer": "cloudfunctions.googleapis.com",
      "first": true
    },
    "receiveTimestamp": "2020-12-01T04:05:13.591996199Z"
  }
jskeet commented 3 years ago

Hmm... that does indeed sound strange. I'll try to reproduce it and see what I can find.

jskeet commented 3 years ago

I've just managed to deploy via Cloud Build myself pretty simply. It tooks a few attempts to get the IAM permissions right, but then it was straightforward.

This is my cloudbuild.yaml file:

steps:
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - functions
  - deploy
  - issue176
  - --entry-point=Issue176.Function
  - --runtime=dotnet3
  - --trigger-http
  - --allow-unauthenticated

I just ran dotnet new -i gcf-http and then gcloud builds submit .

Please could you give more details about your function? Does it deploy correctly if you use gcloud functions deploy locally?

dpinart-uxland commented 3 years ago

Hi @jskeet Thanks for yur interest.

No, I haven't tried to deploy the cloud function locally yet. I'll try it later and will let you know

This is a real project, meaning it references two other projects in the same solution (al same folder) and it uses Dependency Injection. Not sure if the "The library 'libhostpolicy.so' required to execute the application was not found in '/workspace/bin/'." error is meaningful. Maybe I should google about it

Our deployments are a bit complicated. We have several GCP projects in our organization. In fact, for each "project" we have 3 GCP projects, one per each environment (dev, QA, production)

We have an extra project that holds all the repos and all the triggers. So for each repo we have 3 triggers for branches (dev, QA and production).

Below there's the cloudbuild.yaml we are using:

steps:
  - name: gcr.io/cloud-builders/gsutil
    args: ['cp', '${_ENV_PATH}', 'env.yaml']
  - name: gcr.io/cloud-builders/gcloud
    args: [ "functions", 'deploy', "invoice-parser", 
           '--entry-point', 'Galenicum.POInvoiceParser.FileParsedHandler', 
           '--set-build-env-vars=GOOGLE_BUILDABLE=POInvoiceParser',
           '--runtime', 'dotnet3', 
           '--trigger-topic', 'po-invoice-scanned', 
           '--env-vars-file=env.yaml', 
           '--region=europe-west3', 
           '--project=${_TARGET_PROJECT_ID}']

It's a quite common build pipe in our projects. We do store environment variables in the repositories project storage, so we do have a build environment variable "_ENV_PATH" that is first copied to the step as env.yaml, then, this file is used as the env vars file '--env-vars-file=env.yaml'. Another build environment variable, _TARGET_PROJECT_ID, is used to set at wich project the function must be deployed

jskeet commented 3 years ago

Okay, I'd definitely start by getting it working locally first. If that fails, I would suggest reducing it to a minimal example that demonstrates the problem. (I did a bit of research about that error earlier, and it's not a massively uncommon one - but I'd like to see more details before guessing as to the cause.)

If it works when deploying directly but fails when deploying from CloudBuild, that will be a completely different set of diagnostic steps. Either way though, it would be really helpful if you could put together a complete but minimal example that demonstrates the problem.

dpinart-uxland commented 3 years ago

Sure,

thanks a lot Jon. I'll try it later

dpinart-uxland commented 3 years ago

I finally could deploy my function!!! As I'm setting build environment vars, I had to invoke cloud beta functions. Besides, I don't know why but I couldn't deploy my original project. I just created a very basic new project with the cloud functions template that got injected as a dependency the service from my original project and that's all

The final cloudbuild.yaml;

steps:
  - name: gcr.io/cloud-builders/gsutil
    args: ['cp', '${_ENV_PATH}', 'env.yaml']
  - name: gcr.io/cloud-builders/gcloud
    args: [ 'beta', 'functions', 'deploy', "invoice-parser", 
           '--entry-point', 'POInvoiceFunction.Function', 
           '--set-build-env-vars', 'GOOGLE_BUILDABLE=POInvoiceFunction',
           '--runtime', 'dotnet3', 
           '--trigger-topic', 'po-invoice-scanned', 
           '--env-vars-file=env.yaml', 
           '--region=europe-west3', 
           '--project=${_TARGET_PROJECT_ID}']

And that's all!

It's really nice we can finally deploy .net core cloud functions

jskeet commented 3 years ago

I'm very glad to hear it's sorted. Now I look again, I believe it's because of using GOOGLE_BUILDABLE which is only in beta. Sorry for not thinking of that before... Do raise other issues if you come across problems though!