embedded/base64 expected artifact for helm missing artifactAccount

byxorna commented 5 years ago

Issue Summary:

Using an embedded/base64 artifact as an overrides artifact for Helm2 Bake(Manifest) stage fails with An artifact account must be supplied to download this artifact: null

2 issues:

The UI does not surface that error - the pipeline fails but no error message is shown without going to the JSON payload.
The input artifact when selecting an embedded/base64 artifact looks like {"account" : "","id" : "56262a50-1696-47da-9066-c9c6ed6df110"}. Why is this? What should account be, if there is no credentials needed to decode the inlined data?

Cloud Provider(s):

self-hosted, installed in k8s with hal

Environment:

Feature Area (if this issue is UI/UX related, please tag `@spinnaker/ui-ux-team`):

Kubernetes manifest bakery

Description:

I am using Jenkins to export artifacts that are embedded/base64 encoded values.yaml, for later use in the Helm templating bakery step.

Steps to Reproduce:

Use a embedded/base64 values.yaml as an overrides artifact in Bake (Manifest) stage.

Additional Details:

I have a Jenkins stage that exports an embedded/base64 artifact that looks like:

{"reference":"LS0tCmltYWdlOgogIHJlcG9zaXRvcnk6IHJlZ2lzdHJ5LmNvbXBhbnkubmV0L2dhYmUvc2ltcGxlLXdlYmFwcAogIHRhZzogdjAuNi45LTUtZzRhZWIzMmEtbWFzdGVyCg==",
"name":"gabe/simple-webapp-values.yaml",
"type":"embedded/base64"
}

(this decodes to:)

image:
  repository: registry.tumblr.net/gabe/simple-webapp
  tag: v0.6.9-5-g4aeb32a-master

When I configure a Bake (Manifest) stage using HELM2 templating, and add the values.yaml artifact to the Overrides Expected Artifact, the configuration generated looks like:

{
  "expectedArtifacts": [
    {
      "defaultArtifact": {
        "customKind": true,
        "id": "52b9ed28-0915-4c21-9e78-c3b04bf6358d"
      },
      "displayName": "sour-lionfish-0",
      "id": "184c8e1d-12d2-4833-a1de-1374baccef2e",
      "matchArtifact": {
        "artifactAccount": "embedded-artifact",
        "id": "7cb89389-0764-4090-b497-771000d6f154",
        "type": "embedded/base64"
      },
      "useDefaultArtifact": false,
      "usePriorArtifact": false
    }
  ],
  "inputArtifacts": [
    {
      "id": "cf0a986b-2da6-452c-88df-d7ef792104ab"
    },
    {
      "account": "",
      "id": "56262a50-1696-47da-9066-c9c6ed6df110"
    }
  ],
  "name": "Bake (Manifest)",
  "namespace": "trash-island",
  "outputName": "simple-webapp-prod",
  "overrides": {},
  "templateRenderer": "HELM2",
  "type": "bakeManifest"
}

Running this pipeline results in an error during Manifest Baking:

...
      {
         "context" : {
            "exception" : {
               "details" : {
                  "error" : "Internal Server Error",
                  "errors" : [
                     "500 "
                  ],
                  "kind" : "HTTP",
                  "responseBody" : "{\"timestamp\":\"2019-08-08T20:20:09.129+0000\",\"status\":500,\"error\":\"Internal Server Error\",\"message\":\"500 \",\"body\":\"{\\\"error\\\":\\\"Internal Server Error\\\",\\\"message\\\":\\\"An artifact account must be supplied to download this artifact: null\\\",\\\"status\\\":500,\\\"timestamp\\\":1565295609126}\",\"url\":\"http://spin-clouddriver.spinnaker:7002/artifacts/fetch/\"}",
                  "status" : 500,
                  "url" : "http://spin-rosco.spinnaker:8087/api/v2/manifest/bake/helm"
               },
               "exceptionType" : "RetrofitError",
               "operation" : "createBake",
               "shouldRetry" : false,
               "timestamp" : 1565295609133
            },
            "expectedArtifacts" : [
               {
                  "defaultArtifact" : {
                     "customKind" : true,
                     "id" : "52b9ed28-0915-4c21-9e78-c3b04bf6358d"
                  },
                  "displayName" : "sour-lionfish-0",
                  "id" : "184c8e1d-12d2-4833-a1de-1374baccef2e",
                  "matchArtifact" : {
                     "artifactAccount" : "embedded-artifact",
                     "id" : "7cb89389-0764-4090-b497-771000d6f154",
                     "type" : "embedded/base64"
                  },
                  "useDefaultArtifact" : false,
                  "usePriorArtifact" : false
               }
            ],
            "inputArtifacts" : [
               {
                  "id" : "cf0a986b-2da6-452c-88df-d7ef792104ab"
               },
               {
                  "account" : "",
                  "id" : "56262a50-1696-47da-9066-c9c6ed6df110"
               }
            ],
            "namespace" : "trash-island",
            "outputName" : "simple-webapp-prod",
            "overrides" : {},
            "templateRenderer" : "HELM2"
         },
...

Clouddriver logs:

2019-08-19 16:44:04.376 ERROR 1 --- [0.0-7002-exec-9] c.n.s.k.w.e.GenericExceptionHandlers     : Internal Server Error

java.lang.IllegalArgumentException: An artifact account must be supplied to download this artifact: null
        at com.netflix.spinnaker.clouddriver.artifacts.ArtifactCredentialsRepository.getCredentials(ArtifactCredentialsRepository.java:55) ~[clouddriver-artifacts.jar:na]
        at com.netflix.spinnaker.clouddriver.artifacts.ArtifactDownloader.download(ArtifactDownloader.java:33) ~[clouddriver-artifacts.jar:na]
        at com.netflix.spinnaker.clouddriver.controllers.ArtifactController.lambda$fetch$0(ArtifactController.java:65) ~[clouddriver-web.jar:na]
        at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:110) ~[spring-webmvc-5.1.8.RELEASE.jar:5.1.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:97) ~[spring-webmvc-5.1.8.RELEASE.jar:5.1.8.RELEASE]
        at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:327) ~[spring-web-5.1.8.RELEASE.jar:5.1.8.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]

nscmcgrath commented 5 years ago

+1

petreboy14 commented 5 years ago

We are seeing the same on 1.16.0. When editing the pipeline as JSON the we see the input artifacts array in the bake stage as the following:

"inputArtifacts": [
        {
          "id": "50e76bb9-0917-4cb9-8c1f-c2e6b92d6553"
        },
        {
          "account": "",
          "id": "1a7cf603-eb3a-4758-812b-f7314c8b0fa6"
        }
      ],

where one is referencing a helm chart artifact and the other an s3 artifact. If i manually add the correct account values the pipeline will run successfully. This seems to have been not working since 1.15.x

ethanfrogers commented 5 years ago

@byxorna @petreboy14 I just tested this against the latest 1.16 release and am not able to replicate. The account field is populated in each of the scenarios I tried. Have you tried against the latest release? If so, and it's still not working, would you please post some step-by-step instructions on how to replicate?

petreboy14 commented 4 years ago

Hi @ethanfrogers sorry for the delay in response but I am still seeing this issue happening consistently with new pipelines utilizing a helm artifact. I have the following configuration setup:

A Docker registry automated trigger (trigger off any tag)
One docker image artifact
One Helm Chart artifact which is configured to talk to our Chartmuseum repository
One s3 artifact which is what we use for overrides

This is the configuration from json (with some names replaced):

"expectedArtifacts": [
    {
      "defaultArtifact": {
        "artifactAccount": "foo",
        "id": "b8a002bf-6421-4c7c-aef6-2f3e65859221",
        "name": "scp-service-foo",
        "type": "helm/chart",
        "version": "3.8.4"
      },
      "displayName": "Helm chart",
      "id": "96703f0b-519d-40c8-8fb7-ae8056580230",
      "matchArtifact": {
        "artifactAccount": "foo",
        "id": "035b8370-9fb8-4e39-99d3-fb672cc482bb",
        "name": "scp-service-foo",
        "type": "helm/chart",
        "version": "3.8.4"
      },
      "useDefaultArtifact": true,
      "usePriorArtifact": false
    },
    {
      "defaultArtifact": {
        "artifactAccount": "s3",
        "id": "110ae801-f787-44f6-bacf-563edb9e709a",
        "reference": "s3://foo-bucket/service.yaml",
        "type": "s3/object"
      },
      "displayName": "Overrides",
      "id": "2af96c40-61aa-461c-b222-7163e564c12c",
      "matchArtifact": {
        "artifactAccount": "s3",
        "id": "543e3727-0b82-4209-b08e-7f1abccb85a7",
        "name": "s3://foo-bucket/service.yaml",
        "type": "s3/object"
      },
      "useDefaultArtifact": true,
      "usePriorArtifact": false
    },
    {
      "defaultArtifact": {
        "artifactAccount": "docker-registry",
        "id": "71e697a0-6d72-4ac0-9392-23e12cb8a936",
        "name": "index.docker.io/foo/scp-service-foo",
        "reference": "index.docker.io/foo/scp-service-foo:latest",
        "type": "docker/image",
        "version": "latest"
      },
      "displayName": "Docker Image",
      "id": "1ce0eea2-002d-4a14-8e53-12ac1181fb1e",
      "matchArtifact": {
        "artifactAccount": "docker-registry",
        "id": "29b23d0a-3b98-4eca-9be4-9e9bb716a619",
        "name": "index.docker.io/foo/scp-service-foo",
        "type": "docker/image"
      },
      "useDefaultArtifact": true,
      "usePriorArtifact": false
    }
  ]

Then I create a Bake Manifest stage with the following properties:

Render engine: HELM2
Name: scp-service-foo
Namespace: development-shared
Template Artifact - Expected Artifact: Helm Chart (from the Helm Chart artifact)
Overrides - Expected Artifact: Overrides (from the S3 artifact)

I also setup a "produces artifact" but I don't think that section is relevant. Then the following is what i see in the JSON for the stage:

{
  "expectedArtifacts": [
    {
      "defaultArtifact": {
        "customKind": true,
        "id": "e05b59c6-f39c-49fe-b210-00c9ad941551"
      },
      "displayName": "Manifests",
      "id": "8cc15497-f570-4c0b-8c76-cc5e48c2a8c6",
      "matchArtifact": {
        "artifactAccount": "embedded-artifact",
        "id": "d5d62292-4f15-44bc-a80f-02af0308b811",
        "name": "Manifests",
        "type": "embedded/base64"
      },
      "useDefaultArtifact": false,
      "usePriorArtifact": false
    }
  ],
  "inputArtifacts": [
    {
      "account": "",
      "id": ""
    },
    {
      "account": "",
      "artifact": null,
      "id": "96703f0b-519d-40c8-8fb7-ae8056580230"
    },
    {
      "account": "",
      "artifact": null,
      "id": "2af96c40-61aa-461c-b222-7163e564c12c"
    }
  ],
  "isNew": true,
  "name": "Bake (Manifest)",
  "namespace": "development-shared",
  "outputName": "scp-service-test",
  "overrides": {},
  "templateRenderer": "HELM2",
  "type": "bakeManifest"
}

This is occurring on 1.17.0 as well (just tested). Please let me know if I can provide more info.

Forgot to mention I have artifactsRewrite turned on (not sure if that matters).

jaredstehler commented 4 years ago

I am also seeing this, with artifactsRewrite enabled. I have a feeling there's some mapping mismatch between fields named account and artifactAccount when using this feature?

spinnakerbot commented 4 years ago

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

petreboy14 commented 4 years ago

@spinnakerbot remove-label stale

spinnakerbot commented 4 years ago

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

angelcervilla commented 4 years ago

@spinnakerbot remove-label stale

maggieneterval commented 4 years ago

Hi all, thanks for your patience with this issue! Fixes will be backported to the next patch release of each supported version (1.17, 1.18, and 1.19).

amartyamandal commented 4 years ago

I am getting this issue as we speak any resolution spinnaker vs 1.20.5

spinnaker / spinnaker