skaffold run some time stop unexpected

[nvcnvn]

Expected behavior

skaffold run --kube-context=$MINIKUBE_PROFILE successfully deploy to my local cluster.

Actual behavior

Image build canceled and nothing else.

Canceled build for asia.gcr.io/xxxx/xxxx

Help improve Skaffold with our 2-minute anonymous survey: run 'skaffold survey'
To help improve the quality of this product, we collect anonymized usage data for details on what is tracked and how we use this data visit <https://skaffold.dev/docs/resources/telemetry/>. This data is handled in accordance with our privacy policy <https://policies.google.com/privacy>

You may choose to opt out of this collection by running the following command:
    skaffold config set --global collect-metrics false

Information

• Skaffold version: v1.35.1
• Operating system: debian 10
• Installed via: skaffold.dev
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta26
kind: Config
profiles:
  - name: ci
    activation:
      - kubeContext: local-scaffold-profile
    build:
      local:
        tryImportMissing: false
        useDockerCLI: true
        useBuildkit: true
      artifacts:
      - image: asia.gcr.io/xxxxxx/backend
        context: .
        docker:
          dockerfile: ./developments/release.Dockerfile
          target: runner
      - image: asia.gcr.io/xxxxxx/gandalf-ci
        context: .
        docker:
          dockerfile: ./developments/release.Dockerfile
          target: gandalf-ci
      - image: asia.gcr.io/xxxxxx/gandalf-stub
        context: .
        docker:
          dockerfile: ./developments/release.Dockerfile
          target: gandalf-stub
    deploy:
      helm:
        releases:
          - name: nats-jetstream
            wait: false
            chartPath: deployments/helm/platforms/nats-jetstream
            namespace: local-xxxxxx-nats-jetstream
            createNamespace: true
            valuesFiles:
              - "deployments/helm/platforms/nats-jetstream/values.yaml"
            setValueTemplates:
              vendor: '{{.ORG}}'
              environment: '{{.ENV}}'
              serviceAccountEmailSuffix: '{{.SERVICE_ACCOUNT_EMAIL_SUFFIX}}'
              configs.kmsPath: '{{.KMS_PATH}}'
              secrets.serviceCredential: '{{.SVC_CRED}}'
          - name: elastic
            wait: false
            chartPath: deployments/helm/platforms/elastic
            namespace: elastic
            createNamespace: true
            valuesFiles:
              - "deployments/helm/platforms/elastic/values.yaml"
            setValueTemplates:
              vendor: '{{.ORG}}'
              environment: '{{.ENV}}'
              serviceAccountEmailSuffix: '{{.SERVICE_ACCOUNT_EMAIL_SUFFIX}}'
              configs.kmsPath: '{{.KMS_PATH}}'
              secrets.serviceCredential: '{{.SVC_CRED}}'
              elasticsearch:
                replicas: 1
                storage.storageClass: standard
          - name: istio-base
            repo: https://istio-release.storage.googleapis.com/charts
            remoteChart: base
            namespace: istio-system
            createNamespace: true
          - name: istiod
            repo: https://istio-release.storage.googleapis.com/charts
            remoteChart: istiod
            namespace: istio-system
            createNamespace: true
          - name: infras
            chartPath: ./deployments/helm/emulators/infras
            namespace: emulator
            createNamespace: true
            recreatePods: false
            skipBuildDependencies: false
            useHelmSecrets: false
            wait: false
          - name: firebase
            chartPath: ./deployments/helm/emulators/firebase
            namespace: emulator
            createNamespace: true
            recreatePods: false
            skipBuildDependencies: false
            useHelmSecrets: false
            wait: false
          - name: letsencrypt
            chartPath: ./deployments/helm/emulators/letsencrypt
            namespace: emulator
            createNamespace: true
            recreatePods: false
            skipBuildDependencies: false
            useHelmSecrets: false
            wait: false
          - name: xxxxxx-all-in-one
            wait: true
            chartPath: deployments/helm/xxxxxx-all-in-one
            namespace: "backend"
            createNamespace: true
            artifactOverrides:
              global:
                image: asia.gcr.io/xxxxxx/backend
              gandalf:
                stubImage: asia.gcr.io/xxxxxx/gandalf-stub
                ciImage: asia.gcr.io/xxxxxx/gandalf-ci
            imageStrategy:
              helm: {}
            valuesFiles:
              - "./deployments/helm/platforms/gateway/local-xxxx-values.yaml"
            setValueTemplates:
              svc1.replicaCount: 1
              svc2.replicaCount: 1
              svc3.replicaCount: 1
              svc4.replicaCount: 1
              svc5.replicaCount: 1
              svc6.replicaCount: 1
              svc7.replicaCount: 2
              svc8.replicaCount: 1
              svc9.replicaCount: 1
              global:
                localLogLevel: error
                draft.enabled: false
                configs.kmsPath: '{{.KMS_PATH}}'
                hasura.replicaCount: 1
                secrets.serviceCredential: '{{.SVC_CRED}}'
                project: '{{.PROJECT_ID}}'
                environment: '{{.ENV}}'
                vendor: '{{.ORG}}'
                appVersion: '{{.IMAGE_TAG}}'
                sqlProxy:
                  enabled: false
                  connName: '{{.SQL_PROXY_CONN_NAME}}'
                serviceAccountEmailSuffix: '{{.SERVICE_ACCOUNT_EMAIL_SUFFIX}}'

Steps to reproduce the behavior

Actually I cannot reproduce this easily, here is some vtrace log https://gist.github.com/nvcnvn/f3124f016d05b3196e99149129a0dda4

[halvards]

1. Are you able to build all three images from the Dockerfiles?

2. I notice that you don't have a "base" Skaffold config, everything is part of the ci profile. What happens if you move everything out of the profile (so build and deploy are top-level keys)?

[nvcnvn]

Are you able to build all three images from the Dockerfiles?

Let me check, need to add some debug code to see if any image built when the error happen, right now I just stop when the issue happen.

What happens if you move everything out of the profile (so build and deploy are top-level keys)?

I tried that already, same issue happen.

[tejal29]

@nvcnvn Looks the debug line "FIXME" is coming from docker API See this thread, https://githubhot.com/repo/wshihadeh/docker-deployment-action/issues/27

time="2022-01-14T05:11:12Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!: -1" module=api status_code=-1

Not sure if skaffold is breaking the log line. Can you please fix the ssh related issue or just try building the images like @halvards mentioned

[ericzzzzzzz]

@nvcnvn Were you able to attempt fixing the ssh issue and build the image like @tejal29 mentioned?

[nvcnvn]

I think this issue can be close since I surrender after all the methods already (also changed job).