search

kabanero-io / kabanero-pipelines

Default Kabanero Pipelines. This repo will be archived soon.
Apache License 2.0
14 stars 18 forks source link

if a build step in tekton pipeline fails, it should not go on to the next build step #86

Closed tam512 closed 4 years ago

tam512 commented 4 years ago

Using Kabanero 0.2.0 rc1, running a java-microprofile-build-deploy-pipeline in tekton dashboard. There was a problem in the pom.xml which caused the build-bud step fails (red mark), but it continues to go on to the next step build-push and show as green completed with No log available 1) If a build step fails, it should not go on to the next build step. 2) If it has to go on to the next step, then it should be failed also

image

image

kvijai82 commented 4 years ago

My first reaction without reproducing this is that this is a tekton dashboard issue. I don't think we actually run the build-push step. It would have blown up with lots of errors if it did. So I don't think we have a major issue here. I have only seen 2 states for steps in the dashboard. The red cross or the green check. Not sure if there is a 3rd one representing didn't execute. We will investigate it a bit more today to see how it looks like from the command line view today and decide on the next steps.

aadeshpa commented 4 years ago

We were able to re-produce this issue as below

  1. I created webhook for java-microprofile project in my github.ibm.com from the tekton dashboard.
  2. I made some code changes with compile time errors , so the build-bud step fails.

Results:

  1. oc logs -f my-java-mp-webhook-1571152644-build-task-wndcp-pod-2f0e7d -c step-build-bud

Logs Snippet for build-bud container in the pod (which was as expected to build fail):-

Downloaded from central: https://repo.maven.apache.org/maven2/com/google/collections/google-collections/1.0/google-collections-1.0.jar (640 kB at 1.2 MB/s)
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /project/user-app/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /project/user-app/src/main/java/dev/appsody/starter/StarterApplication.java:[8,5] <identifier> expected
[ERROR] /project/user-app/src/main/java/dev/appsody/starter/StarterApplication.java:[9,2] reached end of file while parsing
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  57.022 s
[INFO] Finished at: 2019-10-15T15:27:33Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project starter-app: Compilation failure: Compilation failure: 
[ERROR] /project/user-app/src/main/java/dev/appsody/starter/StarterApplication.java:[8,5] <identifier> expected
[ERROR] /project/user-app/src/main/java/dev/appsody/starter/StarterApplication.java:[9,2] reached end of file while parsing
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
subprocess exited with status 1
subprocess exited with status 1
error building at STEP "RUN mvn install -DskipTests": exit status 1
  1. oc logs -f my-java-mp-webhook-1571152644-build-task-wndcp-pod-2f0e7d -c step-build-push

Logs never returned anything :

[root@crofts1 common]# oc logs -f my-java-mp-webhook-1571152644-build-task-wndcp-pod-2f0e7d -c step-build-push
[root@crofts1 common]#

Observation:

This means that build-bud container step failed, and the build-push never started. However the tekton-dashboard shows the build-push step got completed. So we think it is a Tekton Dashboard issue.

aadeshpa commented 4 years ago

This issue is marked as blocked by tekton and we have opened the issue in Tekton Dashboard repository:- https://github.com/tektoncd/dashboard/issues/644

stephenkinder commented 4 years ago

Unlikely that Tekton would address in time to make release 0.2.0 and I removed the tag. We should pursue the issue under the tekton dashboard repo.

gecock commented 4 years ago

@aadeshpa @stephenkinder it looks like Tekton fixed their issue. Will this be making it into the next Kabanero release?

kvijai82 commented 4 years ago

@gecock from what In understand, this fix in only in tekton 0.8, which we are not picking up in kabanero 0.3.0. We can look into pulling this in for 0.4.0.

kvijai82 commented 4 years ago

I don't think we are going to be able to contain pulling in tekton 0.8 in 0.4.0 with the time we have lost working on 0.3.0 issues in 0.4.0. Might have to wait another iteration for this.

marikaj123 commented 4 years ago

@mtamboli - Are you good if I update the Kabanero release to v0.5.0 based on the comments provided by Vijai above?

marikaj123 commented 4 years ago

@kvijai82 - Is Kabanero 0.5.0 pulling tekton 0.8?

kvijai82 commented 4 years ago

As far as I know the plan is to pull in tekton 0.8 into Kabanero 0.5.0. We were able to successfully test triggers.

aadeshpa commented 4 years ago

I verified with collections 0.4.0 release with below setup

    collections:
      repositories:
      - activateDefaultCollections: true
        name: central
        url: https://github.com/kabanero-io/collections/releases/download/0.4.0/kabanero-index.yaml
    github: {}
    landing: {}
    tekton: {}
    version: 0.4.0
    webhook: {}
  status:
    admissionControllerWebhook:
      ready: "True"
    appsody:
      ready: "True"
      version: 0.2.2
    cli:
      hostnames:
      - kabanero-cli-kabanero.apps.vespers.os.fyre.ibm.com
      ready: "True"
    collectionController:
      ready: "True"
      version: 0.4.0
    kabaneroInstance:
      ready: "True"
      version: 0.4.0
    knativeEventing:
      ready: "True"
      version: 0.9.0
    landing:
      ready: "True"
      version: 0.4.0
    serverless:
      knativeServing:
        ready: "True"
        version: 0.8.1
      ready: "True"
      version: 1.1.0
    tekton:
      ready: "True"
      version: v0.7.0

My Tekton-dashboard version is v0.2.1

[root@vespers-inf auth]# kubectl get pod tekton-dashboard-6c4688f8bb-htmwc -n tekton-pipelines -o yaml | grep image
    image: openshift/oauth-proxy@sha256:6bc1759a3202b4614739f12441461e344907f6b3f758c34314284debe36d4e15
    imagePullPolicy: IfNotPresent
    image: gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard@sha256:fa7cef8ad3cbb1d79af45d87ec7e8913f886b3d55b10acd55995f907aced9ebb
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
    image: docker.io/openshift/oauth-proxy@sha256:6bc1759a3202b4614739f12441461e344907f6b3f758c34314284debe36d4e15
    imageID: docker.io/openshift/oauth-proxy@sha256:6bc1759a3202b4614739f12441461e344907f6b3f758c34314284debe36d4e15
    image: gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard@sha256:fa7cef8ad3cbb1d79af45d87ec7e8913f886b3d55b10acd55995f907aced9ebb
    imageID: gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard@sha256:fa7cef8ad3cbb1d79af45d87ec7e8913f886b3d55b10acd55995f907aced9ebb
  1. Created webhook for Java microprofile collection project in github.
  2. Edited java file with syntax error, and commited the changes, which triggerred build-push pipeline
  3. The pipelinerun failed in build step as expected and in the dashboard it showed that nothing ran after the build step.

This shows if a step fails the pipelinerun in dashboard does not go to next step.

Screen Shot 2020-01-10 at 3 54 53 PM
marikaj123 commented 4 years ago

@tam512 - According to Ashish the issue is not reproducible, can you please test it with the latest Tekton versions?

tam512 commented 4 years ago

I'm testing with Kabanero 0.5.0-rc2 which includes Tekton version v0.8.0. Inject an error to pom.xml to make build step fails and I still see the later steps being green/pass

# oc get kabanero -n kabanero -o yaml
apiVersion: v1
items:
- apiVersion: kabanero.io/v1alpha1
  kind: Kabanero
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kabanero.io/v1alpha1","kind":"Kabanero","metadata":{"annotations":{},"name":"kabanero","namespace":"kabanero"},"spec":{"collections":{"repositories":[{"activateDefaultCollections":true,"name":"central","url":"https://github.com/kabanero-io/collections/releases/download/0.5.0-rc.3/kabanero-index.yaml"}]},"version":"0.5.0"}}
    creationTimestamp: "2020-01-14T22:56:07Z"
    finalizers:
    - kabanero.io.kabanero-operator
    generation: 3
    name: kabanero
    namespace: kabanero
    resourceVersion: "93188"
    selfLink: /apis/kabanero.io/v1alpha1/namespaces/kabanero/kabaneros/kabanero
    uid: 0c8b422a-3721-11ea-b529-00000a1a02d4
  spec:
    admissionControllerWebhook: {}
    che:
      cheOperatorInstance: {}
      enable: false
      kabaneroChe: {}
    cliServices: {}
    collectionController: {}
    collections:
      repositories:
      - activateDefaultCollections: true
        name: central
        url: https://github.com/kabanero-io/collections/releases/download/0.5.0-rc.3/kabanero-index.yaml
    events: {}
    github: {}
    landing: {}
    targetNamespaces:
    - svt-project1
    - svt-project2
    - svt-project3
    tekton: {}
    version: 0.5.0
  status:
    admissionControllerWebhook:
      ready: "True"
    appsody:
      ready: "True"
      version: 0.3.0
    cli:
      hostnames:
      - kabanero-cli-kabanero.apps.toast.os.fyre.ibm.com
      ready: "True"
    collectionController:
      ready: "True"
      version: 0.5.0-rc.1
    kabaneroInstance:
      ready: "True"
      version: 0.5.0
    knativeEventing:
      ready: "True"
      version: 0.9.0
    landing:
      ready: "True"
      version: 0.5.0-rc.1
    serverless:
      knativeServing:
        ready: "True"
        version: 0.9.0
      ready: "True"
      version: 1.2.0
    tekton:
      ready: "True"
      version: v0.8.0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
ncskier commented 4 years ago

@tam512 I just verified that I also see the bug when running Tekton Dashboard v0.3.0 and Tekton Pipelines v0.8.0. Screen Shot 2020-01-16 at 11 53 37 AM

This bug was fixed in the Tekton Dashboard v0.2.1 with PR https://github.com/tektoncd/dashboard/pull/653. However, the fix PR was reverted for Dashboard v0.3.0 when Tekton Pipelines v0.9.0 fixed an underlying issue causing the bug https://github.com/tektoncd/dashboard/pull/795.

Unfortunately, Dashboard v0.3.0 is now bundled with Tekton Pipelines v0.8.0. So, we have the bug again.

I just verified that once the version of Tekton Pipelines is bumped up to v0.9.0 or greater, this bug will be resolved: Screen Shot 2020-01-16 at 11 50 12 AM

UPDATE: I was assuming that the next Kabanero release would bump up the Tekton Pipelines version to v0.9.0 or greater. However, apparently that's not the case. So it might be worthwhile to revert https://github.com/tektoncd/dashboard/pull/795 to patch up the Dashboard to work with Tekton Pipelines v0.8.0.

kvijai82 commented 4 years ago

This was fixed in tekton 0.7 and the fix was successfully tested. The bug showed up again when we pulled tekton 0.8. Opening new tracking issue for it.

tam512 commented 4 years ago

@kvijai82 please post the new issue number here once you have it. Thanks!