search

fabric8io / jenkins-pipeline-library

a collection of reusable jenkins pipelines and pipeline functions
Apache License 2.0
276 stars 180 forks source link

provide an easy way to diagnose CI / CD pipeline failures #194

Closed jstrachan closed 6 years ago

jstrachan commented 6 years ago

we've been finding some builds sometimes fail inside a CI / CD pipeline but not on a users laptop. So it'd be nice if folks had a way to try diagnose build failures a little easier.

Ideally long term we'd turn a build pod into a Che Workspace that folks could just open. Though there's the issue that build pods tend to be composed (maven + clients + jnlp) whereas Che workspaces tend to be one single docker image with all the CLI tools there.

So maybe we just provide a way to let folks fork a failing pipeline and add a dummy input step into the pipeline before the failure line; then re-run that pipeline; then let folks kubectl exec -it nameOfPod -c containerName bash then they can cd into the folder and run whatever commands they need etc.

I wonder if we can improve on this somewhat? e.g. automatically generating the PR for the build failure with the input added? Automating the CLI commands to ssh into the paused build pod, cd to the right folder and run the last failing command?

jstrachan commented 6 years ago

I think CloudBees has a pause/resume/retry thingy but its not available in the OSS version AFAIK

jstrachan commented 6 years ago

we could do something like create a Pod YAML and add it to the Jenkins build as an artifact thats based on the Jenkins pipeline build pods; then make the pod do the git clone on startup (via an init-container) & then wait; so that folks could kubectl exec into it?

here's an example init-container to do a git clone on startup to a volume (which can be mounted into all the containers in the pod) https://github.com/fabric8io/fabric8-platform/blob/master/apps/keycloak/src/main/fabric8/deploymentConfig.yml#L12-L29

so that whether you work in the 'maven' container of the Pod or the clients container - the same git clone is visible?

jstrachan commented 6 years ago

then to test out a build, folks could just run this pod YAML then kubectl exec into the pod? if we put this YAML at a canonical place in Jenkins we could then write a gofabric8 CLI to run a command in the latest build's Pod YAML or something?

rupalibehera commented 6 years ago

should this issue be moved here https://github.com/fabric8io/fabric8-pipeline-library, this repo looks to be deprecated.

jstrachan commented 6 years ago

DOH! I meant to do that sorry ;) lemme do that now

jstrachan commented 6 years ago

replaced with https://github.com/fabric8io/fabric8-pipeline-library/issues/318