search

openshift / jenkins-sync-plugin

Synchronizes OpenShift BuildConfig objects using Jenkins as Jenkins jobs, then synchronizes build statuses into the OpenShift Build objects
Apache License 2.0
32 stars 41 forks source link

Build status doesn't sync #70

Closed spadgett closed 8 years ago

spadgett commented 8 years ago

I'm seeing the following errors in the Jenkins log with the latest plugin. Build status does not sync back to OpenShift.

@jimmidyson @bparees @jwforres

Jun 01, 2016 2:48:31 PM io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onStarted
INFO: starting polling build job/sample-pipeline/1/
Jun 01, 2016 2:48:34 PM io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild
INFO: replacing build in namespace pipelineproject with name: sample-pipeline-1 phase: Running
Jun 01, 2016 2:48:34 PM jenkins.util.ErrorLoggingScheduledThreadPoolExecutor afterExecute
WARNING: failure in task not wrapped in SafeTimerTask
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://kubernetes.default/oapi/v1/namespaces/pipelineproject/builds/sample-pipeline-1. Message: Build "sample-pipeline-1" is invalid: spec: Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Build, name=sample-pipeline-1, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=Build "sample-pipeline-1" is invalid: spec: Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:290)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:243)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:212)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:200)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:535)
    at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:87)
    at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:82)
    at io.fabric8.openshift.api.model.DoneableBuild.done(DoneableBuild.java:32)
    at io.fabric8.openshift.api.model.DoneableBuild.done(DoneableBuild.java:12)
    at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:94)
    at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:32)
    at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.upsertBuild(BuildSyncRunListener.java:228)
    at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.pollRun(BuildSyncRunListener.java:190)
    at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.pollLoop(BuildSyncRunListener.java:177)
    at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener$1.run(BuildSyncRunListener.java:143)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
jimmidyson commented 8 years ago

Is Build spec now immutable as the message says? What version of OpenShift is this running against?

bparees commented 8 years ago

Spec has always been immutable but I thought you were updating status?

Ben Parees | OpenShift On Jun 1, 2016 11:38, "Jimmi Dyson" notifications@github.com wrote:

Is Build spec now immutable as the message says? What version of OpenShift is this running against?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fabric8io/openshift-jenkins-sync-plugin/issues/70#issuecomment-223033184, or mute the thread https://github.com/notifications/unsubscribe/AEvl3uwmjS6mwNKkBc_5TagXU8Wosxzrks5qHaeHgaJpZM4Irq2w .

spadgett commented 8 years ago

@jimmidyson This is latest origin/master. We're debugging from the OpenShift side. I'll see what we can find.

Do the Jenkins build number and OpenShift build number always have to match?

spadgett commented 8 years ago

@sg00dwin and @fabianofranz have both seen this.

fabianofranz commented 8 years ago

From OpenShift master and image id for openshift/jenkins-1-centos7:dev is 76bc0c6deac2. Exactly the same issue: 

Jun 01, 2016 6:58:51 PM INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild
replacing build in namespace pipelineproject with name: sample-pipeline-8 phase: Failed
Jun 01, 2016 6:58:51 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://kubernetes.default/oapi/v1/namespaces/pipelineproject/builds/sample-pipeline-8. Message: Build "sample-pipeline-8" is invalid: spec: Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Build, name=sample-pipeline-8, retryAfterSeconds=null, additionalProperties={}), kind=Status, message=Build "sample-pipeline-8" is invalid: spec: Invalid value: "content of spec is not printed out, please refer to the \"details\"": spec is immutable, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:290)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:243)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:212)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:200)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:535)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:87)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:82)
        at io.fabric8.openshift.api.model.DoneableBuild.done(DoneableBuild.java:32)
        at io.fabric8.openshift.api.model.DoneableBuild.done(DoneableBuild.java:12)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:94)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:32)
        at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.upsertBuild(BuildSyncRunListener.java:228)
        at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.pollRun(BuildSyncRunListener.java:190)
        at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.onFinalized(BuildSyncRunListener.java:170)
        at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:232)
        at hudson.model.Run.onEndBuilding(Run.java:1888)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:540)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1100(WorkflowRun.java:111)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:777)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:843)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$4.run(CpsThreadGroup.java:340)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:32)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
jimmidyson commented 8 years ago

@spadgett No build numbers can be different.

jimmidyson commented 8 years ago

Working on patch support in the Java kubernetes client which is required for this. Seems as though immutability checks have been hardened recently(-ish). The client has always been doing a PUT with full resource, even though only few mutable fields have been changed. Seems as though this isn't supported any longer which is a PITA.

spadgett commented 8 years ago

@jimmidyson I thought you could PUT back the entire object as long as you didn't modify anything in spec. @csrwng @bparees can you confirm?

csrwng commented 8 years ago

@jimmidyson @spadgett that is correct, you can put the whole build, but the update will fail if the spec is different. I should have a pull shortly that returns the diff of the specs in the error message.

One thing that occurred to me why this is happening is that the build spec gets updated with git commit information by the build itself after the build starts. If you didn't retrieve the build before doing an update, you'd run into this. @bparees thoughts?

jimmidyson commented 8 years ago

@csrwng That would be very helpful. What updates the build spec with the git commit info? Didn't know that anything was updating it after create but before Jenkins was picking it up.

csrwng commented 8 years ago

The build itself updates it (at least in the case of docker and s2i builds).

jimmidyson commented 8 years ago

AFAIK nothing other than the Jenkins plugin should be updating the Build after creation (other than for cancellation). I cannot see anywhere in the Jenkins plugin where we are updating the spec but will investigate further early next week.

spadgett commented 8 years ago

@jimmidyson there was a change to add what triggered a build to the build spec.

edit: it looks like triggered by will stay in spec

spadgett commented 8 years ago

See https://github.com/openshift/origin/pull/7518

jwforres commented 8 years ago

So talking with @mfojtik the trigger information is added when the build is created, not after the fact.

@jimmidyson does the kubernetes java client strip things off the build JSON if it doesn't know about it in its model, is it possible its removing the new fields?

mfojtik commented 8 years ago

@jwforres @jimmidyson yeah, the trigger info should be in the build when it is created (either by BC or manual). After the build is created, you should not be changing that info.

jimmidyson commented 8 years ago

@spadgett It's definitely possible, but I don't think there's anything we don't know about. I'll debug it early next week & see what's going on, including @csrwng's PR to get a nice diff.

jimmidyson commented 8 years ago

Switching to patch soon which sends rfc correctly formatted request but openshift/kubernetes don't handle it properly. Issue raised at https://github.com/openshift/origin/issues/7629 with upstream fix at https://github.com/evanphx/json-patch/pull/27.

jimmidyson commented 8 years ago

This is waiting on openshift/origin#9206 being merged. I'm closing this issue here as it's fixed in our code, waiting for merge & release of Origin. If you want to try it out then please pull in openshift/origin#9206 to a local Origin build & run against that.