search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
490 stars 317 forks source link

Broke with Jackson 2 API v2.12.0 #821

Closed felipecrs closed 3 years ago

felipecrs commented 3 years ago

After updating Jackson 2 API to v2.12.0, this plugin stopped working. It did not start any container anymore, with an ERROR statement in the agent template section but when I click on it nothing happens.

Reverting the Jackson 2 API plugin to v2.11.3 solved the problem.

dkrowl commented 3 years ago

@felipecrs With what version of Jenkins are you getting this? I think that there was no problems with the latest weekly version (2.268). But with the LTS (2.249.3) I think I'm getting the same error when it just hangs on startup of a container.

EDIT: it is broken with the weekly release (2.268) of Jenkins too.

pjdarton commented 3 years ago

I'd suggest you take a look in the Jenkins logs - there's probably an exception trace in there somewhere. Exception traces are big & ugly, but they're (usually) highly informative and pretty much essential to debugging this sort of "it's broken in my environment" kind of issues.

Also, take screenshots and post them here too - show folks what you're seeing - it'll help us understand what's going on.

dkrowl commented 3 years ago

@pjdarton sorry, I know how frustrating it is to get these "this is broken" messages with no specific info of the error.

01, 2020 11:31:52 AP. com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
INFO: Started container ID 2ef1b472ff7bdd382b179d22eb621f50a04d55f981b362af583815b5afb49c04 for node label-0006aayhtig36 from image: repo/image:latest
01, 2020 11:31:55 AP. com.github.dockerjava.core.async.ResultCallbackTemplate onError
SEVERE: Error during callback
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `com.github.dockerjava.api.model.Volume` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('/run')
 at [Source: (byte[])"{"Id":"2ef1b472ff7bdd382b179d22eb621f50a04d55f981b362af583815b5afb49c04","Created":"2020-12-01T09:31:51.742120235Z","Path":"setup-sshd","Args":["/bin/sh"],"State":{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":13910,"ExitCode":0,"Error"
:"","StartedAt":"2020-12-01T09:31:54.462548469Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:0d118823ac28e35e1e6142c3ca226b688703e6811ed5f376bfadb6e0b04e3486","ResolvConfPath":"/var/lib/docker/con"[truncated 6041 bytes]; line: 1, column: 3331] (through reference chain: com.github.dockerjava.api.command.Insp
ectContainerResponse["Mounts"]->java.util.ArrayList[0]->com.github.dockerjava.api.command.InspectContainerResponse$Mount["Destination"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1590)
        at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1215)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromString(StdDeserializer.java:311)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromString(BeanDeserializerBase.java:1480)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:207)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:197)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:347)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4591)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3624)
        at com.github.dockerjava.netty.handler.JsonResponseCallbackHandler.channelRead0(JsonResponseCallbackHandler.java:40)
        at com.github.dockerjava.netty.handler.JsonResponseCallbackHandler.channelRead0(JsonResponseCallbackHandler.java:17)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:72)
        at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
        at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

01, 2020 11:31:56 AP. com.nirima.jenkins.plugins.docker.DockerCloud$1 run
SEVERE: Error in provisioning; template='DockerTemplate{configVersion=2, labelString='somelabel', connector=DockerComputerAttachConnector{user=''}, remoteFs='/home/jenkins', instanceCap=2147483647, mode=EXCLUSIVE, retentionStrategy=com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy@29, dockerTemplateBase=DockerTemplateBase{image='repo/image:latest', pullCredentialsId='', dockerCommand='', hostname='', user='', extraGroups=[], dnsHosts=[], network'='', volumes=[], volumesFrom2=[], devices=[], environment=[], bindPorts'='', bindAllPorts=false, privileged=false, tty=false, extraHosts=[]}, removeVolumes=true, stopTimeout=0, pullStrategy=PULL_ALWAYS, pullTimeout=0, disabled=BySystem,0 ms,4 min 59 sec,Template provisioning failed., name='label'}' for cloud='docker'
pjdarton commented 3 years ago

Thanks for that - it's said that, in journalism, "a picture is worth 1000 words" ... but in software, it's "an exception trace is worth 1000 words" 😉

...so, looking at that exception trace, I don't see any part of the docker-plugin get a mention there (no mention of package com.nirima.jenkins.plugins.docker or io.jenkins.docker) so, while I can see that the problem manifests as a failure in the docker-plugin's functionality, I think that this is an argument between the dockerjava library and the Jackson library, and there's little that the docker-plugin code can do about that ☹️

FYI the docker-plugin (i.e. this repo's code) depends on the docker-java-api plugin, and that provides the docker-java library to this plugin (and to other docker-related plugins in Jenkins). So, either the docker-java-api plugin needs to tie down jackson to a compatible (earlier!) version, or it needs to use a docker-java library that's happy with the later Jackson ... which may require changes in the docker-java library.

i.e. the code that needs to change to fix this isn't in this plugin. ...although, if the fix results in a new version of the docker-java-api plugin that is not fully backwards-compatible then this plugin will need to change its pom.xml to demand that new version (and coordinate a new release here with the new release of docker-java-api)

pjdarton commented 3 years ago

PS. Make sure that you're using the latest docker-plugin release (with the latest docker-java-api plugin too), as it's more likely that the older versions will be less compatible with newer versions of jackson. I don't expect there to be any difference between the latest docker-plugin release and the as-yet-unreleased version, but if you're using 1.1.9 or earlier then you'll have the (very) old version of docker-java-api-plugin too.

felipecrs commented 3 years ago

Latest Jenkins LTS Latest Docker plugin Latest Docker API Latest Jackson 2 API

I keep all my plugins up-to-date, so the issue started with the last version of Jackson 2 API as stated in the issue description. There are no other upgrades available in my Jenkins other than Jackson 2 API plugin since I reverted.

ericcitaire commented 3 years ago

Same error. I created an issue on Jenkins JIRA. https://issues.jenkins.io/browse/JENKINS-64343

pjdarton commented 3 years ago

So... my thinking is that this is one of the following possibilities: a) it could be a bug in the docker-java code - maybe it was always "a bit incorrect" w.r.t. its use of Jackson and 2.12.0 finally stopped tolerating that, or b) it's a bug in Jackson 2.12.0 that'll hopefully be fixed ASAP, or c) it's a bug in the docker daemon generating incorrect JSON and Jackson 2.12.0 finally stopped tolerating that, or d) there's some pecularity in the docker-plugin or the docker-java-api-plugin that causes this malfunction (even though neither code is in the stacktrace), or e) I've missed something and it's none of the above...

One would normally hope for there to be no breaking changes without a change in major version number, so while having a bug in Jackson would (I think) normally be unlikely, this is evidence supporting that idea. I've no opinions of the liklihood of the other two options but as this can all be described with the words "that shouldn't happen", I wouldn't advise ruling anything out.

I think that, to make further headway, what we're going to need is a full log of the JSON response that Jackson was being asked to parse. I've examined the exception message and, unfortunately, it says [truncated 6041 bytes] after only 500(ish) characters and so it doesn't show us what the unparsable volume/mount bit was. That means we don't know what Jackson failed to parse.

So that's the next step, I think: Someone who's seeing this issue needs to put some form of logging proxy between Jenkins and the docker daemon and find out exactly what traffic is sent and returned when this failure happens. Post the result here and that'll then help folks figure out which situation we're facing and hence where we should direct our combined investigatory powers next...

d-mankowski-samsung commented 3 years ago

Steps to reproduce this issue:

Dockerfile:

FROM jenkins/jenkins:lts-alpine
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
USER root
ENV JAVA_OPTS="-Djenkins.CLI.disabled=true -Djenkins.install.runSetupWizard=false -Duser.timezone=Europe/Warsaw" \
    JENKINS_OPTS="--logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war" \
    TZ=Europe/Warsaw
# Read more about JENKINS_OPTS here: https://technology.riotgames.com/news/docker-jenkins-data-persists
RUN apk add --update --no-cache \
    tzdata && \
    rm -rf /var/cache/apk/* && \
    mkdir -p /var/log/jenkins /var/cache/jenkins && \
    chown -R jenkins:jenkins /var/log/jenkins /var/cache/jenkins && \
    /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
WORKDIR /var/jenkins_home

where plugins.txt looks like this:

configuration-as-code
disable-github-multibranch-status
docker-plugin
docker-workflow
job-dsl
locale
workflow-aggregator

and docker-compose.yml:

version: "3.8"

services:
  jenkins:
    image: myjenkins:latest
    deploy:
      mode: replicated
      replicas: 1
    ports:
      - 8080:8080
    volumes:
      - jenkins_log:/var/log/jenkins              # Persistent storage for Jenkins master logs.
      - /var/run/docker.sock:/var/run/docker.sock # Run Docker from within container, using host's socket (search: Docker outside of Docker).

volumes:
  jenkins_log:

Run docker image build -t myjenkins . && docker-compose up and in Configure Clouds section set it as follows image

Add a pipeline job:

pipeline {
    agent { label 'agent' } 

    stages {
        stage('Hello') {
            steps {
                echo 'Hello World'
            }
        }
    }
}

and run it and you will see a nice Bad Message 414 image

caused by 

Dec 02, 2020 12:10:40 PM WARNING hudson.slaves.NodeProvisioner lambda$update$6

Unexpected exception encountered while provisioning agent Image of jenkins/agent:4.3-7-alpine
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `com.github.dockerjava.api.model.Volume` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('/home/jenkins/.jenkins')
 at [Source: (byte[])"{"Id":"2f198f9529aa99c23ac24ffb4bbb67166f887c4f81cdf99429b0161af48a9cc1","Created":"2020-12-02T11:10:31.401559848Z","Path":"/bin/sh","Args":[],"State":{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":8200,"ExitCode":0,"Error":"","StartedAt":"2020-12-02T11:10:31.936370137Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:19d42d771ed635a6368de418c72760d1d09db1954dcb707ce5dc64e10356a2be","ResolvConfPath":"/var/lib/docker/containers/2f198"[truncated 6047 bytes]; line: 1, column: 3993] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["Mounts"]->java.util.ArrayList[0]->com.github.dockerjava.api.command.InspectContainerResponse$Mount["Destination"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1590)
    at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1215)
    at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromString(StdDeserializer.java:311)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromString(BeanDeserializerBase.java:1480)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:207)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:197)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:347)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4591)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3624)
    at com.github.dockerjava.netty.handler.JsonResponseCallbackHandler.channelRead0(JsonResponseCallbackHandler.java:40)
    at com.github.dockerjava.netty.handler.JsonResponseCallbackHandler.channelRead0(JsonResponseCallbackHandler.java:17)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:72)
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
    at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Caused: java.lang.RuntimeException
    at com.google.common.base.Throwables.propagate(Throwables.java:156)
    at com.github.dockerjava.core.async.ResultCallbackTemplate.throwFirstError(ResultCallbackTemplate.java:139)
    at com.github.dockerjava.core.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:94)
    at com.github.dockerjava.netty.NettyInvocationBuilder$ResponseCallback.awaitResult(NettyInvocationBuilder.java:60)
    at com.github.dockerjava.netty.NettyInvocationBuilder.get(NettyInvocationBuilder.java:139)
    at com.github.dockerjava.core.exec.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:31)
    at com.github.dockerjava.core.exec.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:13)
    at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at com.github.dockerjava.core.command.InspectContainerCmdImpl.exec(InspectContainerCmdImpl.java:52)
    at io.jenkins.docker.connector.DockerComputerConnector.createLauncher(DockerComputerConnector.java:133)
    at com.nirima.jenkins.plugins.docker.DockerTemplate.doProvisionNode(DockerTemplate.java:731)
    at com.nirima.jenkins.plugins.docker.DockerTemplate.provisionNode(DockerTemplate.java:676)
    at com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:375)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Dec 02, 2020 12:10:50 PM WARNING org.eclipse.jetty.http.HttpParser parseLine

URI is too large >8192
docker --version
Docker version 19.03.13, build 4484c46d9d
thefueley commented 3 years ago

Experiencing the same here. I rolled back Jackson 2 to 2.11.3 but that brought up a new error for me.

Failed to connect to 172.17.0.1:32791 (try 1/11) - There was a problem while connecting to 172.17.0.1:32791

Not sure why it's trying port 32791 when I have it configured on port 2375

pjdarton commented 3 years ago

@thefueley Are you 100% sure that the "new error" is this issue? That's the sort of thing you might see if you're trying to use an SSH connection method to talk to a container that wasn't running sshd at the time. I'm not aware of there being a "try 11 times" retry mechanism for talking to the docker daemon itself. My guess is that rolling back fixed this issue for you ... but you're now seeing an entirely unrelated error now that you've got past this one.

thefueley commented 3 years ago

@pjdarton You're right. I that's the issue I'm facing now.

timja commented 3 years ago

reported to docker-java in https://github.com/docker-java/docker-java/issues/1509

wumpz commented 3 years ago

Same problem here. Reverting the Jackson 2 API plugin to v2.11.3 solved the problem.

lpandzic commented 3 years ago

Most likely cause of the problem I suspect is https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12#annotation-less-constructor-auto-detection.

fuzzmz commented 3 years ago

Unfortunately reverting doesn't work in all cases, as there are several other plugins (such as the Kubernetes one) which rely on the latest version of Jackson 2 API.

afischer211 commented 3 years ago

And unfortunately the docker-api-plugin (which depends on jackson 2 api) is up for adoption. So a quick fix is not on the way...

felipecrs commented 3 years ago

Unfortunately reverting doesn't work in all cases, as there are several other plugins (such as the Kubernetes one) which rely on the latest version of Jackson 2 API.

In this case it is needed to downgrade the kubernetes plugin as well.

afischer211 commented 3 years ago

The downgrade for kubernetes is not enough, also the new junit plugin needs the new jackson-api. So we can not update misc plugins any more...

timja commented 3 years ago

Likely needs a fix in either docker-java or jackson (most likely docker-java).

kleini commented 3 years ago

This requires a fix in the docker-java. I have similar problems with Jackson 2.12.0. A lot of automatic object mapping from either JSON and XML to Java broke for our usages. Often just applying the correct annotations solved the problems.

felipecrs commented 3 years ago

Actually, the right place to be fixed would be in Jackson 2 API, since they pushed a breaking changed without bumping the major version.

lpandzic commented 3 years ago

@cowtowncoder FYI I'm not sure you're aware, I suspect the new creator discovery algorithm isn't backwards compatible for all cases.

mprins commented 3 years ago

Actually, the right place to be fixed would be in Jackson 2 API, since they pushed a breaking changed without bumping the major version.

jackson uses a versionion scheme where any point release should be considered to have API breaking changes like 2.9.x is not 100% compatible with 2.10.x is is not 100% compatible with 2.11.x and is now is not 100% compatible with 2.12.x

felipecrs commented 3 years ago

jackson uses a versionion scheme where any point release should be considered to have API breaking changes like 2.9.x is not 100% compatible with 2.10.x is is not 100% compatible with 2.11.x and is now is not 100% compatible with 2.12.x

Makes sense, I thought it was semver 2.

Is there any way to pin a dependency at a given version for a plugin? For example, docker-java-api could set jackson 2 2.11 as dependency.

timja commented 3 years ago

@jvz fyi

jvz commented 3 years ago

Any plugin that wishes to bundle Jackson itself can do so using the Plugin-First-Class-Loader option in its manifest, though I must warn you that due to the numerous CVEs they publish (particularly egregious in 2.9.x), if you go this route, you must keep the plugin up to date as these CVEs are published. Wasting the Jenkins Security Team's time by bundling outdated and vulnerable dependencies is a surefire way to get removed from the update center at some point (cc @daniel-beck).

Breaking up the Jackson API plugin version to indicate the breaking change isn't so great, either, since Jackson 3.0 is still in development as it is (though I'd imagine that might end up as a separate API plugin depending on how it gets packaged) which would make a version mismatch get confusing.

kleini commented 3 years ago

Will have a look tomorrow into the code. Can't be that hard to fix JSON parsing with the Objectmapper.

cowtowncoder commented 3 years ago

@lpandzic No intentional changes to logic, obviously. All existing tests, regression tests passed between 2.11 and 2.12. All problems reported during 2 release candidates were resolved before 2.12.0 as well; too bad issue here was not encountered by anyone during pre-release testing.

But as to issues reported so far regarding 2.12.0, this one about Jackson module auto-registration:

https://github.com/FasterXML/jackson-databind/issues/2983

could explain problems with single-constructor-argument implicit Creator method auto-detection.

Without reproduction of the problem I cannot comment much more but I wish people resisted jumping to conclusions of there being big intentional breaking change that Jackson project chose to make (compatibility changes that are intended and know are included in release notes like https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12).

It would make sense to first isolate the specific case(s) and see what the root cause is. There is a good chance this could be resolved for 2.12.1 patch.

pjdarton commented 3 years ago

w.r.t. "specific case(s)", I can tell you that the "how we got here" trail is as follows...

  1. This plugin depends on docker-java-api, version 3.1.5, which is the latest available right now.

  2. docker-java-api in turn pulls in docker-java version 3.1.5 which is not the latest version ... but upgrading beyond 3.1.5 would be non-trivial.

  3. docker-java version 3.1.5 contains the com.github.dockerjava.api.model.Volume code which Jackson mentions in the exception trace.

  4. ...and that it probably threw this while parsing a docker inspect command's result into a InspectContainerResponse.

...but that's the limit of my knowledge at present.

cowtowncoder commented 3 years ago

@pjdarton Ah! That is helpful -- I think the underlying issue then is this one:

https://github.com/FasterXML/jackson-databind/issues/2962

which was reported after 2.12.0, has been fixed and will be included in 2.12.1 when it gets released (late December or early January depending how things go).

One suggestion for a work-around wrt 2.12.0 (and arguably improves readability) would be to add @JsonCreator(mode = Mode.DELEGATING) on:

    public Volume(String path) { }

which simply makes it explicit that this constructor should also be used, in addition to explicitly annotated factory method.

pjdarton commented 3 years ago

My personal preference is for DTO code annotations to provide only one entry point so it's clear to anyone which route in is the one being used. That class already has an annotated factory method, so adding another "Hey, Jackson, you can use this" annotation to provide a second route in just creates ambiguity (especially to those, like myself, who aren't sufficiently familiar with Jackson to determine which is used and which is unnecessary). i.e. Two annotated creation methods is one more than ought to be necessary.

...and I can't be certain if that'd be the only bit of code that'd be affected and need tweaking; all we know right now is that's the first place where such an exception is triggered - if we fix it there, we might hit a similar issue in another class, and another etc. We'd really have to go through the entire docker-java codebase looking for DTOs and fix all of them.

...and even if we did that, a more awkward issues is that we're not using the latest docker-java code, it would be non-trivial to upgrade to using the latest docker-java code, and it'd be unreasonable to ask the docker-java maintainer to create a 3.1.x release with such a change when their focus is on 3.2.x (no maintainer likes doing patches against old code, and I'm not geographically co-located so I can't provide beer as an incentive!) TL;DR: Changing docker-java is non-trivial 😢

However, what might be a more viable option would be to make a newer Jenkins Jackson plugin containing the as-yet-unreleased Jackson code, so that folks who are suffering from this issue (who can't use 2.11.3 because of other dependencies demanding 2.12) can then use that as a temporary workaround until the official release happens.

timja commented 3 years ago

would be good to verify if 2.12.1-SNAPSHOT fixes this for people, would need someone to build this with a patched jackson: https://github.com/jenkinsci/jackson2-api-plugin

cowtowncoder commented 3 years ago

@pjdarton Perhaps I did not explain this clearly... annotation is not adding anything that did not exist prior: both entry points were discovered before 2.12.0: one that takes JSON String (non-annotated constructor), and another that takes JSON Object (annotated factory method, where @JsonProperty indicates single property within JSON Object to bind). This expectation is actually explained in class Javadoc comment.

The unintended change in 2.12.0 was to assume that if ANY factory method were annotated then ALL constructors to be considered must also be annotated (actual existing logic was that if one factory method annotated, all factory method must be; similarly for constructors).

So adding annotation would be formally documenting the fact that the constructor is to be used; not to add another entry point but to explicitly specify both, instead of relying on auto-detection for one of them.

I agree that this might not be the only case that would require addition.

MauricioPenteado commented 3 years ago

[ERROR] Failed to execute goal on project jackson2-api: Could not resolve dependencies for project org.jenkins-ci.plugins:jackson2-api:hpi:2.12.0: Failed to collect dependencies at com.fasterxml.jackson.core:jackson-databind:jar:2.12.1: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.12.1: Could not transfer artifact com.fasterxml.jackson.core:jackson-databind:pom:2.12.1 from/to repository.spring.snapshot (http://repo.spring.io/snapshot): Access denied to: http://repo.spring.io/snapshot/com/fasterxml/jackson/core/jackson-databind/2.12.1/jackson-databind-2.12.1.pom , ReasonPhrase:Forbidden.

How can I build the plugin using the jackson-databind:pom:2.12.1 ? I am only getting Access denied when downloading this dependency.

pjdarton commented 3 years ago

@MauricioPenteado You don't build this plugin; what has been suggested is to build the Jenkins jackson2-api-plugin with Jackson 2.12.1 ... but to do that you'll need to grab a 2.12.1-SNAPSHOT pre-release version (because 2.12.1 isn't released yet).

I'm not familiar with the jackson2-api-plugin so I don't know if there's any special measures necessary for doing that; I'd guess that if the pre-releases aren't published alongside the main releases, you'd have to manually download a pre-release & put that into your local maven cache, but then you update your jackson2-api-plugin pom to use the pre-release, build it, then install the .hpi in your Jenkins and hopefully that ought to fix things.

...and if you do do that, please do report back here with your findings; there's probably more folks watching this issue than there are actively involved...

yavuzd commented 3 years ago

By following comments of @cowtowncoder and @pjdarton, took a plugin build from using both jackson and databind 2.12.1-SNAPSHOT and it seems to fix the issue. All cloud builds started running again. Cheers for the explanations. jackson2-api.hpi.zip

pjdarton commented 3 years ago

@yavuzd Excellent news! Thanks for testing this; it's much appreciated.

TobiasMeyster commented 3 years ago

ist it safe to add jackson2-api.hpi in manage-plugins-advanced tab manually? today i did some updates and jackson was again installed as dependency plugin. There are now a lot dependencies and it is pretty difficult to rollback all the plugins.

kubo44 commented 3 years ago

2.12.1 is out https://github.com/FasterXML/jackson-databind/issues/2995#issuecomment-757685904

timja commented 3 years ago

ist it safe to add jackson2-api.hpi in manage-plugins-advanced tab manually? today i did some updates and jackson was again installed as dependency plugin. There are now a lot dependencies and it is pretty difficult to rollback all the plugins.

Not recommended to use a zip file uploaded by someone, it's been released and a new release candidate of jackson2-api plugin has been automatically generated.

I would suggest using that in the meantime: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.1-rc174.8fd0dd0459f3/jackson2-api-2.12.1-rc174.8fd0dd0459f3.hpi

pjdarton commented 3 years ago

Not recommended to use a zip file uploaded by someone

That's technically correct from a security point of view ... but... the official docker-plugin hpi files are (usually) built and uploaded by me and I'm "someone" too. ...and I trusted github, java, maven & release scripts, my PC's CPU's microcode etc, we all trust the Jenkins update site not to let anyone tinker with what was uploaded and the list goes on... i.e. there's a big chain of trust already so it's not a huge leap of faith to trust one extra person, especially someone who volunteered to help fix this.

I agree that an incrementals build is more "official" than "a zip file uploaded by someone" but saying "not recommended" is a tad harsh IMO.

TobiasMeyster commented 3 years ago

merge is done waiting 4 release :)

TobiasMeyster commented 3 years ago

release is done. tested it. Docker Slave is working again. Thanks alot :)

oleg-nenashev commented 3 years ago

https://github.com/jenkinsci/jackson2-api-plugin/releases/tag/jackson2-api-2.12.1 FTR. Thanks to @cowtowncoder for the upstream release

felipecrs commented 3 years ago

Tested and the Docker plugin is working properly.

phreakadelle commented 3 years ago

Works for us as well. Thanks!

mlechner commented 3 years ago

please reopen

still producing error with Volumes:

ERROR: Build step failed with exception
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `com.github.dockerjava.api.model.Volume` (although at least one Creator exists): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{"Args":["-b","0.0.0.0","-bmanagement=0.0.0.0"],"Config":{"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["/opt/jboss/wildfly/bin/standalone.sh","-b","0.0.0.0","-bmanagement=0.0.0.0"],"Domainname":"","Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","WILDFLY_VERSION=16.0.0.Final","HIBERNATE_VERSION=5.4.27.Final","GEOLATTE_GEOM_VERSION=1.4.0","JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/","JBOSS_HOME=/opt/jboss/wildfly","MVN_REPO=https://repo1.maven."[truncated 2872 chars]; line: 1, column: 3241] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["Mounts"]->java.util.ArrayList[0]->com.github.dockerjava.api.command.InspectContainerResponse$Mount["Destination"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1588)
    at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1213)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1415)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3516)
    at org.jenkinsci.plugins.dockerbuildstep.cmd.CreateContainerCommand.execute(CreateContainerCommand.java:232)
Caused: java.lang.IllegalArgumentException
    at org.jenkinsci.plugins.dockerbuildstep.cmd.CreateContainerCommand.execute(CreateContainerCommand.java:240)
    at org.jenkinsci.plugins.dockerbuildstep.DockerBuilder.perform(DockerBuilder.java:77)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803)
    at hudson.model.Build$BuildExecution.build(Build.java:197)
    at hudson.model.Build$BuildExecution.doRun(Build.java:163)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513)
    at hudson.model.Run.execute(Run.java:1907)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Build step 'Execute Docker command' marked build as failure
TobiasMeyster commented 3 years ago

@mlechner There error you provided is not equal to the ticket error

Original: no String-argument constructor/factory method to deserialize from String value ('/run')

Your Error: cannot deserialize from Object value (no delegate- or property-based Creator)

I got the same Plugin Versions but the original error does still not occur.