[Discussion] Getting `jenkins` communication issue with docker daemon

[afzal442]

Hi @devopsjourney1! I have hard finding about setting up jenkins pipeline with docker agent. Since I have followed these steps from the docs here, but when I try to follow the steps from ur video, I am failing to create a label docker agent.

Is it necessary to install jenkins through docker to establish connection b/w docker host and jenkins server?

When I skip the docker installation as I have done through kubernetes and follow the docker socket connection like this docker network create jenkins and docker run -d --restart=always -p 127.0.0.1:2376:2375 --network jenkins -v /var/run/docker.sock:/var/run/docker.sock alpine/socat tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock, I still fail to make the connection successful. Any thoughts here plz?

[afzal442]

While during my investigation, I created a pipeline, but refused to build due to a failure. Here is my below pipeline script

 pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo "Building.."
            }
        }
        stage('Test') {
            steps {
                echo "Testing.."
                sh '''
                git version
                git clone https://github.com/devopsjourney1/jenkins-101
                cd jenkins-101
                cat Jenkinsfile
                '''
                sh "docker version"
            }
        }
        stage('Deliver') {
            steps {
                echo 'Deliver....'
                sh '''
                echo "doing delivery stuff.."
                '''
            }
        }
    }
}

Build error as

[Pipeline]
+ docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Fri May  5 15:36:11 2017
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[afzal442]

hi @devopsjourney1! It's been quite a while now. Maybe you were busy with ur works. Could take a look here and help me debug this issue? TYIA!

[RogerTaylor130]

run

sudo docker version

and attach the output

[afzal442]

hey @RogerTaylor130 here what I found

/var/jenkins_home/workspace/respip@tmp/durable-01d1605b/script.sh: 1: sudo: not found

[RogerTaylor130]

Hi, @afzal442

Can you post all the Jenkins logs since you got /var/jenkins_home/workspace/respip@tmp/durable-01d1605b/script.sh: 1: sudo: not found

You need to know,

• sudo not found is the sudo on your Linux machine has not been installed.
• Which node got selected since you use agent any
• As you figure out which node is selected then run sudo docker version on that node via ssh.

[afzal442]

To make it simpler, plz consider this pipeline

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo "Building.."
            }
        }
        stage('Test') {
            steps {
                echo "Testing.."
                sh "sudo docker version"
            }
        }
        stage('Deliver') {
            steps {
                echo 'Deliver....'
                sh '''
                echo "doing delivery stuff.."
                '''
            }
        }
    }
}

LMK if u r able to run the abv pipeline.

I have sudo installed in my Linux m/c by the way.

Below is the full logs

Started by user [Afzal](http://localhost:8000/user/afzal442)
[Pipeline] Start of Pipeline
[Pipeline] node
Running on [Jenkins](http://localhost:8000/manage/computer/(built-in)/) in /var/jenkins_home/workspace/respip
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] echo
Building..
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Testing..
[Pipeline] sh
+ sudo docker version
/var/jenkins_home/workspace/respip@tmp/durable-b2227a00/script.sh: 1: sudo: not found
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deliver)
Stage "Deliver" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE

Plz suggest how to debug this. Thanks

[RogerTaylor130]

Well. The pipeline looks fine to me but I did not test it.

Looks like your pipeline is running on the built-in node which is your localhost I think.

Try to run sudo docker version on your localhost. It equals what you did in thetest stage` in the pipeline.

[afzal442]

Thanks for quick response.

Looks like your pipeline is running on the built-in node

Try to run sudo docker version on your localhost.

Sorry! How?

But if I ran docker version in the pipeline, it would show up like this Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[RogerTaylor130]

SSH into your Jenkins machine and run sudo docker version.

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? This is probably because docker doesn't provide normal user permission to run the docker commands. You need the root user to run docker version

[afzal442]

I am just looping around;

$ sudo docker version
/bin/sh: 1: sudo: not found

Here you go, I hope you have read the comment abv. Thanks

Since I have followed these steps from the docs here, but when I try to follow the steps from ur video, I am failing to create a label docker agent.

[RogerTaylor130]

Not sure about the k8s installation. Just let u know your pipeline is called the mast node to run sudo docker version locally. Say your Jenkins machine is the docker container in the pod, your container does not have sudo installed for sure from the Jenkins log.

[afzal442]

Ah I see. I don't think sudo is installed in any container in any pod unless root access. By the way, what's abt node agent? Can I create new node agent against docker? I can't find help from docs as well. I am so much exhausted that I have watched almost every official videos abt docker but no chance. Any suggestions?

Edit: What would u say abt it https://www.jenkins.io/doc/book/pipeline/docker/ ?

[RogerTaylor130]

I am not sure how far you have gone with docker, k8s, jenkins... But jenkins does not have very close connection with docker and k8s. I suggest go with jenkins and docker first then k8s. Study a little bit deep since you don't know about some basic concepts of jenkins like node/agent... Well... I am not good as well so it's just my personal advice. Good lucky.

[shadycuz]

@afzal442 If you need docker in your K8's pod you will either have to run the pod with elevated privileges AND pass in the host machine's docker socket.

OR

You can install docker inside your container, which can be very challenging to do but is the better approach from a security perspective.

What you are going to have to do is create a customer jenkins docker container image that has the docker daemon installed. Once that is done you can look up instructions for creating on demand jenkins nodes/agents/runners using Kubernetes.

[afzal442]

Thanks @RogerTaylor130 for digging into it alongside me. I am flattered by your quick responses and suggestions. Again for your advice. 😄

[afzal442]

Hi @shadycuz ! I am glad you came into this discussion. Your suggestions/approaches look promising to me. I can't thank you enough. I will try those approaches whichever work for me. And try to sync with you. Thanks.

[afzal442]

Hi @shadycuz ! I tried to figure out on how to run the pod with elevated privileges AND pass in the host machine's docker socket. Any clue! My installations follows this https://www.jenkins.io/doc/book/installing/kubernetes/#install-jenkins-with-yaml-files

One quick question how do you run this below pipeline after installing JenkinsCI

pipeline {
    agent {
        docker { image 'node:16.13.1-alpine' }
    }
    stages {
        stage('Test') {
            steps {
                sh 'node --version'
            }
        }
    }
}

Does that work for you? This says Meaning that a user can define the tools required for their Pipeline, without having to manually configure agents.

What possibly I could try after installing?

Thanks!

[shadycuz]

@afzal442 I worked on it this weekend. It's very hard. I have done it a number of different ways and its always a pain. If you just wanted to build a docker image and push then its very easy. But what you want to do happens to be extremely hard. Usually I run production Jenkins outside of container and develop my pipelines on container Jenkins.

But what you want to do is completely reasonable... it's just very difficult to implement. I have made my own community post about this here.

[afzal442]

Thanks much Levi getting around that and supporting me on the fly! BTW, It's been pain for me as well. 😖

If you just wanted to build a docker image and push then its very easy.

Absolutely. So do I.

Usually I run production Jenkins outside of container ....

Do you mean Docker container here? If so, I can understand.

But what you want to do is completely reasonable

So what did I next is I tried to create agent node by following this video which I already stated apparently earlier when I was struggling hard, but unfortunately, still can't make it after following the docs entirely as well. There are several errors I was encountering like establishing connection error iteratingly.
but when I first found this video https://youtu.be/ymI02j-hqpU aligned with the docs, everything looked easy at first, but failed. Do you know? I think they have created Jenkins instances using Docker installation, but didn't mention before the video, but next everything is easy. BTW Kubernetes installation is easier and takes less over-head. Also recommended for production. WDYT? I think I will try with Docker Desktop next time as created by @devopsjourney1 in his video.

I have made my own community post about this

So glad to know that. Let me know when have any update. 😄

[shadycuz]

If you just wanted to build a docker image and push then its very easy.

Absolutely. So do I.

Yeah that is easy to do. But the pipeline you show above:

pipeline {
    agent {
        docker { image 'node:16.13.1-alpine' }
    }
    stages {
        stage('Test') {
            steps {
                sh 'node --version'
            }
        }
    }
}

This actually involves sending the build workspace into the Docker container and this will not work very easily if the workspace is already inside of a container.

Usually I run production Jenkins outside of container ....

Do you mean Docker container here? If so, I can understand.

Yes, usually I run the Jenkins agents on bare VM's. This means I don't have any docker in docker issues.

But what you want to do is completely reasonable

So what did I next is I tried to create agent node by following this video which I already stated apparently earlier when I was struggling hard, but unfortunately, still can't make it after following the docs entirely as well. There are several errors I was encountering like establishing connection error iteratingly. but when I first found this video https://youtu.be/ymI02j-hqpU aligned with the docs, everything looked easy at first, but failed. Do you know? I think they have created Jenkins instances using Docker installation, but didn't mention before the video, but next everything is easy. BTW Kubernetes installation is easier and takes less over-head. Also recommended for production. WDYT? I think I will try with Docker Desktop next time as created by @devopsjourney1 in his video.

The guide you linked is a generic guide about creating a single agent. You need a guide for creating Jenkins agents on Kubernetes. You need to configure this plugin. This will involve some extra work, like making sure you Jenkins Controller as access to the Kubernetes API via a Service Account.

It's worth it to understand and have dynamic agents configured first and then move on to Docker. But like I said earlier, you will only be able to build and push docker images if the agent running the pipeline is inside of a container.

[afzal442]

You need a guide for creating Jenkins agents on Kubernetes. You need to configure this plugin. This will involve some extra work, like making sure you Jenkins Controller as access to the Kubernetes API via a Service Account.

Yes! that's perfect. I will come to that once I get rid of that part of docker socket issue. Actually, I wanted to remove the barrier b/w Jenkins kubernetes installation and Docker node agent..

you will only be able to build and push docker images if the agent running the pipeline is inside of a container.

Got that. That means in my case the agent running the pipeline is outside of a container becaz of not having my Docker socket connection. Right?

this will not work very easily if the workspace is already inside of a container.

Sorry if you can be explicit here a bit more. Thanks

[shadycuz]

Are you doing this for fun? Or for production? Because if you are doing this for fun you can skip the agent thing for now.

You need a guide for creating Jenkins agents on Kubernetes. You need to configure this plugin. This will involve some extra work, like making sure you Jenkins Controller as access to the Kubernetes API via a Service Account.

Yes! that's perfect. I will come to that once I get rid of that part of docker socket issue. Actually, I wanted to remove the barrier b/w Jenkins kubernetes installation and Docker node agent..

You need to have the agents running first. Once the agents are working, then you can try and figure out how to get docker working on them. Remember you have 3 options for getting docker running

1. You pass the docker socket from host to the agent container as a bind mount volume.
2. You install docker in your agent container.
3. You connect from the agent container to a remote docker daemon.

Also... The first option might not even be possible depending on how you run your K8 cluster. For example if you are using digital ocean then you dont get access to the k8 workers, you can't install docker on the host to pass it to pods anyways.

You might also run into challenges with the second option. You might want to run an existing Jenkins agent container without modifying it. That leaves the third option which is what other CICD services like circleci and gitlab are using.

you will only be able to build and push docker images if the agent running the pipeline is inside of a container.

Got that. That means in my case the agent running the pipeline is outside of a container becaz of not having my Docker socket connection. Right?

You will still be able to run Jenkins Pipelines, you just wont be able to run on docker agents. You will still be able to build and push docker images if you wanted.

this will not work very easily if the workspace is already inside of a container.

Sorry if you can be explicit here a bit more. Thanks

When you try to run on a docker agent like this:

agent {
        docker { image 'node:16.13.1-alpine' }
    }

Jenkins takes the current workspace and trys to bind mount it inside the container. So if the Jenkins agent that is running the pipeline, is also in a container then it wont work. The workspace will be empty instead of having anyfiles or anything you might want to checkout.

To be fair, this might work for the simple pipeline example you gave but if you have a more realistic pipeline like...

node('dynamic-k8-pod') {
  checkout('my-python-app')

  docker.image('python-black:latest').inside {
        stage('Format Code') {
            sh('black --some args')
        }
  }

  docker.image('python-lint:latest').inside {
        stage('Lint Code') {
            sh('pylint--some args')
        }
  }
}

This pipeline ^ would not work correctly if the agent running the job is already in a container.

[afzal442]

Thanks again. Now it gives much more sense. I am using Linode cluster, maybe the first option goes out due to that. For 2nd option, I have Jenkins installed with kubernetes option, so I can't install Docker inside the container Jenkins-lts running inside the pod per se which is not possible. I will try out with 3rd option left. You connect from the agent container to a remote docker daemon. Can you give me any article/docs link to follow? Thanks

In order to run that pipeline, you mentioned abv I have to create Docker node agent to establish a connection b/w the Jenkins controller and container agent to build the workspace. Right? When we say outside of a container, it means ssh connection is needed as here from outside.

[shadycuz]

@afzal442 You will need to use the Kubernetes plugin to spin up dynamic Jenkins agents on demand. Those agents will run in their own pod and that pod needs a docker sidecar container to provide the remote docker daemon. You can use this guide as a reference, just replace spacelift with Jenkins agent.

[afzal442]

hmm! That will be a nice extensible work. I will try on top of k8s plugin and see if it works for me. Thanks. 😄