Request to have .net core agent image

[LinuxSuRen]

Some users who come from the Chinese forum hope to have the .net core agent. You can find the links below:

• https://kubesphere.com.cn/forum/d/1951-podtemplate-asp-net-core
• https://kubesphere.com.cn/forum/d/3384-kubespheredevopsdotnet-core

/kind feature-request /area devops

[sslyc]

use this Dockerfile to build .net core agent:

FROM kubesphere/builder-base:v2.1.0
RUN rpm -Uvh https://packages.microsoft.com/config/centos/7/packages-microsoft-prod.rpm
RUN yum install -y dotnet-sdk-3.1 #此处可以换成其他版本，也可以同时安装多个版本sdk一步到位
RUN dotnet tool install --global dotnet-sonarscanner --version 5.0.4  #(可选)安装dotnet的sonar-scanner
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/sonar-scanner-3.3.0.1492-linux/bin:/root/.nuget/tools:/root/.dotnet/tools  #此处根据需要修改是否将sonar加入到path
CMD ["dotnet", "--version"]

Then Modify CasC settings for jenkins：

          - name: "dotnetcore"
            namespace: "kubesphere-devops-system"
            label: "dotnetcore"
            nodeUsageMode: "EXCLUSIVE"
            idleMinutes: 0
            containers:
            - name: "dotnetcore"
              image: "xxx.com/builder-dotnet:v1.0.0"     #修改成你的镜像仓库和镜像名称
              command: "cat"
              args: ""
              ttyEnabled: true
              resourceRequestCpu: "100m"
              resourceLimitCpu: "4000m"
              resourceRequestMemory: "100Mi"
              resourceLimitMemory: "8192Mi"
              alwaysPullImage: true
            - name: "jnlp"
              image: "jenkins/jnlp-slave:3.27-1"
              command: "jenkins-slave"
              args: "^${computer.jnlpmac} ^${computer.name}"
              resourceRequestCpu: "50m"
              resourceRequestMemory: "400Mi"
              resourceLimitMemory: "1536Mi"
            imagePullSecrets:                            #（可选）此处如果是使用私有仓库，请提前在密钥里准备好docker仓库密钥。
              - name: docker-local
            workspaceVolume:
              emptyDirWorkspaceVolume:
                memory: false
            volumes:
            - hostPathVolume:
                hostPath: "/var/run/docker.sock"
                mountPath: "/var/run/docker.sock"
            - hostPathVolume:                            #将nuget包缓存持久化到hostPath
                hostPath: "jenkins_nuget_cache"
                mountPath: "/root/.nuget"
            - hostPathVolume:                           #（可选）如果安装了sonar，配置此项
                hostPath: "sonar_cache"
                mountPath: "/root/.sonar/cache"
            yamls:                                       #请注意，此处从ConfigMap挂载了Nuget.Config（因为有nuget私有仓库），如不需要可以删除volumnMounts和volumns部分
            - "spec:\r\n  affinity:\r\n    nodeAffinity:\r\n      preferredDuringSchedulingIgnoredDuringExecution:\r\n      - weight: 1\r\n        preference:\r\n          matchExpressions:\r\n          - key: node-role.kubernetes.io/worker\r\n            operator: In\r\n            values:\r\n            - ci\r\n  tolerations:\r\n  - key: \"node.kubernetes.io/ci\"\r\n    operator: \"Exists\"\r\n    effect: \"NoSchedule\"\r\n  - key: \"node.kubernetes.io/ci\"\r\n    operator: \"Exists\"\r\n    effect: \"PreferNoSchedule\"\r\n  containers:\r\n  - name: \"dotnetcore\"\r\n    resources:\r\n      requests:\r\n        ephemeral-storage: \"1Gi\"\r\n      limits:\r\n        ephemeral-storage: \"10Gi\"\r\n    volumeMounts:\r\n    - name: config-volume\r\n      mountPath: /root/.nuget/NuGet/NuGet.Config\r\n      subPath: NuGet.Config\r\n  volumes:\r\n    - name: config-volume\r\n      configMap:\r\n        name: ks-devops-agent\r\n        items:\r\n        - key: NugetSetting\r\n          path: NuGet.Config\r\n  securityContext:\r\n    fsGroup: 1000\r\n "

[sslyc]

Use this Dockerfile to include both .net core 3.1 and .net 5.0 sdks：

FROM kubesphere/builder-base:v2.1.0
RUN rpm -Uvh https://packages.microsoft.com/config/centos/7/packages-microsoft-prod.rpm &&  yum install -y dotnet-sdk-5.0 && yum install -y dotnet-sdk-3.1
RUN dotnet tool install --global dotnet-sonarscanner --version 5.0.4
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/sonar-scanner-3.3.0.1492-linux/bin:/root/.nuget/tools:/root/.dotnet/tools
CMD ["dotnet", "--list-sdks"]

[sslyc]

The image file is too large, I cannot upload it.

[LinuxSuRen]

hi @sslyc , thanks a lot. @Dishone has created PR #24 . It's in the review progress. Please help to review it.

[ysjjovo]

use this Dockerfile to build .net core agent:

FROM kubesphere/builder-base:v2.1.0
RUN rpm -Uvh https://packages.microsoft.com/config/centos/7/packages-microsoft-prod.rpm
RUN yum install -y dotnet-sdk-3.1 #此处可以换成其他版本，也可以同时安装多个版本sdk一步到位
RUN dotnet tool install --global dotnet-sonarscanner --version 5.0.4  #(可选)安装dotnet的sonar-scanner
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/sonar-scanner-3.3.0.1492-linux/bin:/root/.nuget/tools:/root/.dotnet/tools  #此处根据需要修改是否将sonar加入到path
CMD ["dotnet", "--version"]

Then Modify CasC settings for jenkins：

          - name: "dotnetcore"
            namespace: "kubesphere-devops-system"
            label: "dotnetcore"
            nodeUsageMode: "EXCLUSIVE"
            idleMinutes: 0
            containers:
            - name: "dotnetcore"
              image: "xxx.com/builder-dotnet:v1.0.0"     #修改成你的镜像仓库和镜像名称
              command: "cat"
              args: ""
              ttyEnabled: true
              resourceRequestCpu: "100m"
              resourceLimitCpu: "4000m"
              resourceRequestMemory: "100Mi"
              resourceLimitMemory: "8192Mi"
              alwaysPullImage: true
            - name: "jnlp"
              image: "jenkins/jnlp-slave:3.27-1"
              command: "jenkins-slave"
              args: "^${computer.jnlpmac} ^${computer.name}"
              resourceRequestCpu: "50m"
              resourceRequestMemory: "400Mi"
              resourceLimitMemory: "1536Mi"
            imagePullSecrets:                            #（可选）此处如果是使用私有仓库，请提前在密钥里准备好docker仓库密钥。
              - name: docker-local
            workspaceVolume:
              emptyDirWorkspaceVolume:
                memory: false
            volumes:
            - hostPathVolume:
                hostPath: "/var/run/docker.sock"
                mountPath: "/var/run/docker.sock"
            - hostPathVolume:                            #将nuget包缓存持久化到hostPath
                hostPath: "jenkins_nuget_cache"
                mountPath: "/root/.nuget"
            - hostPathVolume:                           #（可选）如果安装了sonar，配置此项
                hostPath: "sonar_cache"
                mountPath: "/root/.sonar/cache"
            yamls:                                       #请注意，此处从ConfigMap挂载了Nuget.Config（因为有nuget私有仓库），如不需要可以删除volumnMounts和volumns部分
            - "spec:\r\n  affinity:\r\n    nodeAffinity:\r\n      preferredDuringSchedulingIgnoredDuringExecution:\r\n      - weight: 1\r\n        preference:\r\n          matchExpressions:\r\n          - key: node-role.kubernetes.io/worker\r\n            operator: In\r\n            values:\r\n            - ci\r\n  tolerations:\r\n  - key: \"node.kubernetes.io/ci\"\r\n    operator: \"Exists\"\r\n    effect: \"NoSchedule\"\r\n  - key: \"node.kubernetes.io/ci\"\r\n    operator: \"Exists\"\r\n    effect: \"PreferNoSchedule\"\r\n  containers:\r\n  - name: \"dotnetcore\"\r\n    resources:\r\n      requests:\r\n        ephemeral-storage: \"1Gi\"\r\n      limits:\r\n        ephemeral-storage: \"10Gi\"\r\n    volumeMounts:\r\n    - name: config-volume\r\n      mountPath: /root/.nuget/NuGet/NuGet.Config\r\n      subPath: NuGet.Config\r\n  volumes:\r\n    - name: config-volume\r\n      configMap:\r\n        name: ks-devops-agent\r\n        items:\r\n        - key: NugetSetting\r\n          path: NuGet.Config\r\n  securityContext:\r\n    fsGroup: 1000\r\n "

I folollow the steps,Got error:

Started by user admin
Replayed #5
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to http://xxx/root/kubesphere-pipeline-demo-dotnet.git
 > git config remote.origin.url http://xxx/root/kubesphere-pipeline-demo-dotnet.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 34d62cb3f0d24777a27066693007ede7a1d483d6
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘dotnetcore-h594n’ is offline

[LinuxSuRen]

hi @ysjjovo , the following output is not error log.

Still waiting to schedule task
‘dotnetcore-h594n’ is offline

Please check the output of pod dotnetcore-h594n

[ysjjovo]

hi @ysjjovo , the following output is not error log.

Still waiting to schedule task
‘dotnetcore-h594n’ is offline

Please check the output of pod dotnetcore-h594n

You're right,Thanks!It's missing some configuration,I'll check it out.

  Warning  FailedMount  <invalid> (x8 over <invalid>)  kubelet, node4     MountVolume.SetUp failed for volume "config-volume" : configmap references non-existent config key: NugetSetting

[sslyc]

Hi @ysjjovo Yes, the CasC file uses a Nuget config file by default, which is described in yamls field. And I mentioned that you should set a ConfigMap key by the comments(which is not in English). You may follow these steps:

1. Edit the ConfigMap where is named 'ks-devops-agent', in namespace 'kubesphere-devops-system'.
2. Add a key 'NugetSetting', and set the value as below, and save it.

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <packageSources>
       <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
     </packageSources>
   </configuration>

3. Restart your agent.

[ysjjovo]

hi @ysjjovo , the following output is not error log.

Still waiting to schedule task
‘dotnetcore-h594n’ is offline

Please check the output of pod dotnetcore-h594n

Now jenkins stuck in running state for 19 mins.

[root@master ~]# kubectl get po -A|grep dotnet
kubesphere-devops-system                        dotnetcore-mzw4k                                                  2/2     Running            0          19m

and last jenkins log is

and container 'dotnetcore' doesn't have any logs

[root@master ~]# kubectl logs -n kubesphere-devops-system dotnetcore-mzw4k dotnetcore
[root@master ~]#

[ysjjovo]

Hi @ysjjovo Yes, the CasC file uses a Nuget config file by default, which is described in yamls field. And I mentioned that you should set a ConfigMap key by the comments(which is not in English). You may follow these steps:

1. Edit the ConfigMap where is named 'ks-devops-agent', in namespace 'kubesphere-devops-system'.
2. Add a key 'NugetSetting', and set the value as below, and save it.

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
    <packageSources>
      <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    </packageSources>
   </configuration>

3. Restart your agent.

Sorry for not awaring your tips! Now jenkins stuck in running state for 19 mins.

[root@master ~]# kubectl get po -A|grep dotnet
kubesphere-devops-system                        dotnetcore-mzw4k                                                  2/2     Running            0          19m

and last jenkins log is

Running on dotnetcore-mzw4k in /home/jenkins/agent/workspace/demo-devopsrdxdf_dotnet1_master

and container 'dotnetcore' doesn't have any logs

[root@master ~]# kubectl logs -n kubesphere-devops-system dotnetcore-mzw4k dotnetcore
[root@master ~]#

[LinuxSuRen]

and container 'dotnetcore' doesn't have any logs

It's true. It does not have any log output. Please check the log output from the container of jnlp.

[ysjjovo]

and container 'dotnetcore' doesn't have any logs

It's true. It does not have any log output. Please check the log output from the container of jnlp.

logs of container 'jnlp' seems normal.

[root@master ~]# kubectl logs -n kubesphere-devops-system dotnetcore-mzw4k jplp
error: container jplp is not valid for pod dotnetcore-mzw4k
[root@master ~]# kubectl logs -n kubesphere-devops-system dotnetcore-mzw4k
error: a container name must be specified for pod dotnetcore-mzw4k, choose one of: [jnlp dotnetcore]
[root@master ~]# kubectl logs -n kubesphere-devops-system dotnetcore-mzw4k jnlp
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: dotnetcore-mzw4k
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jun 21, 2021 7:59:29 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.27
Both error and output logs will be printed to /home/jenkins/agent/remoting
Jun 21, 2021 7:59:29 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://ks-jenkins.kubesphere-devops-system:80/]
Jun 21, 2021 7:59:29 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jun 21, 2021 7:59:29 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: ks-jenkins-agent.kubesphere-devops-system
  Agent port:    50000
  Identity:      4e:60:d0:43:af:54:9b:c7:5a:e2:c1:b3:b7:3e:33:a5
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to ks-jenkins-agent.kubesphere-devops-system:50000
Jun 21, 2021 7:59:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jun 21, 2021 7:59:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 4e:60:d0:43:af:54:9b:c7:5a:e2:c1:b3:b7:3e:33:a5
Jun 21, 2021 7:59:34 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

[sslyc]

Hi @ysjjovo How did you use this agent? What did your jenkins file look like? Here's my sample, and I used it frequently, built lots of dotnet-based images successfully since last year:

pipeline {
  agent {
    node {
      label 'dotnetcore'                     //here is your agent name, and the same below
    }

  }
  stages {

    stage('pulling code') {
      steps {
        //use your git repository setting
        git(url: 'https://xxx.xxx.com/xxx.git', credentialsId: 'git-credential', branch: 'master', changelog: true, poll: false)
      }
    }
    stage('code analysis') {
      when {
        //from parameters
        environment name: 'WithSonar', value: 'true'
      }
      steps {
        withSonarQubeEnv('sonar') {
          container('dotnetcore') {
            sh 'dotnet sonarscanner begin /k:"projectname" /n:projectname'
            sh 'dotnet build \'src/xxx/xxx.sln\''
            sh 'dotnet sonarscanner end'
          }

        }

        waitForQualityGate 'true'
      }
    }
    stage('build packages') {
      steps {
        container('dotnetcore') {
          script {
            //Projects are from parameter as multi-line string
            def projects = Projects.replace('\r\n', '\n').split('\n');
            projects.each {
              //you should replace the path, to fit your projects
              sh "dotnet publish -c Release 'src/${it}/${it}.csproj' -o 'src/${it}/bin/publish/'"
            }
          }

        }

      }
    }
    stage('imaging and tagging') {
      steps {
        container('dotnetcore') {
          script {
            def projects = Projects.replace('\r\n', '\n').split('\n');
            projects.each {
              def img = it.toLowerCase().replace('.','').replace('_','');
              def pub = "cat > src/${it}/Dockerfile << EOF\n" + "FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim\n" + "WORKDIR /app\n" + "EXPOSE 80\n" + "COPY src/${it}/bin/publish/. .\n" + "ENTRYPOINT [ \"dotnet\",\"${it}.dll\" ]\n" + "EOF";

              sh pub
              sh "docker build -f src/${it}/Dockerfile -t ${img} ."
            }
          }

        }

      }
    }
    stage('push images') {
      steps {
        container('dotnetcore') {
          script {
            def projects = Projects.replace('\r\n', '\n').split('\n');
            projects.each {
              def img = it.toLowerCase().replace('.','').replace('_','');
              sh """docker tag ${img} xxx.xxx.com/${img}
              docker login -u xxx -p xxx xxx.xxx.com
              docker push xxx.xxx.com/${img}"""
            }
          }

        }

      }
    }
  }
}

[sslyc]

It works well for me

[sslyc]

More to mention, if you wanna use sonar to analyze your code, you should have your sonar setup in kubesphere system. Follow the official document at here.

[ysjjovo]

More to mention, if you wanna use sonar to analyze your code, you should have your sonar setup in kubesphere system. Follow the official document at here.

This is my jenkinsfile,I does not use sonar scanner.

pipeline {
    agent {
      node {
        label 'dotnetcore'
      }
    }
    environment {
        VERSION = '%{version}'
        REGISTRY = '%{registry}'
        REPO_PREFIX = '%{repoPrefix}'
        SONAR_CREDENTIAL_ID = 'sonar-id'
        DOCKER_CREDENTIAL_ID = 'docker-id'
        KUBECONFIG_CREDENTIAL_ID = 'kubeconfig-id'
    }

    stages {
        stage ('checkout scm') {
            steps {
                checkout scm
            }
        }
        stage('dotnet build & publish') {
            steps {
                container ('dotnetcore') {
                    sh 'dotnet restore'
                    sh 'dotnet publish -c Release -o publish --no-restore'
                }
            }
        }
        stage ('docker build & push') {
            steps {
                container ('dotnetcore') {
                    sh 'docker build -f Dockerfile -t $REPO_PREFIX/$VERSION:$BRANCH_NAME-$BUILD_NUMBER .'
                    withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push  $REPO_PREFIX/$VERSION:$BRANCH_NAME-$BUILD_NUMBER'
                    }
                }
            }
        }
        stage('deploy') {
          steps {
            kubernetesDeploy(configs: 'deploy.yaml', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID")
          }
        }
    }
}

[sslyc]

You mean, it did not work, and the pipeline didn't start? The agent should be running when you start a pipeline, but isn't when you are not running a pipeline. If you saw a agent pod stuck, keeping running without any output. I suggest you kill the pod manually. It will run automatically the moment you start a dotnet pipeline. Try to use command kubectl delete po your-agent-pod-name -n kubesphere-devops-system.

[sslyc]

If the error continues, you'd better check your image. You may run the agent locally by typing docker run your-dotnet-agent:tag. The container will run and exit immediately. If it is stuck, that means the image has got some mistake. Check your Dockfile and look into 'CMD' section. For more information, you could run docker run -it your-dotnet-agent:tag /bin/bash, to enter the shell of the container. Try commands like dotnet --version, dotnet --list-sdks.

[ysjjovo]

More to mention, if you wanna use sonar to analyze your code, you should have your sonar setup in kubesphere system. Follow the official document at here.

Thanks!After I delete the long time running pod and rerun the pipline, it succeed! Plus，cannot exist environment virable 'VERSION',otherwise it will fail with error

NuGet.targets(128,5): error : 'xxxxxx' is not a valid version string. (Parameter 'value')

[sslyc]

Last thing to remember: two ConfigMaps named "jenkins-casc-config" and "ks-devops-agent" would reset to default when you update kubesphere to a new version, or modify the CRD "ClusterConfiguration" causing ks-installer to work, or use kk to install new features/add new node to k8s cluster. If you append to meet error like "there's no label 'dotnetcore' in Jenkins", your should check these configs out.

[LinuxSuRen]

see also https://github.com/kubesphere/devops-agent/issues/75 /close

[ks-ci-bot]

@LinuxSuRen: Closing this issue.

In response to [this](https://github.com/kubesphere/devops-agent/issues/14#issuecomment-1203579377): >see also https://github.com/kubesphere/devops-agent/issues/75 >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.