Pipeline no longer works with private repositories

[preinking]

I'm no longer able to use minishift with a jenkins pipeline

Version

oc v3.7.1+ab0f056
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth

Server https://192.168.64.68:8443
openshift v3.7.1+a8deba5-34
kubernetes v1.7.6+a08f5eeb62

Steps To Reproduce

minishift config set disk-size 40GB
minishift config set memory 12GB
minishift addon enable registry-route
minishift addon enable admin-user
minishift start

oc new-project cicd --display-name='CI/CD' --description='Namespace for CI/CD pipeline'
oc new-app -n cicd jenkins-persistent -p MEMORY_LIMIT=1Gi -p VOLUME_CAPACITY=32Gi

a) create a pipeline based on a Jenkins file pulled from a private repository b) use a git ssh url and a secret called sshsecret

Result

a) you are not able to see the secret in jenkins b) you cannot configure a secret in jenkins (keeps being deleted automatically)

Workaround

oc login admin 

oc project openshift
oc delete imagestream jenkins
oc create -f jenkins.json

Basically that rolls back the jenkins image to something that I believe should be the default install. Well at least I do not understand why the standaard install provides me with a v3.9 jenkins image. Latest change on that image was done two days ago and since then it seems to fail

cat jenkins.json
{
  "kind": "ImageStream",
  "apiVersion": "v1",
  "metadata": {
    "name": "jenkins",
    "annotations": {
      "openshift.io/display-name": "Jenkins"
    }
  },
  "spec": {
    "tags": [
      {
        "name": "latest",
        "annotations": {
          "openshift.io/display-name": "Jenkins (Latest)",
          "openshift.io/provider-display-name": "Red Hat, Inc.",
          "description": "Provides a Jenkins server on CentOS 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.\n\nWARNING: By selecting this tag, your application will automatically update to use the latest version of Jenkins available on OpenShift, including major versions updates.",
          "iconClass": "icon-jenkins",
          "tags": "jenkins"
        },
        "from": {
          "kind": "ImageStreamTag",
          "name": "2"
        }
      },
      {
        "name": "2",
        "annotations": {
          "openshift.io/display-name": "Jenkins 2.X",
          "description": "Provides a Jenkins v2.x server on CentOS 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.",
          "iconClass": "icon-jenkins",
          "openshift.io/provider-display-name": "Red Hat, Inc.",
          "tags": "jenkins",
          "version": "2.x"
        },
        "from": {
          "kind": "DockerImage",
          "name": "openshift/jenkins-2-centos7:v3.7"
        }
      }
    ]
  }
}

[bparees]

@gabemontero sounds like an issue w/ the 3.9 jenkins image secret syncing logic deleting secrets in openshift? that would be very bad if so.

[gabemontero]

there is nothing in the sync plugin that deletes secrets in openshift @bparees

In fact the only thing the sync plugin has ever deleted in openshift, and this has been this way from the beginning, was delete the build config if the corresponding job was deleted

I suspect what may have been meant was continual deleting of the credential in jenkins mapping to a secret

Also, from what I gather from the description, we are talking about a secret associated with the src section of a BuildConfig ... prior to the new 3.9 feature, those secrets were getting mapped to jenkins credentials.

After double checking the code, I still see what I expected ... we are only deleting jenkins credentials if a secret is deleted, or a build config with a src secret is deleted.

So unless I am simply messing something there, either those delete are happening, or the credential is never getting created in the first place. Though though bc with git src with secret was part of the regression testing for the card.

I'll create a sample test case and sanity check on Monday.

In conjunction, @preinking - would it be possible for you to provide 1) the yaml for your pipeline BC 2) the yaml for your secret, though certainly XXXX out any sensitive data ... I just want to confirm the format 3) the logs from your Jenkins pod when this occurs

[preinking]

Its not the openshift secret that gets deleted but the jenkins side is synced with a blanc. Hence the Jenkins logs and ssh debug show that no certificate is found and no certificate is send to github.

Its indeed the Source Secret in the Jenkins Pipeline Build Strategy that exist in openShift but is not visible in Jenkins. The logs seems to say that something is transmitted and until the latest v3.9 change in Jenkins you could see a credential named 'cicd-sshsecret' being created in Jenkins.

Maybe it does work if you run a full v3.9 however it does not seem right that a standard v3.7 deployment pulls in v3.9 containers. I see that as a separate issue that needs to be addressed.

In effect now al Jenkins Pipeline Build Strategies with private repositories are now broken and its quite possible that another, similar compatibility issue, would occur again in the near future.

I'm now using Jenkins 2 image with digest sha256:f9ed9dec7a0120f6e5781fae35a62db28a49d06b21f96a73d1fb2fb8d73b24ad from 20170911 and then our lab works basically as it did last week.

To recreate:

oc new-project cicd --display-name='CI/CD' --description='Namespace for CI/CD pipeline'
oc new-app -n cicd jenkins-persistent -p MEMORY_LIMIT=1Gi -p VOLUME_CAPACITY=32Gi
oc secrets -n cicd new-sshauth sshsecret --ssh-privatekey=$HOME/.ssh/ids/test-pipeline
oc secrets -n cicd add serviceaccount/builder secrets/sshsecret
oc new-app -n cicd git@github.com:private-repo-name --name test-pipeline --source-secret=sshsecret

[gabemontero]

OK, thanks for the confirmation of my interpretation of the situation @preinking

Next, intermediate checkpoint: for comparison with @preinking 's repro steps, I created a pipeline bc with source/sourceSecret and confirmed what we saw during 3.9 regression testing ... a credential is created in jenkins to match the secret and it stays there.

I'll move onto @preinking next, of course subbing ssh-privatekey and private-repo-name as needed.

On the 3.7/3.9 point/question, a relatively recent change from @bparees made the default image streams in openshift version specific, so that the jenkins image pulled should match whatever version you master is in.

For example, see https://github.com/openshift/origin/blob/master/examples/image-streams/image-streams-rhel7.json#L1025

When we put out new versions of openshift, that line will be getting updated accordingly. In theory minishift should be pulling in this image stream change, but I myself do not know exactly what is happening within minishift.

I'll report back after trying @preinking 's repro recipe.

[gabemontero]

I believe I have reproduced ... it seems specific to ssh based secrets/credentials. My comparison test was username/password. And yep, the fix last week to protect creating credentials with missing information incorrectly failed things if the username was empty ... duh, username is optional for ssh secrets.

Will have fix out by EOB today.