search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.58k stars 788 forks source link

`jx import --no-draft` adding two repos to the Prow configmap `config`: One invalid (no org) and one valid. Pushes to the `master` branch not creating pipelines. #5776

Closed kdevu closed 5 years ago

kdevu commented 5 years ago

Summary

Possibly related to https://github.com/jenkins-x/jx/issues/5456, but I didn't notice this issue with jx import, I noticed it with jx import --no-draft.

I've notice that when I run jx import --no-draft, two repos (instead of one) are added to the Prow ConfigMap config; one invalid (has no org, similar to https://github.com/jenkins-x/jx/issues/5456) and one valid.

The impact of this bug is that I'm seeing that although the project is successfully imported to Jenkins X (after jx import --no-draft). All changes pushed to the master branch do not create any pipelines. Instead, the LightHouse pod logs don't show any pipelines being created and the Tide Pod logs are repeatedly outputting the following.

$ kubectl get logs tide-dc5867fc-57fvz -n

{"error":"tide query (index 0) is invalid: repos[0]: \"/imported_project\" is not of the form \"org/repo\"","level":"error","msg":"Error processing the prow Config YAML","time":"2019-10-11T10:04:34Z"}
{"error":"tide query (index 0) is invalid: repos[0]: \"/imported_project\" is not of the form \"org/repo\"","level":"error","msg":"Error processing the prow Config YAML","time":"2019-10-11T10:04:34Z"}

In order to deploy my changes, I'm force to manually start a pipeline by running jx start pipeline and selecting the valid repo that was added.

I've compared the behaviour between jx import and jx import --no-draft and after testing I've found..

Running kubectl describe configmap config after jx import adds a single valid repo to the Prow configmap config.

$ kubectl describe configmap config -n jx

...

    repos:
    - org/serviceA
 target_url: http://deck-jx-DOMAIN

Running kubectl describe configmap config after jx import --no-draft adds two repos to the Prow configmap config.

$ kubectl describe configmap config -n jx

...

    repos:
    - /serviceA
    - org/serviceA
 target_url: http://deck-jx-DOMAIN

This is a snippet of the describe output. A copy of a full output is below under Actual Behaviour.

Steps to reproduce the behavior

1.jx create quickstart --dry-run --git-provider-url GITLAB_URL (I first encountered the error with a non-quickstart project, but I've also experience the issue when I was testing with quickstart projects)

  1. Select a quickstart project (I've tested node-http, golang-http, golang-http-from-jenkins-x-yml and aspnet-app)
  2. cd new-quickstart-project
  3. jx import --no-draft
  4. Check the updated Prow configmap with kubectl describe configmap config -n jx

Expected behavior

Only one valid repo should be added to the Prow configmap config. Running jx import --no-draft and then pushing a change to the master branch should create a pipeline.

Actual behavior

Output of step 5 is

$ kubectl describe configmap config -n jx

Name:         config
Namespace:    jx
Labels:       <none>
Annotations:  <none>

Data
====
config.yaml:
----
branch-protection:
  orgs:
    MY_ORG:
      repos:
        environment-test-cluster-dev:
          required_status_checks:
            contexts:
            - promotion-build
        environment-test-cluster-production:
          required_status_checks:
            contexts:
            - promotion-build
        environment-test-cluster-staging:
          required_status_checks:
            contexts:
            - promotion-build
        new-quickstart-project:
          required_status_checks:
            contexts:
            - pr-build
  protect-tested-repos: true
  required_status_checks:
    contexts:
    - pr-build
deck:
  spyglass: {}
gerrit: {}
owners_dir_blacklist:
  default: null
  repos: null
plank: {}
pod_namespace: jx
postsubmits:
  /new-quickstart-project:
  - agent: tekton
    branches:
    - master
    context: ""
    name: release
  MY_ORG/environment-test-cluster-dev:
  - agent: tekton
    branches:
    - master
    context: ""
    name: promotion
  MY_ORG/environment-test-cluster-integration:
  - agent: tekton
    branches:
    - master
    context: ""
    name: release
  MY_ORG/environment-test-cluster-production:
  - agent: tekton
    branches:
    - master
    context: ""
    name: promotion
  MY_ORG/environment-test-cluster-staging:
  - agent: tekton
    branches:
    - master
    context: ""
    name: promotion
  MY_ORG/new-quickstart-project:
  - agent: tekton
    branches:
    - master
    context: ""
    name: release
presubmits:
  /new-quickstart-project:
  - agent: tekton
    always_run: true
    context: pr-build
    name: pr-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)
  MY_ORG/environment-test-cluster-dev:
  - agent: tekton
    always_run: true
    context: promotion-build
    name: promotion-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)
  MY_ORG/environment-test-cluster-production:
  - agent: tekton
    always_run: true
    context: promotion-build
    name: promotion-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)
  MY_ORG/environment-test-cluster-staging:
  - agent: tekton
    always_run: true
    context: promotion-build
    name: promotion-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)
  MY_ORG/new-quickstart-project:
  - agent: tekton
    always_run: true
    context: pr-build
    name: pr-build
    rerun_command: /test this
    trigger: (?m)^/test( all| this),?(\s+|$)
prowjob_namespace: jx
push_gateway: {}
sinker: {}
tide:
  context_options:
    from-branch-protection: true
    required-if-present-contexts: null
    skip-unknown-contexts: false
  merge_method:
    /new-quickstart-project: merge
    MY_ORG/environment-test-cluster-dev: merge
    MY_ORG/environment-test-cluster-production: merge
    MY_ORG/environment-test-cluster-staging: merge
    MY_ORG/new-quickstart-project: merge
  queries:
  - labels:
    - approved
    missingLabels:
    - do-not-merge
    - do-not-merge/hold
    - do-not-merge/work-in-progress
    - needs-ok-to-test
    - needs-rebase
    repos:
    - /new-quickstart-project
    - MY_ORG/environment-test-cluster-dev
    - MY_ORG/environment-test-cluster-production
    - MY_ORG/environment-test-cluster-staging
    - MY_ORG/new-quickstart-project
  - labels:
    - updatebot
    missingLabels:
    - do-not-merge
    - do-not-merge/hold
    - do-not-merge/work-in-progress
    - needs-ok-to-test
    - needs-rebase
    repos:
    - /new-quickstart-project
    - MY_ORG/new-quickstart-project
  target_url: http://deck-jx.DOMAIN

Events:  <none>

These are the logs of the LightHouse pod after running jx import --no-draft

$ kubectl logs jenkins-x-lighthouse-64d68b649c-vqxqz -n jx

{"error":"tide query (index 0) is invalid: repos[0]: \"/new-quickstart-project\" is not of the form \"org/repo\"","level":"error","msg":"Error processing the prow Config YAML","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"config-updater","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"config-updater","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"config-updater","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2019-10-11T10:04:34Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2019-10-11T10:04:34Z"}
{"level":"info","msg":"updating the prow plugins configuration","time":"2019-10-11T10:04:34Z"}

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.837
Kubernetes cluster v1.13.7-gke.24
kubectl            v1.16.1
helm client        Client: v2.13.1+g618447c
git                2.17.1
Operating System   Ubuntu 18.04.3 LTS

Jenkins type

Kubernetes cluster

I'm using a GKE cluster. I created it manually using the GCP web GUI.

abayer commented 5 years ago

Looks like @rawlingsj fixed this with #5791 - yay!

kdevu commented 5 years ago

@abayer @rawlingsj

I've tested it with JX version 2.0.861 and it has definitely been fixed.

Unfortunately, I accidentally ran into another bug which has the same symptoms as the above. I ran jx import --no-draft on a local project that didn't have a repository on Gitlab and two repos (one invalid (no org) and another valid) are added to the Prow configmap config. 

$ kubectl describe configmap config -n jx

...
    repos:
    - /serviceA
    - org/serviceA
 target_url: http://deck-jx-DOMAIN

Steps to reproduce

This is how I tested it

  1. Delete the repository on gitlab
  2. Run git remote remove origin
  3. Run jx import --no-draft and let JX create the repository
  4. Check the Prow configmap config with kubectl describe configmap config -n jx

I imagine the only time a user would run into this issue if they were porting a project over from one git provider over to gitlab and letting JX handle the creation of the repository for them.

abayer commented 5 years ago

@KDvu - mind opening a new issue for this case? I'll try to get it fixed this week. Thanks!

kdevu commented 5 years ago

@abayer I created the issue https://github.com/jenkins-x/jx/issues/5806

abayer commented 5 years ago

Thanks!