Pull request generator - Githubissues

This is a simple Pull Request generator that can query from upstream APIs.

From the example generator...

This shows a single PR detected, detecting a new PR, and reacting to a closed PR.

{"level":"info","ts":"2023-01-27T14:55:07.826Z","msg":"gitops set loaded","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3"}
{"level":"info","ts":"2023-01-27T14:55:07.827Z","msg":"generating params","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","repo":"bigkevmcd/go-demo"}
{"level":"info","ts":"2023-01-27T14:55:07.927Z","msg":"querying pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","repo":"bigkevmcd/go-demo","driver":"github","serverURL":""}
{"level":"info","ts":"2023-01-27T14:55:08.442Z","msg":"queried pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","repo":"bigkevmcd/go-demo","count":1}
{"level":"info","ts":"2023-01-27T14:55:08.443Z","msg":"rendered templates","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","resourceCount":2}
{"level":"info","ts":"2023-01-27T14:55:08.451Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","objNamespace":"default","objName":"pr-10-gitrepository","kind":"GitRepository"}
{"level":"info","ts":"2023-01-27T14:55:08.464Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"9449d5e6-674f-40b2-a8b5-b281cce705c3","objNamespace":"default","objName":"pr-10-demo","kind":"Kustomization"}
{"level":"info","ts":"2023-01-27T15:00:08.272Z","msg":"gitops set loaded","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f"}
{"level":"info","ts":"2023-01-27T15:00:08.272Z","msg":"generating params","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","repo":"bigkevmcd/go-demo"}
{"level":"info","ts":"2023-01-27T15:00:08.272Z","msg":"querying pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","repo":"bigkevmcd/go-demo","driver":"github","serverURL":""}
{"level":"info","ts":"2023-01-27T15:00:08.898Z","msg":"queried pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","repo":"bigkevmcd/go-demo","count":2}
{"level":"info","ts":"2023-01-27T15:00:08.899Z","msg":"rendered templates","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","resourceCount":4}
{"level":"info","ts":"2023-01-27T15:00:08.899Z","msg":"creating new resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","objNamespace":"default","objName":"pr-13-gitrepository","kind":"GitRepository"}
{"level":"info","ts":"2023-01-27T15:00:08.909Z","msg":"creating new resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","objNamespace":"default","objName":"pr-13-demo","kind":"Kustomization"}
{"level":"info","ts":"2023-01-27T15:00:08.923Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","objNamespace":"default","objName":"pr-10-gitrepository","kind":"GitRepository"}
{"level":"info","ts":"2023-01-27T15:00:08.932Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"62101a37-d4a4-4d50-baa9-c6e5cdbf398f","objNamespace":"default","objName":"pr-10-demo","kind":"Kustomization"}
{"level":"info","ts":"2023-01-27T15:05:08.738Z","msg":"gitops set loaded","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9"}
{"level":"info","ts":"2023-01-27T15:05:08.738Z","msg":"generating params","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","repo":"bigkevmcd/go-demo"}
{"level":"info","ts":"2023-01-27T15:05:08.739Z","msg":"querying pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","repo":"bigkevmcd/go-demo","driver":"github","serverURL":""}
{"level":"info","ts":"2023-01-27T15:05:09.156Z","msg":"queried pull requests","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","repo":"bigkevmcd/go-demo","count":1}
{"level":"info","ts":"2023-01-27T15:05:09.157Z","msg":"rendered templates","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","resourceCount":2}
{"level":"info","ts":"2023-01-27T15:05:09.166Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","objNamespace":"default","objName":"pr-10-gitrepository","kind":"GitRepository"}
{"level":"info","ts":"2023-01-27T15:05:09.181Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","objNamespace":"default","objName":"pr-10-demo","kind":"Kustomization"}
{"level":"info","ts":"2023-01-27T15:05:09.188Z","msg":"deleting resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","objNamespace":"default","objName":"pr-13-demo","kind":"Kustomization"}
{"level":"info","ts":"2023-01-27T15:05:09.196Z","msg":"deleting resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"pull-requests-sample","namespace":"default"},"namespace":"default","name":"pull-requests-sample","reconcileID":"acecfd54-7f64-4117-9493-a281271b2df9","objNamespace":"default","objName":"pr-13-gitrepository","kind":"GitRepository"}

There's more that can be done to this...

Additional fields from the PR (fork!)
Update the enum on the driver to reflect all the variants supported by go-scm
Add support for requesting more than just the first 20 PRs

Was having trouble getting my little configmap updating

apiVersion: templates.weave.works/v1alpha1
kind: GitOpsSet
metadata:
  name: gitopsset-configmaps
  namespace: default
spec:
  generators:
    - pullRequests:
        interval: 1m
        driver: github
        repo: foot/wge-dev
        serverURL: https://github.com
        secretRef:
          name: my-github-secret
  templates:
    - content:
        kind: ConfigMap
        apiVersion: v1
        metadata:
          name: "pr-{{ .element.number }}-info-configmap"
          namespace: default
        spec:
          data:
            foo: bar
            bar: foo
            number: "{{ .element.number }}"
            dump: |
              {{ .element.number }}

Not currently re-creating if deleted

I deleted the configmap to see if it would re-gen but we get

❯ kubectl get gitopssets
NAME                   AGE   READY   STATUS
gitopsset-configmaps   37m   False   failed to load existing Resource: configmaps "pr-1-info-configmap" not found

We should re-gen in this case I guess?

~Not updating~

Update: ConfigMap's don't have a spec 🤦

Not sure what is happening exactly yet.

❯ kubectl create configmap pr-1-info-configmap
configmap/pr-1-info-configmap created

❯ kubectl get gitopssets
NAME                   AGE   READY   STATUS
gitopsset-configmaps   38m   True    1 resources created

❯ kubectl get configmap pr-1-info-configmap -oyaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: "2023-01-31T09:47:35Z"
  name: pr-1-info-configmap
  namespace: default
  resourceVersion: "13604"
  uid: fed849a4-1cfb-48cb-b8ca-12126efba7dd

On debugging output

  templates:
    - content:
        kind: ConfigMap
        apiVersion: v1
        metadata:
          name: "pr-{{ .element.number }}-info-configmap"
          namespace: default
        data:
          foo: bar
          number: "{{ .element.number }}"
          dump: |
            {{ .element | toRawJson }}

Gives a

gitopsset-configmaps   47m   False   failed to render template params for set gitopsset-configmaps: failed to parse rendered template: json: offset 38: invalid character 'b' after object key:value pair

But I'm not sure whats going on exactly, don't know if we can help here, the output would quite large to put in the status / logs to aid debugging.

gitopsset-configmaps 47m False failed to render template params for set gitopsset-configmaps: failed to parse rendered template: json: offset 38: invalid character 'b' after object key:value pair
But I'm not sure whats going on exactly, don't know if we can help here, the output would quite large to put in the status / logs to aid debugging.

I'd

On debugging output

  templates:
    - content:
        kind: ConfigMap
        apiVersion: v1
        metadata:
          name: "pr-{{ .element.number }}-info-configmap"
          namespace: default
        data:
          foo: bar
          number: "{{ .element.number }}"
          dump: |
            {{ .element | toRawJson }}

Gives a

gitopsset-configmaps   47m   False   failed to render template params for set gitopsset-configmaps: failed to parse rendered template: json: offset 38: invalid character 'b' after object key:value pair

Does {{ .element | toRawJson | b64enc }} work?

Does {{ .element | toRawJson | b64enc }} work?

Yes it does!

❯ echo eyJicmFuY2giOiJmb290LXBhdGNoLTEiLCJjbG9uZV9zc2hfdXJsIjoiZ2l0QGdpdGh1Yi5jb206Zm9vdC93Z2UtZGV2LmdpdCIsImNsb25lX3VybCI6Imh0dHBzOi8vZ2l0aHViLmNvbS9mb290L3dnZS1kZXYuZ2l0IiwiaGVhZF9zaGEiOiI5MjU4NzllMDg5NzU1ZmUxYTAyMGYxMGU0YzkyYjExYWU2ZjY3NzI5IiwibnVtYmVyIjoiMSJ9 | base64 -d {"branch":"foot-patch-1","clone_ssh_url":"git@github.com:foot/wge-dev.git","clone_url":"https://github.com/foot/wge-dev.git","head_sha":"925879e089755fe1a020f10e4c92b11ae6f67729","number":"1"}%

Yes it does!

❯ echo eyJicmFuY2giOiJmb290LXBhdGNoLTEiLCJjbG9uZV9zc2hfdXJsIjoiZ2l0QGdpdGh1Yi5jb206Zm9vdC93Z2UtZGV2LmdpdCIsImNsb25lX3VybCI6Imh0dHBzOi8vZ2l0aHViLmNvbS9mb290L3dnZS1kZXYuZ2l0IiwiaGVhZF9zaGEiOiI5MjU4NzllMDg5NzU1ZmUxYTAyMGYxMGU0YzkyYjExYWU2ZjY3NzI5IiwibnVtYmVyIjoiMSJ9 | base64 -d {"branch":"foot-patch-1","clone_ssh_url":"git@github.com:foot/wge-dev.git","clone_url":"https://github.com/foot/wge-dev.git","head_sha":"925879e089755fe1a020f10e4c92b11ae6f67729","number":"1"}%

I'm guessing the b is from "branch" :-)

I'm seeing a funny thing where my configmap is not being updated. Added some more logging

[manager] {"level":"info","ts":"2023-01-31T11:09:24.912Z","msg":"updating existing resource if necessary","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"gitopsset-configmaps","namespace":"default"},"namespace":"default","name":"gitopsset-configmaps","reconcileID":"5281bf7c-727b-4ea2-b413-994d58576477","objNamespace":"default","objName":"pr-1-info-configmap","kind":"ConfigMap"}

[manager] {"level":"info","ts":"2023-01-31T11:09:24.913Z","msg":"updating existing resource","controller":"gitopsset","controllerGroup":"templates.weave.works","controllerKind":"GitOpsSet","GitOpsSet":{"name":"gitopsset-configmaps","namespace":"default"},"namespace":"default","name":"gitopsset-configmaps","reconcileID":"5281bf7c-727b-4ea2-b413-994d58576477","resource":"apiVersion: v1\ndata:\n dump: |\n \"eyJicmFuY2giOiJmb290LXBhdGNoLTEiLCJjbG9uZV9zc2hfdXJsIjoiZ2l0QGdpdGh1Yi5jb206Zm9vdC93Z2UtZGV2LmdpdCIsImNsb25lX3VybCI6Imh0dHBzOi8vZ2l0aHViLmNvbS9mb290L3dnZS1kZXYuZ2l0IiwiaGVhZF9zaGEiOiI5MjU4NzllMDg5NzU1ZmUxYTAyMGYxMGU0YzkyYjExYWU2ZjY3NzI5IiwibnVtYmVyIjoiMSJ9\"\n foo: bar2\n number: \"1\"\nkind: ConfigMap\nmetadata:\n creationTimestamp: \"2023-01-31T10:39:56Z\"\n managedFields:\n - apiVersion: v1\n fieldsType: FieldsV1\n fieldsV1:\n f:data:\n .: {}\n f:dump: {}\n f:foo: {}\n f:number: {}\n f:metadata:\n f:ownerReferences:\n .: {}\n k:{\"uid\":\"79ce755b-e983-4645-9952-c432a4083587\"}: {}\n manager: manager\n operation: Update\n time: \"2023-01-31T10:39:56Z\"\n name: pr-1-info-configmap\n namespace: default\n ownerReferences:\n - apiVersion: templates.weave.works/v1alpha1\n kind: GitOpsSet\n name: gitopsset-configmaps\n uid: 79ce755b-e983-4645-9952-c432a4083587\n resourceVersion: \"27306\"\n uid: 62a11d74-a6bb-4aa7-b237-7c68b72da4ed\n"}

Note bar2

but

❯ kubectl get configmap pr-1-info-configmap -oyaml
apiVersion: v1
data:
  dump: |
    "eyJicmFuY2giOiJmb290LXBhdGNoLTEiLCJjbG9uZV9zc2hfdXJsIjoiZ2l0QGdpdGh1Yi5jb206Zm9vdC93Z2UtZGV2LmdpdCIsImNsb25lX3VybCI6Imh0dHBzOi8vZ2l0aHViLmNvbS9mb290L3dnZS1kZXYuZ2l0IiwiaGVhZF9zaGEiOiI5MjU4NzllMDg5NzU1ZmUxYTAyMGYxMGU0YzkyYjExYWU2ZjY3NzI5IiwibnVtYmVyIjoiMSJ9"
  foo: bar
  number: "1"
kind: ConfigMap
metadata:
  creationTimestamp: "2023-01-31T10:39:56Z"
  name: pr-1-info-configmap
  namespace: default
  ownerReferences:
  - apiVersion: templates.weave.works/v1alpha1
    kind: GitOpsSet
    name: gitopsset-configmaps
    uid: 79ce755b-e983-4645-9952-c432a4083587
  resourceVersion: "27306"
  uid: 62a11d74-a6bb-4aa7-b237-7c68b72da4ed

So the patch.Patch is ignoring the update.. I think this was working w/ listgen. Will test..

I think this was working w/ listgen. Will test..

No, not working

Could add a note to readme that we are using https://github.com/jenkins-x/go-scm not drone
Actual docs https://pkg.go.dev/github.com/jenkins-x/go-scm ! jenkins-x fork links to wrong docs

weaveworks / gitopssets-controller

Pull request generator #16

Not currently re-creating if deleted

~Not updating~

On debugging output

On debugging output