Support multiple parallel actions for one event

[christian-kreuzberger-dtx]

As a user, I want to run similar (but not necessarily the same task) for the same Keptn CloudEvent (e.g., test.triggered) in parallel.

Note: Right now it seems that job-executor-service will only execute the top-most one (it exists once it has found the first one).

actions:
  - name: "Run health tests"
    events:
      - name: "sh.keptn.event.test.triggered"    # <-- same Keptn CloudEvent type
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "health"                                     # <-- different jsonpath match 

    tasks:
      - name: "Run health task"
        files:
          - jmeter/health.jmx
        args:
          - '-n'
          - '-t'
          - '/keptn/jmeter/health.jmx' # <-- different file
          - '-JPROTOCOL=http'
          - '-JSERVER_PROTOCOL=http'
          - '-JVUCount=5'                 # <-- different JVU Count
          - '-JLoopCount=10'            # <-- different Loop Count
          - '-JSERVER_URL=$SERVICE.$PROJECT-$STAGE.svc.cluster.local'
          - '-j'
          - '/keptn/jmeter/test.log'
          - '-l'
          - '/keptn/jmeter/log.tlf'
        env:
          - name: SERVICE
            value: "$.data.service"
            valueFrom: event
          - name: PROJECT
            value: "$.data.project"
            valueFrom: event
          - name: STAGE
            value: "$.data.stage"
            valueFrom: event

  - name: "Run performance tests"
    events:
      - name: "sh.keptn.event.test.triggered"  # <-- same Keptn CloudEvent type
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"                         # <-- different jsonpath match 
    tasks:
      - name: "Run performance task"
        files:
          - jmeter/performance.jmx
        image: "docker.io/yourorg/jmeter:latest"
        args:
          - '-n'
          - '-t'
          - '/keptn/jmeter/performance.jmx' # <-- different file
          - '-JPROTOCOL=http'
          - '-JSERVER_PROTOCOL=http'
          - '-JVUCount=10'                          # <-- different JVU Count
          - '-JLoopCount=50'                       # <-- different Loop Count
          - '-JSERVER_URL=$SERVICE.$PROJECT-$STAGE.svc.cluster.local'
          - '-j'
          - '/keptn/jmeter/test.log'
          - '-l'
          - '/keptn/jmeter/log.tlf'
        env:
          - name: SERVICE
            value: "$.data.service"
            valueFrom: event
          - name: PROJECT
            value: "$.data.project"
            valueFrom: event
          - name: STAGE
            value: "$.data.stage"
            valueFrom: event

As a very simple test-case, this needs to work:

apiVersion: v2
actions:
  - name: "My first remote task"
    events:
      - name: "sh.keptn.event.my-task.triggered"
    tasks:
      - name: "Greetings"
        image: "alpine"
        cmd:
          - echo
        args:
          - "Hello World"

  - name: "My second remote task"
    events:
      - name: "sh.keptn.event.my-task.triggered"
    tasks:
      - name: "More greetings"
        image: "alpine"
        cmd:
          - echo
        args:
          - "Hello Universe"

whereas both tasks need to be executed (preferably in parallel).

Definition of Done

• [ ] job-executor-service allows multiple tasks to be defined with the same event type (jsonpath does not need to be different)
• [ ] JES sends a .started and .finished event for every started job/task (use message='Job $job-name started' and message='Job $job-name finished' respectively)
• [ ] Jmeter example updated

[agardnerIT]

An extension to this: Responding to both a triggered and a finished event. The triggered event fires but obviously the finished event isn't sent until the job exits so basically it never fires.

Here, I see curl --version but never the GET call.

apiVersion: v2
actions:
  - name: "First"
    events:
      - name: "sh.keptn.event.hello-world.triggered"
    tasks:
      - name: "task1"
        image: "curlimages/curl:7.81.0"
        args:
          - "--version"
  - name: "Second"
    events:
      - name: "sh.keptn.event.hello-world.finished"
    tasks:
      - name: "task 1"
        image: "curlimages/curl:7.81.0"
        args:
          - "-X GET https://webhook.site/d90064cf-b141-4912-bb7a-594f2b0197e4"

[christian-kreuzberger-dtx]

Adam, is this on a remote execution plane? If so, this is to be expected, as remote execution plane unfortunately only works with .triggered event. That's a limitation of Keptn itself.

---

From: Adam Gardner @.> Sent: Saturday, February 26, 2022 05:17 To: keptn-contrib/job-executor-service @.> Cc: Kreuzberger, Christian @.>; Author @.> Subject: Re: [keptn-contrib/job-executor-service] Support multiple actions for one event (Issue #167)

An extension to this: Responding to both a triggered and a finished event. The triggered event fires but obviously the finished event isn't sent until the job exits so basically it never fires.

Here, I see curl --version but never the GET call.

apiVersion: v2 actions:

• name: "First" events:
  • name: "sh.keptn.event.hello-world.triggered" tasks:
  • name: "task1" image: "curlimages/curl:7.81.0" args:
    • "--version"
• name: "Second" events:
  • name: "sh.keptn.event.hello-world.finished" tasks:
  • name: "task 1" image: "curlimages/curl:7.81.0" args:
    • "-X GET https://webhook.site/d90064cf-b141-4912-bb7a-594f2b0197e4"

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeptn-contrib%2Fjob-executor-service%2Fissues%2F167%23issuecomment-1051563298&data=04%7C01%7Cchristian.kreuzberger%40dynatrace.com%7C940ce91b0af64911802008d9f8ded8c7%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637814458273393778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tMGRW9p1XntJAF3VyN8CgPkHJwgqpeK7ZyMbR%2B5TLRI%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANLXZPNGCUEGWRJGKSJH4MDU5BH35ANCNFSM5NJGRB3A&data=04%7C01%7Cchristian.kreuzberger%40dynatrace.com%7C940ce91b0af64911802008d9f8ded8c7%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637814458273393778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Pnp0tMbELyClwFoZWeC%2F%2B0%2BItIJkJCHfk9OiJ5SEfuc%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cchristian.kreuzberger%40dynatrace.com%7C940ce91b0af64911802008d9f8ded8c7%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637814458273393778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dMgJeVKD0xXLqwnSmypsG%2Bu2KqYKeFzzLZfTLuEPQFk%3D&reserved=0 or Androidhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cchristian.kreuzberger%40dynatrace.com%7C940ce91b0af64911802008d9f8ded8c7%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637814458273393778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YIpcQ0khsVt0gdjiLRPHI5qBaFWxExX8Xlej2k3sGxw%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

[agardnerIT]

Nope, this is a local k3d cluster connected via NATS. I'll double-check this is still an issue tomorrow and feedback to this thread.

[agardnerIT]

I can confirm my issue persists on NAT.

Distributor Logs

kubectl -n keptn logs --follow -c distributor --selector app.kubernetes.io/name=job-executor-service
time="2022-02-26T22:05:42Z" level=info msg="Starting UniformLog for Keptn service with integration ID 704d0649781b1742debd5bee03ebdb6506eec111"
time="2022-02-26T22:05:42Z" level=info msg="Connection type: nats"
time="2022-02-26T22:05:42Z" level=info msg="Starting NATS event Receiver"
time="2022-02-26T22:05:42Z" level=info msg="Connecting to NATS server at nats://keptn-nats-cluster ..."
time="2022-02-26T22:05:42Z" level=info msg="Subscribing to topic <sh.keptn.>> with queue group <>"
time="2022-02-26T22:05:52Z" level=info msg="Unsubscribed from NATS topic: sh.keptn.>"
time="2022-02-26T22:05:52Z" level=info msg="Subscribing to topic <sh.keptn.event.hello-world.triggered> with queue group <>"
time="2022-02-26T22:12:12Z" level=info msg="Unsubscribed from NATS topic: sh.keptn.event.hello-world.triggered"
time="2022-02-26T22:12:12Z" level=info msg="Subscribing to topic <sh.keptn.event.hello-world.finished> with queue group <>"
time="2022-02-26T22:12:12Z" level=info msg="Subscribing to topic <sh.keptn.event.hello-world.triggered> with queue group <>"
time="2022-02-26T22:16:10Z" level=info msg="Received a message for topic [sh.keptn.event.hello-world.triggered]\n"
time="2022-02-26T22:16:10Z" level=info msg="Sending CloudEvent with ID 434ca9f3-7464-49a4-b1eb-c77cede34f5c to 127.0.0.1"
time="2022-02-26T22:16:10Z" level=info msg="sent event 434ca9f3-7464-49a4-b1eb-c77cede34f5c"
time="2022-02-26T22:16:10Z" level=info msg="Incoming request: host=, path=/configuration-service/v1/project/wht/stage/demo/service/demoservice/resource/job%2Fconfig.yaml, URL=/configuration-service/v1/project/wht/stage/demo/service/demoservice/resource/job%2Fconfig.yaml"
time="2022-02-26T22:16:10Z" level=info msg="Forwarding request to host=configuration-service:8080, path=v1/project/wht/stage/demo/service/demoservice/resource/job%2Fconfig.yaml, URL=http://configuration-service:8080/v1/project/wht/stage/demo/service/demoservice/resource/job%2Fconfig.yaml"
time="2022-02-26T22:16:12Z" level=info msg="Received response from API: Status=200"
time="2022-02-26T22:16:12Z" level=info msg="Received CloudEvent with ID 2c5dcb0f-64ff-42ae-8200-21cb64fb4ad3 - Forwarding to Keptn\n"
time="2022-02-26T22:16:12Z" level=error msg="No external API endpoint defined. Forwarding directly to NATS server"
time="2022-02-26T22:16:12Z" level=info msg="UniformLogger: received event: sh.keptn.event.hello-world.started"
time="2022-02-26T22:16:12Z" level=info msg="Sent: 2c5dcb0f-64ff-42ae-8200-21cb64fb4ad3, accepted: true"
time="2022-02-26T22:16:22Z" level=info msg="Received CloudEvent with ID 9adea062-3e80-4e2a-b05f-9cee837a2fd0 - Forwarding to Keptn\n"
time="2022-02-26T22:16:22Z" level=error msg="No external API endpoint defined. Forwarding directly to NATS server"
time="2022-02-26T22:16:22Z" level=info msg="UniformLogger: received event: sh.keptn.event.hello-world.finished"
time="2022-02-26T22:16:22Z" level=info msg="Sent: 9adea062-3e80-4e2a-b05f-9cee837a2fd0, accepted: true"
time="2022-02-26T22:16:22Z" level=info msg="Received a message for topic [sh.keptn.event.hello-world.finished]\n"
time="2022-02-26T22:16:22Z" level=info msg="Sending CloudEvent with ID 9adea062-3e80-4e2a-b05f-9cee837a2fd0 to 127.0.0.1"
time="2022-02-26T22:16:22Z" level=info msg="sent event 9adea062-3e80-4e2a-b05f-9cee837a2fd0"

To reproduce:

Note: Webhook.site gives you an endpoint and it'll show full traffic of anything sent to it. Very useful for debugging.

1. Install JES and delete the wildcard subscription
2. Create 2 separate subscriptions, one for task.triggered and one for the same task.finished event
3. Visit https://webhook.site in a new tab to get the first URL
4. Visit https://webhook.site in a private tab to get a second URL
5. Use the first webhook.site URL in your triggered event and hte second for the finished event
6. Send the Keptn event

apiVersion: v2
actions:
  - name: "triggered event"
    events:
      - name: "sh.keptn.event.hello-world.triggered"
    tasks:
      - name: "task 1"
        image: "curlimages/curl:7.81.0"
        cmd:
          - "curl"
        args:
          - "-X"
          - "GET"
          - "https://webhook.site/...YOUR-FIRST-URL-HERE..."
  - name: "finished event"
    events:
      - name: "sh.keptn.event.hello-world.finished"
    tasks:
      - name: "task 1"
        image: "curlimages/curl:7.81.0"
        cmd:
          - "curl"
        args:
          - "-X"
          - "GET"
          - "https://webhook.site/...YOUR-SECOND-URL-HERE..."

Expected output We should see webhook.site list a GET in each tab. One in response for the .triggered event and the private tab should show a GET for the finished event URL.

Actual Output I only ever see the .triggered GET call.

[christian-kreuzberger-dtx]

Thanks for the details, we will try to reproduce it.

In any case, while it sounds familiar, it is unrelated to the feature described in this GH issue. This sounds more like a bug.

Generally speaking, JES can listen and react on .finished events, and in my opinion it looks like everything is configured properly.

[christian-kreuzberger-dtx]

The problem that adam described has been fixed in PR https://github.com/keptn-contrib/job-executor-service/pull/182 and was related to #95.