Added changes made for Eventing only

[AndreiOla]

Changed I made this branch for splitting this PR: https://github.com/boomerang-io/flow.service.workflow/pull/210 This branch contains the changes related to Eventing. However, this branch should be merged after this one: 'feature/generalFixes'

[tlawrie]

@AndreiOla @corcoja @gchickma some of the eventing changes have been cherry-picked to the new engine and updated to use the new WorkflowRun and TaskRun concepts.

https://github.com/boomerang-io/flow.service.engine

[corcoja]

Hey @tlawrie, this is cool! I've noticed it is the simplified version o eventing though (I quickly looked at it so I might be wrong) – let me know if you want at any point in the future if you want to advance the development on this 😊

[tlawrie]

Hey @corcoja I'm actually thinking of a slight change to the eventing architecture. Following on from how tekton / knative etc do event sinks, I'm thinking that we actually move the eventing that contains the NATs integration into a separate service.

That way, the workflow service is just accepting events and firing events, and another service is set up as the sink and it handles putting it on NATs or any cloud services from Azure / Google / etc.

This keeps the code base clean and extensible.

What are your thoughts?

[AndreiOla]

Hey @tlawrie @corcoja I started extracting the NATS part out of the CE Service, but for a different reason. Currently the EventingService tries to connect to a NATS server and there is no way to disable this. So, in order to disable just the NATS part, I put the code in a different service. Please take a look and, if it seems right, I can port the changes to this branch also. https://github.com/boomerang-io/flow.service.workflow/commit/e359205f5ca8693095aa88afeff6b49898707e86

[corcoja]

@tlawrie, this definitely is an improvement over current architecture. Quick question though, when you say service, do you mean a Spring Boot service or a standalone java µS? On the later, my only worry is the timeline these changes can be implement, since extracting code and putting it in a new, dedicated µS, usually takes time to build and test. But I'm happy to hear that @AndreiOla has already made some progress on this.

[tlawrie]

Hi @AndreiOla @corcoja w.r.t to the word 'service', i mean a different standalone microservice.

V4 Microservice Architecture

• the Listener service is no longer and just an API endpoint on the Workflow service.
• the Workflow service no longer handles the events related to WorkflowRuns and TaskRuns, this has moved to the Engine service which owns the data entities
• the Engine service fires CloudEvents blindly to 0..n event sink URLs
• the Controller (soon to be TektonHandler) service is just one event sink

Proposal I propose that for Eventing, any implementation specific details for NATs are moved to a separate microservice (NATSHandler) or alternatively DAPR is used (ill circle back to this option).

The NATSHandler would allow the service to receive the events for WorkflowRun and TaskRun and do with them as it pleases. Additionally, the domain separation of concerns would allow this eventing microservice to have its own lifecycle separate to Engine.

DAPR Sink An alternative to your own microservice for NATS would be to use DAPR as a sidecar and that way a sink url would be localhost (in the container) which is picked up by DAPR.

This would allow support outside of NATS and allow additional event systems (i.e. allow IBM in the future to use a Cloud based event service and not have to maintain NATS). Reference

Questions

1. Are there additional events besides WorkflowRun and TaskRun?

cc: @gchickma @amhudson

[AndreiOla]

@tlawrie I agree that for V4, there should be a Broker pattern implemented or something similar (where you can publish CEs and consume them just by plugging-in a message system - NATS in our case), but for the already developed code in the current v3, I would keep things consistent (use the 'internal' services for CEs and NATS being plugged-in using Aspects and a property for enable/disable). I already extracted the NATS related code (in workflow service. There is also lib.eventing that would need refactoring), so it can be put later in a microservice. Refactoring and re testing this code, would take a lot of time and it won't be entirely reusable in v4 anyway, so it has very short life.

[tlawrie]

I'm fine if that's the aim, to get it merged in for v3 (and also not break when Eventing is disabled)

I wanted to highlight that the code will be immediately lost on the v4 branch of workflow and not be used. - In current state. The code is already being ported for producing the events

Do we want to bother with v3 and focus on making it work with v4?

[AndreiOla]

As migrating to v4 will take some changes and as we already spent time in developing this code, I would like to merge this in v3 and make it better in v4.

[AndreiOla]

@tlawrie @corcoja @amhudson Please review these changes.

[AndreiOla]

@corcoja I am not sure to which APIs you are referring. There are already some tests there.

[corcoja]

@corcoja I am not sure to which APIs you are referring. There are already some tests there.

Integration tests for controller methods that use CloudEventsService, e. g. acceptEvent(...) inside InternalController. Mock with Mockito any other intermediary services that are used by this method, this way you'll be testing just the functionality of acceptEvent(...).

The tests that you mentioned are solely for testing the CloudEvents payloads, there are no integration tests afaik.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

| GitGuardian id | Secret | Commit | Filename | | | -------------- | ------------------------- | ---------------- | --------------- | -------------------- | | [5729846](https://dashboard.gitguardian.com/incidents/5729846?occurrence=84773537) | Generic High Entropy Secret | e02e7a067823f99840c9b7ecfcd2bf20d48ff58a | src/test/java/io/boomerang/tests/controller/CloudEventsServiceTests.java | [View secret](https://github.com/boomerang-io/flow.service.workflow/commit/e02e7a067823f99840c9b7ecfcd2bf20d48ff58a#diff-a809d63b6530c931ab1f88c2768b7e5e9b65ec36587f07c5b33f46da585444a3R77) |

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secret safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate this secret](https://docs.gitguardian.com/secrets-detection/detectors/generics/generic_high_entropy_secret#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.

---

<sup>🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!</sup>

[AndreiOla]

@corcoja @amhudson @tlawrie I added some Tests. Please review the changes, so I can merge this branch in v3.

[tlawrie]

Ill defer this to @gchickma and @amhudson for an IBM approval.

v4 has already been forked and is nearing completion with a different eventing implementation - refer to comment

[AndreiOla]

@amhudson @gchickma Please find time and review, so I can merge this code