[Enhancement] Add an integration connector for Strimzi

davidradl commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

currently no Strimzi integration is present

Expected Behavior

Bring in Strimzi information from the CRD using a polling integration connector. The connector will create an event broker and associated topics, theotpics will contain the name,description, partitions and replicas information

Alternatives

the Topic integrator can be used to bring in the topic name, but not the description, partitions and replicas.

Any Further Information?

No response

Would you be prepared to be assigned this issue to work on?

[X] I can work on this

planetf1 commented 2 years ago

I would like to understand more about the connector

What kind of dependencies does it have
How big is it
What are the testing needs - or plan to test
Who do we think the community of people a) using b) developing is?
What is the release cycle - are we going to regular update or leave as is
Who is going to be the repo owner(s) and responsible for setup & maintainance especially on an ongoing basis for dependencies, build, publish etc

We should look at

java build
container build
security scans
static code scans
maven artifact naming
maven publishing
Adding to docs (ideally...)
dependabot
Automated test (without this hard to validate what we have works on an ongoing basis)

The actual build may need modifying to work with the above & some of the techniques we use

Despite the additional work, I am a little more inclined towards a distinct repository just for clarity & agility - though this has the downside of increasing the overall release/maint workload - especially when a change is needed.

Also I would suggest we take the opportunity to ensure the required info to do the setup is documented, for the benefit of future repos. - to the extent it makes sense, and understanding the nature of this particular connector vs the others I think it's important we integrate the code into our broader approach if it is to sit alongside the egeria repos..

See also https://github.com/odpi/egeria/issues/5943 where we touch on some of the challenges and pros/cons - and ideally needing some more common framework/shared assets to coordinate managing the processes.

davidradl commented 2 years ago

@planetf1 raised some questions What kind of dependencies does it have :

   implementation "org.slf4j:slf4j-api:${logbackVersion}"
    implementation "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"
    implementation "com.fasterxml.jackson.core:jackson-annotations:${jacksonVersion}"
    implementation "com.fasterxml.jackson.core:jackson-core:${jacksonVersion}"
    implementation "org.junit.jupiter:junit-jupiter:${junitjupiterVersion}"
    implementation "org.odpi.egeria:open-connector-framework:${egeriaversion}"
    implementation "org.odpi.egeria:audit-log-framework:${egeriaversion}"
    implementation "org.odpi.egeria:repository-services:${egeriaversion}"
    implementation "org.odpi.egeria:repository-services-apis:${egeriaversion}"
    implementation "org.odpi.egeria:topic-integrator-api:${egeriaversion}"
    implementation "org.apache.httpcomponents:httpclient:4.5.13"
    implementation "org.apache.atlas:atlas-intg:${atlasversion}"
    implementation "org.apache.atlas:atlas-client-v2:${atlasversion}"
    implementation "org.apache.atlas:atlas-common:${atlasversion}"
    implementation "org.apache.atlas:atlas-client-common:${atlasversion}"

How big is it - pretty small 2 small classes ffdc and unit tests What are the testing needs - or plan to test. It has extensive unit tests.it would be good to bring this into charts
Who do we think the community of people a) using b) developing is? TBA What is the release cycle - are we going to regular update or leave as is. I suggest the release be updated Who is going to be the repo owner(s) and responsible for setup & maintainance especially on an ongoing basis for dependencies, build, publish etc TBA We should look at java build container build security scans static code scans maven artifact naming maven publishing Adding to docs (ideally...) dependabot

planetf1 commented 2 years ago

Also to clarify - I fully support the suggestion we deliver a Strimzi connector. It's a very interesting Kafka deployment for Kubernetes,.

mandy-chessell commented 2 years ago

What are the Atlas dependencies for?
Why is org.odpi.egeria:repository-services:${egeriaversion} included
How is it working without a dependency on the Data Manager API?

Could you explain the statement under alternatives - The Topic integrator can be used to bring in the topic name, but not the description, partitions and replicas. I thought this was using the topic integrator OMIS? Or is topic integrator something else?

davidradl commented 2 years ago

What are the Atlas dependencies for?

Why is org.odpi.egeria:repository-services:${egeriaversion} included

How is it working without a dependency on the Data Manager API?

Could you explain the statement under alternatives - The Topic integrator can be used to bring in the topic name, but not the description, partitions and replicas. I thought this was using the topic integrator OMIS? Or is topic integrator something else?

the Atlas dependancies should be removed
I need to investigate org.odpi.egeria:repository-services:${egeriaversion} I am not sure I will try to remove it and see if it breaks.
I assume the dependancy on the topic-integrator-api would bringi n the omas client dependancies.

I do not think I was clear enough on the alternatives. It should say It is possible to use KafkaMonitorIntegrationConnector if only the topic name is required.

planetf1 commented 2 years ago

Adding myself as assignee to track ongoing work on creating repositories . I will create a template & new issues to manage the specifics (please leave for now). Target: w/c 14 Mar ( tues on)

planetf1 commented 2 years ago

@davidradl The 'new repo' template is now merged. Thanks for the feedback. Could you give it a go for the strimzi integration connector first. Then can add a link to that issue here.

For feedback on the template/process - suggest we use #6298

odpi / egeria