DataCater / datacater

The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
https://datacater.io
Other
82 stars 4 forks source link

Refactor: stream, deployment, config and pipeline #218

Closed ChrisRousey closed 1 year ago

ChrisRousey commented 1 year ago

This PR refactors all the top level resources in datacater.

Refactors include:

Please feel free to make more suggestions.

ChrisRousey commented 1 year ago

Some of these utilities contain functionality that is also needed in DataCater EE. I am wondering whether it is feasible and would make sense to outsource such functionality into an own utility project, similar to what we did with the SerDe classes, such that we could include the shared functionality in both the open core and ee and effectively get rid of duplicated code.

@flippingbits do you have any examples of which utility methods could be shared off the to of you head so i can look further into this? Otherwise i'll have a look myself. In general I think this is a good idea. I'd have to test how smoothly we could migrate mutiny worker methods like StreamUtilities.getStreamFromDeployment() since they are entity dependant. We might also lose some sonarqube analysis metrics for the utilities, stuff like test coverage might not work correctly since, my guess is, we'd have to implement the tests in the main repo? But those are just guesses i'd have to confirm by testing this first.

HknLof commented 1 year ago

Just to jump in with another comment :D. From a Java Quarkus perspective, I do believe, that these would have to be externalized into Quarkus extensions or published as JARs.

flippingbits commented 1 year ago

@flippingbits do you have any examples of which utility methods could be shared off the to of you head so i can look further into this? Otherwise i'll have a look myself.

Yes, ConfigUtilities or StreamUtilities are examples of functionality that might be needed in EE. However, I would suggest to wait with outsourcing these classes for now, since we currently discuss other ways for a more efficient integration of the open core and EE. If it stays relevant, we could address it in a separate PR.

sonarcloud[bot] commented 1 year ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

62.0% 62.0% Coverage
0.0% 0.0% Duplication