Debezium Dev Services

[lordofthejars]

Description

I was thinking it would be great to have a Debezium devservice. I have bandwidth to implement it and my idea would be to add it in as a quarkiverse project.I have the bandwidth to implement it, and my idea would be to add it in as a quarkiverse project.It

Implementation ideas

For implementing this DevService I would like to find an example of a DevService that waits until other DevServices are up and running. Also, an example of a DevService getting the configuration properties of other DevServices. I need this as Debezium needs to start when the DB devservice and the Kafka devservice is up and running.

[quarkus-bot[bot]]

/cc @geoand (devservices), @stuartwdouglas (devservices)

[gsmet]

There is a Debezium extension hosted in the Debezium project.

Maybe it would be a better place for it?

If so, probably a good idea to close this and open an issue on the DBZ side.

[lordofthejars]

The extension is for the outbox pattern IIRC. This is more generic.

Enviat amb Gmail Mobile

El dj, 14 des. 2023 a les 17:56 Guillaume Smet @.***> va escriure:

There is a Debezium extension hosted in the Debezium project.

Maybe it would be a better place for it?

If so, probably a good idea to close this and open an issue on the DBZ side.

— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/37749#issuecomment-1856206168, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALSMYJ63E75Q2R4SSWTJ23YJMVTBAVCNFSM6AAAAABAVA54HWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGIYDMMJWHA . You are receiving this because you authored the thread.Message ID: @.***>

[geoand]

What would a generic Debezium extension provide?

[lordofthejars]

Only Devservice to start debezium conneacting to DB devservice and Kafka devservice

Enviat amb Gmail Mobile

El dj, 14 des. 2023 a les 18:36 Georgios Andrianakis < @.***> va escriure:

What would a generic Debezium extension provide?

— Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/37749#issuecomment-1856288820, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALSMYOK2VYLLQ2RZXLR7D3YJM2LLAVCNFSM6AAAAABAVA54HWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGI4DQOBSGA . You are receiving this because you authored the thread.Message ID: @.***>

[michalvavrik]

Only Devservice to start debezium conneacting to DB devservice and Kafka devservice

Hello, I don't think you do need a dev service for that, you need processor that configure Debezium connectors https://debezium.io/documentation/reference/stable/connectors/ based on Quarkus extensions configuration like DB Dev Services etc. Is that what you meant @lordofthejars or do you have something else in mind?

Problem with dev service is that it would only work for dev mode, but I think it would be better to be able to just add PG JDBC extension and have configured https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-example-configuration like it was said here https://www.youtube.com/live/MQR1SLJVpgM?feature=shared&t=2635 and similarly for other connectors.

I'm playing with Debezium these days and I agree it could be much easier if it were integrated. As Debezium project decided to put Outbox extension in here https://github.com/debezium/debezium maybe it would be better to open issue there instead? I'm very much interested in this.

[michalvavrik]

What would a generic Debezium extension provide?

Only Devservice to start debezium conneacting to DB devservice and Kafka devservice

+1

Ideally Kafka Connect https://debezium.io/documentation/reference/stable/configuration/avro.html#deploying-with-debezium-containers would be integrated with Apicurio Regisry (Dev svc) https://quarkus.io/guides/kafka-schema-registry-avro and so would database connectors so that I don't have to set connection string and start container manually etc.

[michalvavrik]

I have the bandwidth to implement it, and my idea would be to add it in as a quarkiverse project.

That's great. I had a look to Debezium project and Debezium server more carefully now (that is based on Quarkus runtime too) and I still think that if you will propose this in Debezium project, there is a chance this extension will have better quality because you will get review from people who write Debezium and who are obviously using Quarkus so they know thing or two.

For implementing this DevService I would like to find an example of a DevService that waits until other DevServices are up and running.

So the idea is to start Debezium engine when Dev Svcs like Kafka and Dbs are running, but why do you want to isolate it to DEV mode? How is situation different to production mode when you can gather same information in a same fashion? For example if Quarkus application has configured datasource or whether the datasource is configured by Dev Svc, they are still runtime configuration properties available when runtime config setup is complete (well, that's how I read the code, correct me if I'm wrong).

I need this as Debezium needs to start when the DB devservice and the Kafka devservice is up and running.

For Quarkus application, I consumed config props and it works but it's ugly

https://github.com/michalvavrik/current-design-patterns-and-their-practical-impacts/blob/ec6a711df7016102909f1c20396d3b4ce3a89504/theory-change-data-capture/src/main/java/edu/michalvavrik/cdc/DatabaseChangeListener.java#L28

but since you are writing extension that is not a container, but a Java code, you can use lifecycle events to start it / stop or hookup on build items like RunningDevService (bad option). If you really want to produce RunningDevService then I don't know the way and I think you will need to tweak Db Dev Svcs and Kafka to be able to do that (like produce some build item).

There is one else thing - I tried PG Dev Svc and it doesn't work OOTB, I had to set quarkus.datasource.devservices.command=postgres -c wal_level=logical so I think you will need to do more than just consume other dev svcs config, but I don't know Debezium that well.

Good luck!