search

streamshub / flink-sql-examples

Apache License 2.0
1 stars 6 forks source link

Add GitHub action to push image to Quay.io #10

Closed katheris closed 2 months ago

katheris commented 2 months ago
katheris commented 2 months ago

@MikeEdgar could you create the data-generator repository in quay?

MikeEdgar commented 2 months ago

@MikeEdgar could you create the data-generator repository in quay?

@katheris , can we call it flink-data-generator ? There is also a console-datagen in StreamsHub.

SamBarker commented 2 months ago

@MikeEdgar could you create the data-generator repository in quay?

@katheris , can we call it flink-data-generator ? There is also a console-datagen in StreamsHub.

I don't have a problem with flink-data-generator but do we really need two different data generators?

MikeEdgar commented 2 months ago

I don't have a problem with flink-data-generator but do we really need two different data generators?

Ultimately, probably not. I see that this one is using Avro to generate data and the one we developed for the console is meant to generate variable producer/consumer traffic to put interesting data in the console's metrics graphs.

SamBarker commented 2 months ago

Build is timing out waiting for someone to agree to install the packages 😬 need to add a -y to the microdnf commands to say yes we do mean it.

katheris commented 2 months ago

I don't have a problem with flink-data-generator but do we really need two different data generators?

Ultimately, probably not. I see that this one is using Avro to generate data and the one we developed for the console is meant to generate variable producer/consumer traffic to put interesting data in the console's metrics graphs.

The data generator in this repo is designed specifically to generate data for the examples in this repo, I avoided creating a generic data generator, as there are already other examples of generic data generators for Kafka. I've gone with the name flink-examples-data-generator so it's very clear what it is for. @MikeEdgar @SamBarker does that work for you?

SamBarker commented 2 months ago

I'm not going to hold up anything over it but I do think it's odd to have two unrelated data generators in the same organisation. And yes I get that they have different goals today but I would expect they could be unified into a single tool which could service the needs of both.