janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
196 stars 33 forks source link

What is the formula for kafka topic names? #293

Closed stuarteberg closed 5 years ago

stuarteberg commented 5 years ago

I have a question about the listed RepoUUID for a particular data instance.

Have a look at the info for segmentation_todo: http://emdata3:8900/api/node/28841c8277e044a7b187dda03e18da13/segmentation_todo/info The RepoUUID is listed as 4ea3bd5698204089b82408c178fe1d55 But that is not the root node for the production repo. (The root node is a776af0b132f44c3a428fe7607ba0da0) Is that expected? The weird thing is that the kafka server lists TWO different topics for this instance. Well, three if you count the hemiMirror topic:

dvidrepo-a776af0b132f44c3a428fe7607ba0da0-data-1e76b4afa159488b836a828afe772d86 dvidrepo-4ea3bd5698204089b82408c178fe1d55-data-1e76b4afa159488b836a828afe772d86 hemiMirror-dvidrepo-a776af0b132f44c3a428fe7607ba0da0-data-1e76b4afa159488b836a828afe772d86

However, dvidrepo-4ea3bd5698204089b82408c178fe1d55-data-1e76b4afa159488b836a828afe772d86 contains no messages.

Looking at other data instances, I guess the RepoUUID is defined as "the first node in which this data instance appeared"? But the KAFKA topic is defined using the ROOT node for the repo, regardless of the instance's declared RepoUUID? Is that the rule?

DocSavage commented 5 years ago

Here's the actual rule for general messages:

func (d *Data) ProduceKafkaMsg(b []byte) error {
    // create topic (repo ID + data instance uuid)
    // NOTE: Kafka server must be configured to allow topic creation from
    // messages sent to a non-existent topic
    rootuuid, _ := d.DAGRootUUID()
    datauuid := d.DataUUID()
    topic := "dvidrepo-" + string(rootuuid) + "-data-" + string(datauuid)

    // send message if kafka initialized
    return storage.KafkaProduceMsg(b, topic)
}
DocSavage commented 5 years ago

If you are using a kafka with topic creation ON you could potentially create an empty topic by querying it I think.