Kafka/mutations message enhancements

janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice

http://dvid.io

Other

196 stars 33 forks source link

Kafka/mutations message enhancements #267

Closed stuarteberg closed 1 year ago

stuarteberg commented 6 years ago

It would be helpful for debugging if the Kafka log messages included more info. And these enhancements should apply to the JSON mutations log, too.

Edit: Summarizing the complete list here:

[x] 1. All messages should include a timestamp in the JSON.
[x] 2. Include user ID (and app?) in all messages.
[x] 3. split-supervoxel messages could include the body ID
[x] 4. split-supervoxel-complete could include the sizes of both the 'remain' and 'split' supervoxels.
[x] 5. cleave-complete could include the size of both bodies aftwards.
[x] 6. All complete messages should include all of the original fields that triggered the message. Generally speaking, I'm not interested in messages that didn't succeed. I'd rather only parse the complete messages. Right now, if I want to only process "successful" events, I have to pair up the original message with the corresponding "complete" message.

stuarteberg commented 5 years ago

Other enhancements:

Include user ID (and app?) in all messages.
(It might be too late to change this.) It is inconvenient (for manual analysis) that different messages use different names for the "target" body. For instance, cleave messages list the OrigLabel, whereas merge and split messages list the Target body.
It would be convenient if the split-supervoxel message also mentioned the body ID that owned the supervoxel at the time of the split. Of course, I can figure it out via a separate call to /mapping, but I know DVID is already looking up body ID before performing the split, so including it in the kafka message would not be costly.

umayaml commented 4 years ago

It would be helpful for debugging if the Kafka log messages included more info.

All messages should include a timestamp in the JSON.

split-supervoxel messages could include the body ID, and split-supervoxel-complete could include the sizes of both the 'remain' and 'split' supervoxels.

I would also like to see the same reporting for cleaves. Where we can get the sizes of the new body and the Target body after the cleave.

stuarteberg commented 3 years ago

@DocSavage Would it be difficult to implement item 3 from above?

split-supervoxel messages could include the body ID

It would be useful for debugging and detective work. For example, I needed it earlier today, but didn't have it.

stuarteberg commented 3 years ago

I know we're going to be cranking through the technical debt list soon, so I wanted to flag this one more time. I'm currently trying to do some detective work on a body that's somehow been totally screwed up, and it would have been nice to have the above feature implemented in the kafka log. I really need this before we start the CNS.

stuarteberg commented 1 year ago

@DocSavage It appears that split-supervoxel-complete messages in the JSON mutations response do not contain a user ID.

DocSavage commented 1 year ago

This was fixed in commit 3bde644

DocSavage commented 1 year ago

@stuarteberg Is this a new issue pertaining to JSON mutation endpoint and not Kafka after the above update?