search

janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
197 stars 33 forks source link

/supervoxel-splits response is missing some events #316

Open stuarteberg opened 5 years ago

stuarteberg commented 5 years ago

The /supervoxel-splits endpoint is supposed to mention all** of the split and split-supervoxel mutations that can be found in the kafka log, but I have a few examples from the kafka log that are missing from the /supervoxel-splits output.

The following 7 split events are missing from the /supervoxel-splits output:

In [34]: msgs_df.query('mutid in @missing_muts').drop(columns=['msg'])
Out[34]:
                     timestamp                              uuid       mutid            action  target_body   target_sv
1076   2018-10-16 14:02:59.627  ef1da1a01c4941b6876ab77502843d0c  1002217029             split   5812981673           0
1079   2018-10-16 14:03:00.628  ef1da1a01c4941b6876ab77502843d0c  1002217029    split-complete   5812981673           0
1082   2018-10-16 14:04:50.683  ef1da1a01c4941b6876ab77502843d0c  1002217032             split   5812981673           0
1083   2018-10-16 14:04:51.403  ef1da1a01c4941b6876ab77502843d0c  1002217032    split-complete   5812981673           0
1214   2018-10-16 14:22:18.843  ef1da1a01c4941b6876ab77502843d0c  1002217098             split    394768379           0
1225   2018-10-16 14:22:19.984  ef1da1a01c4941b6876ab77502843d0c  1002217098    split-complete    394768379           0
1440   2018-10-16 14:25:18.639  ef1da1a01c4941b6876ab77502843d0c  1002217211             split    394768379           0
1441   2018-10-16 14:25:19.763  ef1da1a01c4941b6876ab77502843d0c  1002217211    split-complete    394768379           0
1454   2018-10-16 14:27:04.146  ef1da1a01c4941b6876ab77502843d0c  1002217218             split    394768379           0
1455   2018-10-16 14:27:05.032  ef1da1a01c4941b6876ab77502843d0c  1002217218    split-complete    394768379           0
1456   2018-10-16 14:28:40.281  ef1da1a01c4941b6876ab77502843d0c  1002217219             split    394768379           0
1457   2018-10-16 14:28:40.995  ef1da1a01c4941b6876ab77502843d0c  1002217219    split-complete    394768379           0
1488   2018-10-16 14:31:51.632  ef1da1a01c4941b6876ab77502843d0c  1002217235             split    394768379           0
1489   2018-10-16 14:31:52.726  ef1da1a01c4941b6876ab77502843d0c  1002217235    split-complete    394768379           0

**Unless the event has no corresponding split-complete message in the kafka log. But that doesn't apply to the examples above, obviously.

DocSavage commented 4 years ago

@stuarteberg In the case of a kafka queue overload, there could be a situation where a split-complete message is successful but the actual split is not. Given that we have the overflow logs more recently, do we still believe this is a problem on the DVID side and not a kafka queue overload issue?

stuarteberg commented 4 years ago

@DocSavage

there could be a situation where a split-complete message is successful but the actual split is not

I don't understand. I guess I'll talk to you in person.

DocSavage commented 4 years ago

The above was based on incorrect view of events missing from kafka log not from the internal representation. Just to verify, the only missing supervoxel splits were from the broad split label request and not missing supervoxel-split requests, right? (The actions are listed as "split" and not "split-supervoxel".)