Instead of calling merge_priority_function (usually a feature transform and a call to scikit-learn) directly as we encounter each edge, we add the edge to a batch and evaluate the function vectorized across all the accumulated edges.
This results in dramatic speedups at both learning and test-time, at no cost to memory:
# master
Timing results:
--- build RAG 0.399227459
--- build feature caches 1.0142317930000004
--- learn flat 6.226274373
--- learn agglo 437.25951286
--- classifier training 1.7207173679999528
--- segment test volume 135.74367634500004
Memory results:
--- base RAG 28.049 MB
--- feature caches 0.629 MB
--- training data 1.490 MB
--- classifier training 0.071 MB
--- segment test volume 27.468 MB
# batch-edges
Timing results:
--- build RAG 0.3964842690000001
--- build feature caches 1.0158853469999998
--- learn flat 6.258743795999999
--- learn agglo 79.627681051
--- classifier training 1.5787226850000025
--- segment test volume 18.391299654999997
Memory results:
--- base RAG 28.049 MB
--- feature caches 0.629 MB
--- training data 1.546 MB
--- classifier training 0.071 MB
--- segment test volume 27.607 MB
That's 5.5x train, 7.5x test speedup, on top of the speed gains from #82.
Coverage increased (+0.2%) to 49.08% when pulling 0056ae66e8dc57db5406776c67e736934c910d52 on jni:batch-edges into a55ceaa5c757366a26946b2c6ad0dd547966876b on janelia-flyem:master.
Instead of calling
merge_priority_function
(usually a feature transform and a call to scikit-learn) directly as we encounter each edge, we add the edge to a batch and evaluate the function vectorized across all the accumulated edges.This results in dramatic speedups at both learning and test-time, at no cost to memory:
That's 5.5x train, 7.5x test speedup, on top of the speed gains from #82.