janelia-flyem / gala

Automatic segmentation of electron microscopy volumes
BSD 3-Clause "New" or "Revised" License
75 stars 29 forks source link

Implement @mjanusz's edge batching #84

Closed jni closed 8 years ago

jni commented 8 years ago

Instead of calling merge_priority_function (usually a feature transform and a call to scikit-learn) directly as we encounter each edge, we add the edge to a batch and evaluate the function vectorized across all the accumulated edges.

This results in dramatic speedups at both learning and test-time, at no cost to memory:

# master
Timing results:
---  build RAG 0.399227459
---  build feature caches 1.0142317930000004
---  learn flat 6.226274373
---  learn agglo 437.25951286
---  classifier training 1.7207173679999528
---  segment test volume 135.74367634500004
Memory results:
---  base RAG 28.049 MB
---  feature caches 0.629 MB
---  training data 1.490 MB
---  classifier training 0.071 MB
---  segment test volume 27.468 MB

# batch-edges
Timing results:
---  build RAG 0.3964842690000001
---  build feature caches 1.0158853469999998
---  learn flat 6.258743795999999
---  learn agglo 79.627681051
---  classifier training 1.5787226850000025
---  segment test volume 18.391299654999997
Memory results:
---  base RAG 28.049 MB
---  feature caches 0.629 MB
---  training data 1.546 MB
---  classifier training 0.071 MB
---  segment test volume 27.607 MB

That's 5.5x train, 7.5x test speedup, on top of the speed gains from #82.

coveralls commented 8 years ago

Coverage Status

Coverage increased (+0.2%) to 49.08% when pulling 0056ae66e8dc57db5406776c67e736934c910d52 on jni:batch-edges into a55ceaa5c757366a26946b2c6ad0dd547966876b on janelia-flyem:master.

jni commented 8 years ago

Thanks so much for the tip, @mjanusz!