Closed juhoinkinen closed 1 year ago
Patch coverage: 100.00
% and project coverage change: +0.01
:tada:
Comparison is base (
3e8f42f
) 99.56% compared to head (f280342
) 99.57%.
:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
Kudos, SonarCloud Quality Gate passed!
0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells
No Coverage information
0.0% Duplication
I rerun the NN ensemble suggest and eval timings and updated the results table after the last change: it seemed to give a few seconds improvement more.
This adds the
_suggest_batch
method to ensemble backends.In all ensemble backends (simple, PAV, NN) the suggestions from the source projects are fetched by using batched suggest calls (this is on project level, so whether a backend actually uses batched suggest depends on the backend).
In case of NN ensemble the prediction (now via
_merge_hit_sets_from_sources
) is performed on the whole batch of the base suggestions in one call of the NN model. The prediction is made using the model's__call__()
method instead ofpredict()
as it is the recommended way for "small numbers of inputs that fit in one batch" and it should offer better performance; this should also fix #674.Also the EnsembleOptimizer is made to use batched suggest. Quickly testing the
hyperopt
command on an ensemble project gives very similar weights and best NDCG scores before and after this PR.This PR improves somewhat the performance of NN ensemble suggest functionality while the results remain (only nearly?) identical (I think there were small differences in suggestion scores on my laptop). I think most of the increase comes from using
__call__
of the model instead ofpredict
.The below results are from runs at kj-kk using the current Finto AI YSO NN ensemble model (having MLLM, fastText and Omikuji base projects).
suggest
Targeting 6 times
tests/corpora/archaeology/fulltext/*.txt
:eval
Targeting 200 documents from
kirjaesittelyt2021/yso/fin/test
:With 1 job
With 4 jobs