The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.
Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in QAAccuracyConfig to disable it?
The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.
Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in
QAAccuracyConfigto disable it?