Add regression tests for Cometoid metrics

[thammegowda]

• setup.sh script downloads test data (src, ref, mt), metric weights and expected scores
• test_cometoid.sh script runs cometoid model and compares scores

[thammegowda]

Tests are passing for Cometoids, Chrfoid, and BLEURT20. ~But COMET20-QE tests are failing (see file tests/metrics/test_comet20.sh). Here is the error message:~ Update: The issue was only on tg/pybind branch and due to the wrong order of vocabulary and model loading. Fixed it. We now load vocabs first, and then load model.

[2024-01-03 11:55:48] [data] Setting vocabulary size for input 1 to 250,000
[2024-01-03 11:55:48] Error: Requested shape shape=21912x1024 size=22437888 for existing parameter 'Wemb' does not match original shape shape=250000x1024 size=256000000
[2024-01-03 11:55:48] Error: Aborted from marian::Expr marian::ExpressionGraph::param(const string&, const marian::Shape&, marian::Ptr<marian::inits::NodeInitializer>&, marian::Type, bool, bool) in /mnt/home/tg/work/repos/marian-dev/src/graph/expression_graph.h:409
[2024-01-03 11:55:48] Error: Requested shape shape=21912x1024 size=22437888 for existing parameter 'Wemb' does not match original shape shape=250000x1024 size=256000000
[2024-01-03 11:55:48] Error: Requested shape shape=21912x1024 size=22437888 for existing parameter 'Wemb' does not match original shape shape=250000x1024 size=256000000
[2024-01-03 11:55:48] Error: Aborted from marian::Expr marian::ExpressionGraph::param(const string&, const marian::Shape&, marian::Ptr<marian::inits::NodeInitializer>&, marian::Type, bool, bool) in /mnt/home/tg/work/repos/marian-dev/src/graph/expression_graph.h:409
[2024-01-03 11:55:48] Error: Aborted from marian::Expr marian::ExpressionGraph::param(const string&, const marian::Shape&, marian::Ptr<marian::inits::NodeInitializer>&, marian::Type, bool, bool) in /mnt/home/tg/work/repos/marian-dev/src/graph/expression_graph.h:409

[CALL STACK]
[0x5585773d4f23]                                                       + 0xd93f23
[0x5585773d6519]    marian::Embedding::  Embedding  (std::shared_ptr<marian::ExpressionGraph>,  std::shared_ptr<marian::Options>) + 0x4b9
[0x5585773d7e48]    marian::EncoderDecoderLayerBase::  createEmbeddingLayer  () const + 0x1078
[0x5585773d8d41]    marian::EncoderDecoderLayerBase::  getEmbeddingLayer  (bool) const + 0x71
[0x55857706173c]    marian::models::CometBatchEncoder::  apply  (std::shared_ptr<marian::data::SubBatch>) const + 0x7c
[0x558577062030]    marian::models::CometBatchEncoder::  build  (std::shared_ptr<marian::ExpressionGraph>,  std::shared_ptr<marian::data::CorpusBatch>) + 0x4f0
[0x5585770008e9]    marian::EncoderPooler::  apply  (std::shared_ptr<marian::ExpressionGraph>,  std::shared_ptr<marian::data::CorpusBatch>,  bool) + 0xc9
[0x558576b140a6]    marian::Evaluator::  build  (std::shared_ptr<marian::ExpressionGraph>,  std::shared_ptr<marian::data::CorpusBatch>) + 0xf6
[0x558576b5b1b1]    marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}::  operator()  (unsigned long) const + 0x111
[0x558576b5c131]    marian::ThreadPool::enqueue<marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long>(marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long&&)::{lambda()#1}::  operator()  () const + 0x31
[0x558576b5cf20]    std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base,std::__future_base::_Result_base::_Deleter> (),std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>,std::__future_base::_Result_base::_Deleter>,std::__future_base::_Task_state<marian::ThreadPool::enqueue<marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long>(marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long&&)::{lambda()#1},std::allocator<int>,void ()>::_M_run()::{lambda()#1},void>>::  _M_invoke  (std::_Any_data const&) + 0x30
[0x558576aa53fd]    std::__future_base::_State_baseV2::  _M_do_set  (std::function<std::unique_ptr<std::__future_base::_Result_base,std::__future_base::_Result_base::_Deleter> ()>*,  bool*) + 0x2d
[0x7fd10da45ee8]                                                       + 0x99ee8
[0x558576aab23d]    std::__future_base::_Task_state<marian::ThreadPool::enqueue<marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long>(marian::Evaluate<marian::Evaluator>::run<marian::data::CorpusBase>(std::shared_ptr<marian::data::BatchGenerator<marian::data::CorpusBase>>,std::shared_ptr<marian::VectorCollector>)::{lambda(unsigned long)#1}&,unsigned long&&)::{lambda()#1},std::allocator<int>,void ()>::  _M_run  () + 0x10d
[0x558576aac5ee]    std::thread::_State_impl<std::thread::_Invoker<std::tuple<marian::ThreadPool::reserve(unsigned long)::{lambda()#1}>>>::  _M_run  () + 0x19e
[0x7fd10dcb0253]                                                       + 0xdc253
[0x7fd10da40ac3]                                                       + 0x94ac3
[0x7fd10dad2660]                                                       + 0x126660

Tested on marianversion v1.12.14 f12326f3f 2023-11-30 06:23:42 +0000

[thammegowda]

Thanks for the review, @snukky Close this PR because a better alternative (pymarian-eval + pytest) has been submitted to the marian-dev repo.

[snukky]

That's fine. We will still need a PR here to run those tests, same as we run unit tests or examples here.