Metric Chrf

[i55code]

Hi, BLEU, Bertscore and COMET runs very well, however Chrf constantly has problem. I made sure the two system predictions have no empty lines, and checked thoroughly. Please take a look and let me know. Thanks!

++++++ Unable to init server: Could not connect: Connection refused Unable to init server: Could not connect: Connection refused

(telescope:77231): Gdk-CRITICAL **: 18:45:57.927: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed 2021-11-15 18:45:59.449 INFO comet.download_utils: wmt20-comet-da is already in cache. Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.bias', 'lm_head.layer_norm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.dense.bias']

• This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 2021-11-15 18:46:08.371 INFO comet.models.base: Encoder model frozen. 2021-11-15 18:46:09.273 INFO pytorch_lightning.utilities.distributed: GPU available: True, used: True 2021-11-15 18:46:09.273 INFO pytorch_lightning.utilities.distributed: TPU available: False, using: 0 TPU cores 2021-11-15 18:46:09.274 INFO pytorch_lightning.accelerators.gpu: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1] 2021-11-15 18:53:13.018 INFO pytorch_lightning.utilities.distributed: GPU available: True, used: True 2021-11-15 18:53:13.018 INFO pytorch_lightning.utilities.distributed: TPU available: False, using: 0 TPU cores 2021-11-15 18:53:13.027 INFO pytorch_lightning.accelerators.gpu: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1] Traceback (most recent call last): File "/home/.local/bin/telescope", line 8, in sys.exit(telescope()) File "/home/.local/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/.local/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/.local/lib/python3.6/site-packages/telescope/cli.py", line 194, in compare for m in metric File "/home/.local/lib/python3.6/site-packages/telescope/cli.py", line 194, in for m in metric File "/home/.local/lib/python3.6/site-packages/telescope/metrics/metric.py", line 47, in pairwise_comparison x_result = self.score(testset.src, testset.system_x, testset.ref) File "/home/.local/lib/python3.6/site-packages/telescope/metrics/chrf/metric.py", line 28, in score chrf = sacrebleu.corpus_chrf(cand, [ref]) File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 745, in corpus_chrf corpus_statistics = get_corpus_statistics(hypotheses, references, order=order, remove_whitespace=remove_whitespace) File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 697, in get_corpus_statistics statistics = get_sentence_statistics(hypothesis, reference, order=order, remove_whitespace=remove_whitespace) File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 678, in get_sentence_statistics reference = delete_whitespace(reference) if remove_whitespace else reference File "/home/.local/lib/python3.6/site-packages/sacrebleu/sacrebleu.py", line 670, in delete_whitespace return re.sub(r'\s+', '', text).strip() File "/usr/lib/python3.6/re.py", line 191, in sub return _compile(pattern, flags).sub(repl, string, count) TypeError: expected string or bytes-like object

[ricardorei]

Could you share the files you used?