search

Unbabel / COMET

A Neural Framework for MT Evaluation
https://unbabel.github.io/COMET/html/index.html
Apache License 2.0
490 stars 76 forks source link

Get list of models #53

Closed mjpost closed 2 years ago

mjpost commented 2 years ago

🚀 Feature

COMET should output a list of available models if -m is used with an invalid model name.

Motivation

It's a bit of a pain to figure out the available models from the CLI. I had to come to the Github page.

Alternatives

Additional context

alvations commented 2 years ago

On a related question, how does this dictionary map to the list of models in https://unbabel.github.io/COMET/html/models.html#faq

{'emnlp20-comet-rank': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/emnlp20-comet-rank.tar.gz',
 'wmt20-comet-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/wmt20-comet-da.tar.gz',
 'wmt20-comet-qe-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/wmt20-comet-qe-da.tar.gz',
 'wmt21-comet-mqm': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-mqm.tar.gz',
 'wmt21-cometinho-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-cometinho-da.tar.gz',
 'wmt21-comet-qe-mqm': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-qe-mqm.tar.gz'
}
ricardorei commented 2 years ago

@alvations that's something I have to fix a long time ago already! Documentation is extremely outdated...

wmt-large-da-estimator-1719 wmt20-comet-da
↑wmt-base-da-estimator-1719 not supported in v1.0
↑wmt-large-qe-estimator-1719 wmt20-comet-qe-da
↓wmt-large-hter-estimator not supported in v1.0
↓wmt-base-hter-estimator not supported in v1.0
↑emnlp-base-da-ranker emnlp20-comet-rank

All other models are from our 2021 paper and are not in that documentation!

ricardorei commented 2 years ago

Regarding @mjpost request, it does print the available metrics when you pass an invalid model name:

comet-score -s src.de -t hyp1.en -r ref.en --gpus 0 --model test

usage: comet-score [-h] [-s SOURCES] [-t TRANSLATIONS [TRANSLATIONS ...]] [-r REFERENCES] [-d SACREBLEU_DATASET] [--batch_size BATCH_SIZE] [--gpus GPUS] [--accelerator {dp,ddp}] [--to_json TO_JSON]
                   [--model MODEL] [--model_storage_path MODEL_STORAGE_PATH] [--mc_dropout MC_DROPOUT] [--seed_everything SEED_EVERYTHING] [--num_workers NUM_WORKERS] [--disable_bar] [--disable_cache]
                   [--disable_length_batching] [--print_cache_info]
comet-score: error: test is not a valid checkpoint path or model choice. Choose from dict_keys(['emnlp20-comet-rank', 'wmt20-comet-da', 'wmt20-comet-qe-da', 'wmt21-comet-mqm', 'wmt21-cometinho-da', 'wmt21-comet-qe-mqm'])
vince62s commented 2 years ago

When I pass --model wmt21-comet-qe-da

Global seed set to 12 Downloading refless-wmt21-comet-da.tar.gz refless-wmt21-comet-da.tar.gz: 0.00B [00:00, ?B/s] Traceback (most recent call last): File "comet/cli/score.py", line 327, in score_command() File "comet/cli/score.py", line 170, in score_command model_path = download_model(cfg.model, saving_directory=cfg.model_storage_path) File "/content/COMET/comet/download_utils.py", line 209, in download_model available_metrics[model], directory=saving_directory File "/content/COMET/comet/download_utils.py", line 165, in download_file_maybe_extract urllib.request.urlretrieve(url, filename=filepath, reporthook=_reporthook(t)) File "/usr/lib/python3.7/urllib/request.py", line 247, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.7/urllib/request.py", line 569, in error return self._call_chain(args) File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(args) File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

which means the model name (taken from doc) is not wrong according to your post, but not able to download. I'll use wmt20-comet-qe-da instead

ricardorei commented 2 years ago

you are right @vince62s! The link is broken for that model. I'll create a new issue for this