About MNTP task

[Hanser14Forever]

Hi, when I run MNTP task, I got the errors below:

File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 719, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
  File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 80, in import_main_class
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1017, in get_code
  File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--accuracy/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/accuracy.py", line 16
    }
    ^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14

How to fix it? I remember i dont have this problem when i reproduce it two weeks ago...

[vaibhavad]

Hi @Hanser14Forever,

The MNTP training is working fine on my end with the current version of llm2vec. Can you share more details of your setup? E.g. transformers version, llm2vec version, the exact command you are running

[Hanser14Forever]

My transformers==4.39.3, evaluate==0.4.2 I just run

torchrun --nproc_per_node=4 experiments/run_mntp.py train_configs/mntp/Llama2.json

[Hanser14Forever]

I just figured out this problem. The reason is I added this line in my script:

export HF_ENDPOINT=https://hf-mirror.com/

I cannot remove this line, otherwise i wouldn't be able to connect to HF on my server. (mainland China). So I clone 'evaluate' to my local path, and change this line:

metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir)

to:

metric = evaluate.load(MY_PATH+"/evaluate/accuracy/accuracy.py")

It turns out to work well :>. Hope this could help others.

[vaibhavad]

Glad you figured it out. Thanks for sharing your findings.