Hi, I met the following ImportError during generation evaluations.

generations were saved at ./output_humaneval/humaneval_humaneval.json Evaluating generations... Traceback (most recent call last): File "/root/software/bigcode-evaluation-harness/main.py", line 414, in main() File "/root/software/bigcode-evaluation-harness/main.py", line 398, in main results[task] = evaluator.evaluate( File "/root/software/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 107, in evaluate results = task.process_results(generations, references) File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/humaneval.py", line 98, in processresults results, = compute_code_eval( File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/code_eval.py", line 154, in compute_code_eval result = future.result() File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.get_result() File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 389, in get_result raise self._exception File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/execute.py", line 36, in check_correctness manager = multiprocessing.Manager() File "/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/context.py", line 55, in Manager from .managers import SyncManager ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers' (most likely due to a circular import) (/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/managers.py)

Here is my script: '''

!/bin/bash

taskname=humaneval mkdir -p output_$taskname

python /root/software/bigcode-evaluation-harness/main.py \ --model /data00/models/meta-llama_Meta-Llama-3.1-8B-Instruct/ \ --trust_remote_code \ --tasks $taskname \ --limit 8 \ --max_length_generation 512 \ --temperature 0.8 \ --do_sample True \ --n_samples 16 \ --batch_size 16 \ --precision fp16 \ --allow_code_execution \ --save_generations \ --save_generationspath ./output$taskname/$taskname.json '''

Could you share some tips on how to solve this problem?

bigcode-project / bigcode-evaluation-harness

ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers' #267

!/bin/bash