Open HyunggyuJang opened 11 months ago
The code is missing the implementation for reverting shuffling in the voting process. Currently, the shuffling ensemble is implemented in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/problem_utils.py#L63 However, I couldn't find a section in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/analyze.py#L77-L98 where the original order is recovered to account for the voting result.
The code should include the necessary logic to revert the shuffling and account for the voting result.
Issue Description
The code is missing the implementation for reverting shuffling in the voting process. Currently, the shuffling ensemble is implemented in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/problem_utils.py#L63 However, I couldn't find a section in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/analyze.py#L77-L98 where the original order is recovered to account for the voting result.
Expected Behavior
The code should include the necessary logic to revert the shuffling and account for the voting result.