Adding support for transformers>=4.40.2 to avoid crash with mbpp

[meher-m]

Description

• Since transformers 4.38, the library will raise an exception if max_length is set to a value that doesn't include the input size. This means that bigcode will fail when running mbpp with a max_length < 2048.
• In the test split of mbpp, the largest tokenized prompt has length 1748. I verified that setting max_length = 2048 does succeed. However, this results in more than a 3x slow down in running the evaluation (from 1min 3secs to 3 min 30 seconds for 20 data points on 4 H100s).

Change

• Added support to handle this exception if the length of input_ids equals the max_length.
• Allows running mbpp with bigcode with much smaller values of max_length (512) and maintaining behavior consistent with transformers < 4.38.

Testing

Upgrading transformers package to version 4.40.2 and testing the following command succeeds.

accelerate launch main.py \
  --model ~/models/hf-code-llama-7b-instruct \
  --tasks mbpp \
  --max_length_generation 512 \
  --allow_code_execution \
  --precision bf16 \
  --do_sample False

Verify that the results of evaluation are the same as running mbpp before the package upgrade. Both provide results

"mbpp": {
    "pass@1": 0.388
}

Further, the below two files show the generations resulting from the above command before and after the package upgrade. Asserted that the generations were equivalent for each task_id.

• validate_transformers_new.json
• validate_transformers_old.json

[meher-m]

cc @loubnabnl Please let me know what you think of this change!

[meher-m]

@loubnabnl Yes! I have added the change.