Tuning multimodal pretrained model, using BloomTokenizerFast

[TonyAlbertWan]

As mentioned in https://github.com/X-PLUG/mPLUG-Owl/issues/141, when tuning multimodal pretrained model, we can use BloomTokenizerFast. i change the code in line 141 from train.py and add some nessecary packges:

from transformers import Trainer, BloomTokenizerFast

tokenizer = BloomTokenizerFast.from_pretrained(args.pretrained_ckpt)
# tokenizer = MplugOwlTokenizer.from_pretrained(args.pretrained_ckpt)

However, after i changing the code, i got such error:

Traceback (most recent call last):
  File "/workdir/./pipeline/train.py", line 217, in <module>
    main()
  File "/workdir/./pipeline/train.py", line 177, in main
    train_data, valid_data = train_valid_test_datasets_provider(
  File "/workdir/pipeline/data_utils/__init__.py", line 7, in train_valid_test_datasets_provider
    train_ds, valid_ds = build_train_valid_test_datasets(
  File "/workdir/pipeline/data_utils/__init__.py", line 21, in build_train_valid_test_datasets
    train_ds = MultiModalDataset(input_file[0], tokenizer, train_processors, max_length)
  File "/workdir/pipeline/data_utils/xgpt3_dataset.py", line 49, in __init__
    self.dataset += load_jsonl(input_file)
  File "/workdir/pipeline/data_utils/xgpt3_dataset.py", line 35, in load_jsonl
    return [json.loads(l.strip("\n")) for l in f.readlines()]
  File "/workdir/pipeline/data_utils/xgpt3_dataset.py", line 35, in <listcomp>
    return [json.loads(l.strip("\n")) for l in f.readlines()]
  File "/workdir/conda_envs/mplug_owl/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/workdir/conda_envs/mplug_owl/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

How to fix this? is there other issues should i concern? thanks!

[TonyAlbertWan]

found another one in line 149 at train.py:

        peft_config = LoraConfig(
            target_modules=r'.*language_model.*\.(query_key_value)', 
            # target_modules=r'.*language_model.*\.(q_proj|v_proj)', 
            inference_mode=args.inference_mode, 
            r=args.lora_r, 
            lora_alpha=args.lora_alpha, 
            lora_dropout=args.lora_dropout
        )

now its ok for tuning multimodal model