weifei7
commented
4 months ago Sorry for the late reply. Please run this script on your dataset to finetune on your own dataset:
bash run.sh mobilevlm_v2_1.7b finetune ${LANGUAGE_MODEL} ${VISION_MODEL} ${OUTPUT_DIR}And check the --data_path and --image_folder in run.sh, which should match to your self-made dataset.
When you build your dataset, you need to put all your data samples in a list and store them in a json file (format is same with llava/sharegpt4v). Each sample corresponds to a dict containing "id", "image" and "conversations". Here are some examples:
[
# Example 1: multi_modal_data
{
'id': 37,
'image': 'coco/train2017/000000203564.jpg',
'conversations': [
{'from': 'human', 'value': '<image>\nWhat are the key elements in this picture?'},
{'from': 'gpt', 'value': 'A bicycle replica with a clock as the front wheel.'}
]
},
# Example 2: text_only_data
{
'id': 0,
'conversations': [
{'from': 'human', 'value': 'Give three tips for staying healthy.'},
{'from': 'gpt', 'value': '1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.'}
]
},
# Example 3: multi_modal_dialog_data
{
'id': 287140,
'image': 'coco/train2014/COCO_train2014_000000287140.jpg',
'conversations': [
{'from': 'human', 'value': '<image>\nhow many bikes there'},
{'from': 'gpt', 'value': '3'},
{'from': 'human', 'value': 'what color are bikes'},
{'from': 'gpt', 'value': 'i see green red and white'},
{'from': 'human', 'value': 'are they parked on stock parking'},
{'from': 'gpt', 'value': 'no'},
{'from': 'human', 'value': 'are there any people'},
{'from': 'gpt', 'value': '2'},
{'from': 'human', 'value': 'what are their genders'},
{'from': 'gpt', 'value': "can't tell have on helmets"},
{'from': 'human', 'value': 'what color of their helmets'},
{'from': 'gpt', 'value': 'green and white and other is black'},
{'from': 'human', 'value': 'do they pose to picture'},
{'from': 'gpt', 'value': 'no'},
{'from': 'human', 'value': 'are they standing by their bikes'},
{'from': 'gpt', 'value': 'no'},
{'from': 'human', 'value': 'what are they doing'},
{'from': 'gpt', 'value': 'sitting on bikes'},
{'from': 'human', 'value': 'white time of day is it'},
{'from': 'gpt', 'value': 'looks like afternoon'}
]
},
... ...
]
QvQKing
Muhammad4hmed
Bobby-youngking
hi all, thx for your work. I wonder how to finetune on user's self-made dataset like qwen-vl here: https://github.com/QwenLM/Qwen-VL/blob/master/README_CN.md#%E5%BE%AE%E8%B0%83 we hace tested the pretrained mobile-vlm model on specific domain dataset,the result seems not promising enough, so we want to have a try on our own dataset, thx.