search

BYC-Sophie / llava-finetune

0 stars 0 forks source link

LLaVA Finetune #1

Open BYC-Sophie opened 11 months ago

BYC-Sophie commented 11 months ago

Hi @anonymous-atom! Sorry for any inconvenience brought about to you by reaching out like this. I noticed you also wanted to try fine-tuning on LLaVA as was mentioned in https://github.com/haotian-liu/LLaVA/issues/564#issuecomment-1765789727_ I wonder if you have tried the fine-tuning yet? It seems a little bit confusing as I do not have computation recourses as instructed in the repo. Appreciate it if we could discuss the fine-tuning process and figure out a solution together!

anonymous-atom commented 11 months ago

Yeah Sure, I will be more than happy to help.

anonymous-atom commented 11 months ago

I was having issue with the dataset generation part earlier, Are you having issue in there too ? For fine-tuning I can help you out, l will be interested to get into details.

Thank You & Clear Skies!

aneet-javis commented 11 months ago

@anonymous-atom Is it possible to finetune it for document extraction purpose? Is it able to extract texts from image well? How would the prompt look like if output is desired in json format? Please let me know if you have any thoughts.

anonymous-atom commented 11 months ago

@BYC-Sophie I was finally able to pretrain the model on our custom dataset, let me know if you still need help. I will also post the workflow in a while.

anonymous-atom commented 11 months ago

@aneet-javis I won't recommend it using for that purpose, but yeah you can give it a try. For you second question regarding the output in desired json format you can refer here: OpenAI

aneet-javis commented 11 months ago

@anonymous-atom Thanks for your reply. Please do share your workflow. Would be helpful. I'm trying to fine-tune using this. For some reason I was getting OOM error on a 24GB GPU even when batch_size_per_device & grad_accu_step were both set to 1. Please let me know incase you have any clue about this issue. Thanks

YiDa858 commented 11 months ago

Hi, I'm new in this field. I now have a dataset for fine-tuning the llava, but I don't sure if I need to pre-train as this, or I just need to run the finetune script. Looking forward to your reply!

AI-Aether commented 11 months ago

Hi @anonymous-atom If possible could you please share the workflow. It would be a great help. Thank you.

ilyesbk11 commented 10 months ago

hello @anonymous-atom If possible could you please share the workflow. It would be a great help. Thank you.

Kikzter commented 9 months ago

Hi @anonymous-atom ,

Can you teach me how to do it? it would be of great help. My email address : md.nayeem18899@gmail.com

Thank you

Kikzter commented 9 months ago

@BYC-Sophie I was finally able to pretrain the model on our custom dataset, let me know if you still need help. I will also post the workflow in a while.

@anonymous-atom Hi brother, could you please help me out with the fine tuning part of this model. gmail: md.nayeem18899@gmail.com. Please let me know when can we connect on this.. Thank you