search

GoogleCloudPlatform / generative-ai

Sample code and notebooks for Generative AI on Google Cloud, with Gemini on Vertex AI
https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview
Apache License 2.0
6.72k stars 1.8k forks source link

[Bug]: Tuning Gemini with 500 rows takes endless hours and never stops #656

Open sidoncloud opened 4 months ago

sidoncloud commented 4 months ago

File Name

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/tuning/gemini_supervised_tuning_qa.ipynb

What happened?

Tuning Gemini with 500 rows takes and basic minimum hyperparam values takes 3+ hours if you cancel the job after 3 hours else it goes on and on forever and you'll see a hefty cloud bill - This is something all google cloud paying users need to be made aware of before they try out something that does not work.

Relevant log output

No output

Code of Conduct

Ali-Razmjoo commented 2 months ago

I launched two different tunings, with a 660MB and 1100MB dataset, 18 hours ago. The first one is now 5% processed, and the second one is still validating the dataset. Now, the Gemini tuning is in preview, and it doesn't cost anything, but I am unsure if I would do it if it was billed. It's heartbreaking how long it takes.