search

sijohndevoteam / falcon-llm-gke-cluster

4 stars 1 forks source link

Question about Synology NAS/Cloud solution #1

Open nekiee13 opened 1 year ago

nekiee13 commented 1 year ago

It is less of a question and more of a plea - can you describe a method to run this app locally (preffered) or on local Synology NAS/Cloud ? I don't have coding background...

sijohndevoteam commented 1 year ago

Are you looking to run this in local for learning purposes ? or to run for any production use cases ?

sijohndevoteam commented 1 year ago

The thing is running the small models like falcon7b is possible in mocal machine. But still it consumes a lot of CPU and Memory. If you still want it i can write a high level step

nekiee13 commented 1 year ago

Just for learning purposes. I hardly cope with python, this is way out of my reach.. I would like to assess capabilities of RAG, and your software seems to enable that. To my understanding, local virtual cloud can be made locally via virtualbox? I've got Win11 OS, nVidia RTx4070 12Gb VRAM, 64 Gb RAM.

nekiee13 commented 1 year ago

Any help would be greatly appreciated..