search

Pan-ML / panml

PanML is a high level generative AI/ML development and analysis library designed for ease of use and fast experimentation.
MIT License
113 stars 16 forks source link

Keep model in memory #36

Open vsraptor opened 1 year ago

vsraptor commented 1 year ago

Is there a way to load the model in memory i.e. when you run a script you just connect to the model instead of loading it every time.

This way you can speed up the startup time !

wanoz commented 1 year ago

Good question, let's look into it!

wanoz commented 1 year ago

Hey, just want to follow up on this question. Not sure if this is what you're after: we can load in the fine tuned model using: ModelPack(model='./results/', source='local')

vsraptor commented 1 year ago

no I meant...something like:

ModelPack(model='gpt2', source='in-mem-connect')

wanoz commented 1 year ago

I see... happy to chat further to discuss how to make this work

vsraptor commented 1 year ago

I think it will be hard... An idea is to have a Server that runs the model and expose let say REST API (.predict(), .embedings() ...) ... or socket...

I found easier way .... just load the model in interactive ipython session and use autoreload for the code i'm testing... this way model is in memory until i quit the session.. and I can change the code w/o reloading the model.