search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.94k stars 169 forks source link

[TextGeneration] Add `model_path` alias #1505

Closed dsikka closed 6 months ago

dsikka commented 6 months ago

The following now work:

from deepsparse import TextGeneration

model_1 = TextGeneration(model="hf:mgoin/TinyStories-1M-ds")
model_2 = TextGeneration(model_path="hf:mgoin/TinyStories-1M-ds")

from deepsparse import Pipeline

pipeline = Pipeline.create(
     task="text_generation",
     model_path=model_path,
     engine_type="deepsparse",
     internal_kv_cache=True,
     continuous_batch_sizes=[2, 4]
 )

pipeline = Pipeline.create(
     task="text_generation",
     model=model_path,
     engine_type="deepsparse",
     internal_kv_cache=True,
     continuous_batch_sizes=[2, 4]
 )