Closed dsikka closed 9 months ago
The following config file will create an openai server:
num_workers: 1 num_streams: 1 integration: openai endpoints: - task: text_generation model: "hf:mgoin/TinyStories-1M-ds" kwargs: {"continuous_batch_sizes": [4, 8, 16], "force_max_tokens": True}
Summary
Testing
The following config file will create an openai server: