This PR enables the server to use the new refactored pipeline, while also supporting the legacy pipeline for tasks which have not yet been migrated
Key Changes/Additions:
add a Pipeline.from_config function to the new pipeline.py file. This function is basically identical to the old function with some slight changes
Update Pipeline.create(...) to parse kwargs such that only the supported kwargs for v2 are passed in (for example: executor is not used in v2 but is needed for v1 endpoints/pipelines)
Update the server.predict function to handle async calls and use the new pipeline's run_async method. If the run_async method is not available, as is the case for v1, then the pipeline is called how it was previously
Update tests to be reflective of the new changes
Testing
Updated/ran all server tests
Ran inference locally, using the sample config file shown below. This sample includes a task using the old pipeline as well as text_generation, which uses the new pipeline
{'created': '2023-12-11T23:11:31.675603', 'prompts': ['The sun shined', 'Oh hello!'], 'generations': [[{'text': 'through the tree branches, and it looked like a brilliant green star. Its light was so much br', 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': 'in all the clouds. They weren’t black, or blue, or red, but the', 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': 'on us in the hot afternoon. A few small birds were called out (the pigeons,', 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': 'on her,\nIn the warmest weltan of colors.\nThe breezes ble', 'score': None, 'finished': True, 'finished_reason': 'length'}], [{'text': 'I’m on a ship!\r\nWow, what a surprise! On an enormous', 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': "We're getting ready for our first day at U-Tam in four days time. I", 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': "Well you've probably heard of the Pagan Festivals in Europe but had never thought to", 'score': None, 'finished': True, 'finished_reason': 'length'}, {'text': "\nLet's just start from the beginning – and where were we? Now it wasn't", 'score': None, 'finished': True, 'finished_reason': 'length'}]], 'input_tokens': None}
Using updated print statements, we can see the engine using batch sizes of 2 and 4
Summary
Key Changes/Additions:
Pipeline.from_config
function to the new pipeline.py file. This function is basically identical to the old function with some slight changesPipeline.create(...)
to parse kwargs such that only the supported kwargs for v2 are passed in (for example:executor
is not used in v2 but is needed for v1 endpoints/pipelines)server.predict
function to handle async calls and use the new pipeline'srun_async
method. If therun_async
method is not available, as is the case for v1, then the pipeline is called how it was previouslyTesting
Launch the server:
Endpoints with both models are available and can run using their respective pipelines.
With continuous batching
internal_kv_cache
as False for nowSend a request:
Output: