This release contains 2 bug fixes and 2 documentation improvements.
đ Bug Fixes
Make Gateway load balancer stream results (#6024)
Streaming endpoints in Executors can be deployed behind a Gateway (when using include_gateway=True in Deployment).
In this case, the Gateway acts as a load balancer. However, prior to this release, when the HTTP protocol is used, the Gateway would wait until all chunks of the responses had been streamed from the Executor.
Only when all the chunks were received would it send them back to the client. This resulted in delays and suppressed the desired behavior of a streaming endpoint (namely, displaying tokens streamed from an LLM with a typewriter effect).
This release fixes this issue by making the Gateway stream chunks of responses from the forwarded request as soon as they are received from the Executor.
No matter whether you are setting include_gateway to True or False in Deployment, streaming endpoints should give the same desired behavior.
Fix deeply nested Schemas support in Executors and Flows(#6021)
When a deeply nested schema (DocArray schema with 2+ levels of nesting) was specified as an input or output of an Executor endpoint, and the Executor was deployed in a Flow, the Gateway would fail to fetch information about the endpoints and their input/output schemas:
from typing import List, Optional
from docarray import BaseDoc, DocList
from jina import Executor, Flow, requests
class Nested2Doc(BaseDoc):
value: str
class Nested1Doc(BaseDoc):
nested: Nested2Doc
class RootDoc(BaseDoc):
nested: Optional[Nested1Doc]
class NestedSchemaExecutor(Executor):
@requests(on='/endpoint')
async def endpoint(self, docs: DocList[RootDoc], **kwargs) -> DocList[RootDoc]:
rets = DocList[RootDoc]()
rets.append(
RootDoc(
text='hello world', nested=Nested1Doc(nested=Nested2Doc(value='test'))
)
)
return rets
flow = Flow().add(uses=NestedSchemaExecutor)
with flow:
res = flow.post(
on='/endpoint', inputs=RootDoc(text='hello'), return_type=DocList[RootDoc]
)
...
2023-08-07 02:49:32,529 topology_graph.py[608] WARNING Getting endpoints failed: 'definitions'. Waiting for another trial
This was due to an internal utility function failing to convert such deeply nested JSON schemas to DocArray models.
This release fixes the issue by propagating global schema definitions when generating models for nested schemas.
đ Documentation Improvements
Remove extra backtick in create-app.md (#6023)
Fix streaming endpoint reference in README (#6017)
đ¤ Contributors
We would like to thank all contributors to this release:
Release Note
This release contains 2 bug fixes and 2 documentation improvements.
đ Bug Fixes
Make Gateway load balancer stream results (#6024)
Streaming endpoints in Executors can be deployed behind a Gateway (when using
include_gateway=True
inDeployment
).In this case, the Gateway acts as a load balancer. However, prior to this release, when the
HTTP
protocol is used, the Gateway would wait until all chunks of the responses had been streamed from the Executor.Only when all the chunks were received would it send them back to the client. This resulted in delays and suppressed the desired behavior of a streaming endpoint (namely, displaying tokens streamed from an LLM with a typewriter effect). This release fixes this issue by making the Gateway stream chunks of responses from the forwarded request as soon as they are received from the Executor.
No matter whether you are setting
include_gateway
toTrue
orFalse
inDeployment
, streaming endpoints should give the same desired behavior.Fix deeply nested Schemas support in Executors and Flows(#6021)
When a deeply nested schema (DocArray schema with 2+ levels of nesting) was specified as an input or output of an Executor endpoint, and the Executor was deployed in a Flow, the Gateway would fail to fetch information about the endpoints and their input/output schemas:
This was due to an internal utility function failing to convert such deeply nested JSON schemas to DocArray models. This release fixes the issue by propagating global schema definitions when generating models for nested schemas.
đ Documentation Improvements
create-app.md
(#6023)đ¤ Contributors
We would like to thank all contributors to this release: