search

tmokmss / dify-on-aws-cdk

Dify on AWS managed and easy
MIT License
45 stars 7 forks source link

SSE won't work with Amazon API Gateway #1

Closed tmokmss closed 1 month ago

tmokmss commented 1 month ago

Currently respong streaming using server-sent events (SSE) is not working due to API Gateway limitation (ref).

Instead of streaming, responses are returned once the backend api returned all of the chunks. It does make the UX worse but still usable. We'll keep the current design and see if APIGW has other kind of issues.

tmokmss commented 1 month ago

Another cons of API Gateway is 30 seconds timeout. (the quota can be increased but...) The recent models are so fast that a response is usually returned within 30s, but there can be some usecases that require more response time.

Possible improvement: CloudFront -> Lambda FURL -> Dify API on Lambda (with response streaming and LWA) (#3)

We can use response streaming even on Python, using Docker runtime (example). Using Lambda, the coldstart might become a problem though.

edit) Lambda cold start time for api container is 32s on average... (10s timeout + 22s actual cold start)

Then the obvious fallback is: ALB -> Dify API (#5)

Maybe we can put CloudFront in front of ALB. In that case, ALB can be protected by this mechanism.

If we use an ALB, we do not have to use APIGW even in web frontend because there's little benefit to do that.