Beanow
commented
3 years ago Had a similar, related issue with "streaming" requests.
Use case
Using MJPEG-streamer to expose a webcam.
The way this "stream" is implemented (http handler here), is by sending a
Content-Type: multipart/x-mixed-replace;boundary=...
Multipart response body, and sending an infinite amount of
Content-Type: image/jpeg parts as frames, as long as the connection is open.
The upstream server flushes response body data whenever it can buffer a frame.
This particular software exposes such a stream on GET /?action=stream.
Expected result
Louketo is able to proxy this request as an unbuffered, infinite response body.
Actual result
While buffering is not an obvious issue here, the stream is closed after 10 seconds.
A workaround for this would be to set --server-write-timeout=0s, to disable the timeout.
Reproducing
Requirements
- Docker + Docker Compose
- A V4L2 device on
/dev/video0. - A testing authentication OIDC client, which allows
http://localhost:3000/*redirect urls.
Start up this compose file with docker-compose up.
# docker-compose.yml
version: '3.7'
services:
oidc-gate:
image: quay.io/louketo/louketo-proxy
command: >-
--server-write-timeout=0s
--upstream-url=http://webcam:80
--listen=:3000
--enable-default-deny=true
--discovery-url=https://example-keycloak/auth/realms/local-testing
--client-id=local-app
--client-secret=12345678-1234-1234-1234-123456789012
--encryption-key=AgXa7xRcoClDEU0ZDSH4X0XhL5Qy2Z2j
ports:
- 3000:3000
webcam:
image: sixsq/mjpg-streamer
devices:
# Streams from a V4L2 camera. Like a laptop/usb webcam.
- /dev/video0
ports:
# Runs on :80 internally,
# we expose 8080 for testing without auth.
- 8080:80GET http://localhost:8080/?action=streamshould serve the upstream server's MJPEG stream correctly.GET http://localhost:3000/?action=streamshould first require authentication, then stream indefinitely.- Change the
--server-write-timeout=0soption to--server-write-timeout=3sanddocker-compose upthe changes. GET http://localhost:3000/?action=streamwill end the stream after 3 seconds.
Additional Information
Other reverse proxy setups, do not require changing such a timeout in the first place.
For example I've also proxied this through Caddy. There the reverse_proxy directive needs an option flush_interval -1.
The proxy buffers responses by default for wire efficiency:
- flush_interval is a duration value that defines how often Caddy should flush the buffered response body to the client. Set to -1 to disable buffering.
This is closer to the problem @pahrens is observing, because Caddy will disable buffering of the response this way. Setting this to -1 also avoids any timeout issues. Caddy will just happily send an infinite response body.
Which makes a lot of sense, because no buffers means no latency, no congestion (for the proxy), no memory hogging... It becomes a problem of the upstream and client to set sane timeouts.
Gatekeeper prevents streaming output
Summary
I have a PHP based website that has a feature that sends already partly output to the requester. Below you can find a simple php example for this logic.
When using Gatekeeper before this website with this feature, only the complete output is displayed at the requester and not already the parts of the output.
Environment
Expected Results
I would like to have a way to configure gatekeeper to return already parts of the output of a website instead of the complete output.
Actual Results
see summary
Steps to reproduce
Additional Information
During my research, I have found out that there is a similar issue related to Nginx, were the internal buffer from Nginx is preventing this kind of logic to work. Maybe it's a similar reason here as well.