Events Only Sent On Server Exit

[mando238]

I searched through the issues but couldn't find anything quite like this - though https://github.com/fanout/django-eventstream/issues/49 may be related. I have setup django-eventstream per the readme (with channels setup) and the client is able to connect to event channels. It is a very similar setup to the chat example.

However, calling send_event seemingly does nothing, no events are sent - at least it seems that way- until the server exits. Whenever the server exits (or restarts in the case of the hot loading django dev server) all of the events that should have been sent will be sent at once and received by the client. I cannot find another way to make the server finally send all these events that it is (presumably) keeping cached aside from restarting it. It seems as if there is some sort of sending buffer that needs to be flushed?

This has been tried with Daphne, Daphne with HTTP 2.0 support, and Django's runserver interestingly all produce the same results. To be thorough - though it seems as though events are never sent by the server - I've also swapped between a webpack dev server on the frontend and an nginx server - both producing the same result.

Versions: Django-Eventstream: 3.1.0 Django: 3.0.3 Daphne: 2.5.0 Python: 3.7.5

Is there some sort of hidden buffer setting? I searched through the code for this repo and the related GRIP repos but didn't see any smoking gun...

[mando238]

Additionally - if I do something like try to close pubcontrol myself. send_event(...) get_pubcontrol().wait_all_sent() get_pubcontrol().close()

It has no effect, still nothing is sent until the server is restarted or existed. So it looks like this is not related to the atexit piece of pubcontrol?

[jkarneges]

I just tried the chat example using Django 3.0.3, Daphne 2.5.0, Channels 2.4.0, and Python 3.6.9, and it worked fine.

Can you try the chat example the same way, but with your Python 3.7.5?

[mando238]

Tried the chat example with Python 3.7.5 and its functional (events are received immediately as expected).

I've been trying to find the differences between the chat example and mine. The only 2 remaining is that the static files are defined separately (react frontend) and hosted separately as indicated in the first post. With Django providing the backend API server. I've used other SSE libraries with other frameworks though with the same overall setup and never had an issue like this, and again I do receive all of the SSE's.. just when the server restarts.

The other difference is that Djoser is used for authentication. It doesn't seem to be interfering though since the client can still connect to the channel and ultimately receive SSEs.

[jkarneges]

I wonder if Djoser is buffering the response?

[mando238]

Getting close. So I took the Chat example that was apparently working (when the static files are served from Django) and pointed my Frontend to use its API. And the same issue manifested. No receiving anything until the server is stopped or restarted, then all events are received. So it appears as though there are no issues with when Django hosts static files, but when hosting them on a separate static server (as is common) this issue comes up. I'll see if I can find a configuration for either the static server OR django that gets around this. Though I haven't encountered this with other SSE libraries, so I wonder if its an issue with a dependency for this project...

[jkarneges]

Hmm, so the chat example works when connected to via the client code served by the app, but not when connecting to it from client code not served by the app?

Maybe a CORS issue?

[mando238]

Well I've modified every CORS setting I can find, no luck - and again normal REST API calls for the same server have no issue. I also tried the settings in the readme for allow origin and withcredentials, no effect.

Is there a way to set custom headers in the SSE response? I'm noting the response header from events has

Content-Encoding: gzip
content-type: text/event-stream
transfer-encoding: chunked

I've seen with other projects where both gzip and chunked transfer encoding can cause issues. With those we could user custom headers to enable/disable them in the response - is there a way to do it with this library?

---

EDIT: Actually I may have indirectly figured this out, but the above question does stand because its definitely the more ideal solution. I eventually served the frontend out an nginx container with the following parameters for its "proxy pass" (i.e. handling traffic with the backend server):

         proxy_buffering off;
      proxy_cache off;
      chunked_transfer_encoding off;

And it worked. However, I can't get it to work in any other setup. These parameters have corresponding headers that can be sent in an SSE response so the above question about setting them stands.

For reference, a working response header consisted only of:

Access-Control-Allow-Origin: *
Cache-Control: no-cache
Connection: close
Content-Type: text/event-stream
Date: Tue, 04 Aug 2020 01:00:06 GMT
Server: nginx/1.17.5

While a not working one had the additions of

Content-Encoding: gzip
transfer-encoding: chunked
Vary: Accept-Encoding
x-accel-buffering: no
X-Powered-By: Express

end EDIT

Also, when running with anything besides runserver I'm also getting a lot of RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one When I try to send events with send_event("game-38", 'message', "testmessage") or any similar message. But I'm trying not to mix issues so I'll just leave it at that.

[jkarneges]

If Nginx is in front, you definitely need to disable buffering.

I'm not seeing gzip when testing directly against Daphne. Maybe that's coming from Nginx also?

[mando238]

Yes I see the need for disabling the compression/buffering and I can remove it with Nginx configs like above and then for webpack by disabling compression in their config.js. All of the setups are now working, so this issue is now solved! I can put a couple succinct sentences together for a note in your readme somewhere if you would like. Just to help out anyone else who's setting this up.

A final note, most of the above configurations can be set in request/response headers and will be honored by the servers. I guess my question, or maybe now more of a suggestion is, how can I set the response headers for /events/? Easy enough to do for a normal django view, but the view for events is buried within the django-eventstream package. I suppose one could write their own little middleware, but that seems a bit excessive for this. I suggest a parameter than can be passed into django eventstream, maybe a setting in settings.py that allows one to pass in custom response headers that will be attached to the response to connecting clients.

[jkarneges]

Ah, I just remembered django-eventstream does set X-Accel-Buffering: no to supposedly disable Nginx buffering. Maybe that part is working, and the issue is compression?

[mando238]

Thats very possible. Thats partly why I recommend having an easy way for people using this to set the response headers. It seems for each server there's something that makes in unhappy with SSE's. This time nginx required the 3 lines above while webpack only required turning off Gzip. So if people can just set their own response headers fpr /events to whats works for their setup we can get around the issue... Maybe I'll submit a PR if I find the time.

Thanks for you help with this!