Swagger for Server Sent Events

[bmeck]

It would be nice if Swagger supported Server Sent Events. Which is a Uni-directional alternative for streaming content. This still would fall within REST to my knowledge (in particular due to last seen ID). A more concrete example can be seen at HTML5Rocks

semi-related : #55

[buremba]

+1

[wiltshiretom]

+1!

[casualjim]

I think you can kind of represent it (without tooling support though) like this

paths:
  /:
    get:
      produces: ["text/event-stream"]
      responses:
        default:
          description: example of defining events for sse 
          schema:
            type: object
            x-events:
              added:
                type: object
                properties:
                  message:
                    type: string
                  createdAt:
                    type: string
                    format: date-time
              updated:
                type: object
                  properties:
                    previous:
                      type: string
                    newValue:
                      type: string
                    updatedAt:
                      type: string
                      format: date-time

[elbrujohalcon]

+1

[artem-korolev]

Well. But why to represent it in swagger without tooling support? Does it have any use cases?

[casualjim]

step 1, make the proposal to start the discussion

[ikitommi]

:+1:

[mrhornsby]

:+1:

[casualjim]

parent: #586

[jroper]

I don't think this should be limited to uni directional streaming. While of course, web browsers and many existing HTTP implementations require a request to be fully consumed/produced before sending/reading the response, HTTP itself does not impose this limitation, nor does anything about the SSE protocol, and so there's no reason why two services using an HTTP client/server implementation that supports it can't speak SSE both up and down at the same time.

Another reason why it shouldn't be limited to uni directional streaming is that this and a possible WebSockets streaming extension could share a lot in common, it would be a shame to have two completely different ways of specifying an SSE stream vs a WebSocket event stream.

[dolmen]

@jroper:

• With HTTP you don't have a bidirectional link over a single TCP connection: the client speaks first and the server answers.
• A link over WebSockets is not HTTP anymore, so I think it is out of scope for Swagger/OpenAPI.

Personally, for a text oriented protocol over a bidirectionnal link (TCP or WebSocket) I would use JSON RPC 2.0 for fully asynchronous dialog, not Event Source.

[jroper]

With HTTP you don't have a bidirectional link over a single TCP connection: the client speaks first and the server answers.

That's not actually true. There is nothing in the HTTP spec that says that a server has to fully consume a client request before it starts sending the response, in fact, the spec actually has provisions that allow for this to be done, from section 8.2.3:

If an origin server receives a request that does not include an Expect request-header field with the "100-continue" expectation, the request includes a request body, and the server responds with a final status code before reading the entire request body from the transport connection, then the server SHOULD NOT close the transport connection until it has read the entire request, or until the client closes the connection.

This implies that the server is allowed to send a response before the entire request body is read. So bidirectional communication over a single TCP connection is definitely allowed. A number of asynchronous HTTP servers (eg akka-http, Netty, and Play framework, which I'm a maintainer of) do explicitly support this too.

[dolmen]

Please stop those polluting +1 and use instead built-in reactions on the original submission. Do you know that you can even delete your own useless comments?

[jroper]

Getting everyone in your company to +1 an issue is the dumbest strategy ever, as a maintainer of a number of open source projects, when I see lots of noise on an issue, and then I see most of that noise is from one company, it leads me to believe that that company is the only company that wants the feature, which leads me to deprioritise the issue. Please streamdata.io, there are serious people here who are serious about wanting to see this become part of the spec and want to have serious discussion about it, the only thing you're doing is hurting our efforts.

[darrelmiller]

Now that V3 supports OneOf in JSON Schema, I'm not exactly sure what it is that people want more from OpenAPI in order to describe SSE payloads. Perhaps if someone could describe the challenges currently faced then we would have a better idea as to what needs enhancing.

[allandenis38]

Sorry @jroper, our intent was not to hurt your efforts at all. In order to show that you are also serious people ;-) we will remove those useless +1 and use reactions instead.

[jroper]

@darrelmiller SSE events have a few properties, and these are not JSON properties, they are part of the SSE protocol. Most interesting to OpenAPI is the event and data properties. The data property is the payload, this would typically contain the event data as JSON, or maybe as a plain text string. The event field however contains the events type. So, to support idiomatic SSE, OpenAPI would have to allow specifying multiple event types, most likely with a schema for each.

A non idiomatic approach could be taken where the OneOf JSON schema support is used with no event type, but even then there are a few questions that should probably be answered by the spec to allow tooling to generate interoperable implementations. For example, how should an OpenAPI spec indicate that SSE is being used? By listing text/event-stream in the produces? What if the spec lists both text/event-stream and application/json in the produces? How should a tool interpret that? Because in one case, a client code generator will need to generate a method that returns the object directly, and in the other it would need to return a stream. And if text/event-stream is listed in produces, how does the client then know that the contents of text/event-stream of the data payload is application/json, or text/plain?

[dolmen]

In my use case I want to return text/event-stream for status 200, but application/json for other statuses (structured error reporting for bad usage of the API or internal errors). For now I'm specifying this by cheating with the headers property of a Response object. But I do not expect Swagger clients to properly handle that.

      produces: ["text/event-stream","application/json"]
      responses:
        '200':
          description: Stream is starting
          x-produces: ["text/event-stream"]
          headers:
            'Content-Type':
              type: string
              enum: ["text/event-stream"]
        default:
          $ref: '#/responses/DefaultError'

[whitlockjc]

What is really being discussed (in my opinion) is how one would use OAS to describe a streaming API, regardless of which technology (SSE, WebSockets, ...) you're using. Since all of these technologies all use HTTP, the OAS already has the primitives in place so we just need to decide is at what level we want to add support to the OAS, and what that looks like.

In a recent discussion with @darrelmiller, the main question I needed answered was to the following: "When describing a streaming API via OAS, is the response schema suppose to describe each "chunk" of the response or for the response as a whole?" This came up when looking at the OAS for the Kubernetes API. For their OAS usage, the response schema was written to describe each chunk in the stream as a JSON Object. To me, this is intuitive but from a tooling perspective, there is no way for the tooling to know that this is a streaming API and would potentially have an issue where it could fail to validate appropriately in all situations.

My hope in answering this is for consistency, so OAS users know that there is a well-defined approach for documenting streaming APIs.

[dolmen]

@whitlockjc For SSE we want to be able to describe multiple types of chunks (called events), each being identified by an event name. So if events are sent are JSON, the spec format should allow to describe the schema of the message for each event name.

[darrelmiller]

@dolmen Would it be sufficient to be able to indicate that the payload schema only applies to a "chunk" and then use oneOf in JSON Schema to define all of the events? The other option might be to have a chunkSchema property that is an array of schema objects. The end result is effectively the same though. At least by using oneOf a JSON Schema validator could validate all the chunks using a single schema.

[allandenis38]

@darrelmiller In case of SSE, an event contains non-JSON properties and this is true for every event/chunk. So applying a JSON validation to the entire event would not make sense but rather the value of an SSE property.

[darrelmiller]

@allandenis38 Unfortunately OpenAPI is currently limited in its ability to describe non-JSON payloads. If we can fix that then we would be a step closer to being able to describe things like SSE. I don't believe it would be wise to try and address the description of SSE events without first addressing the larger issue.

[whitlockjc]

@dolmen I agree. Many of the WebSocket APIs I've seen can have multiple message types for the same event.

@allandenis38 While @darrelmiller is right about the limitations of OAS describing non-JSON payloads, for unstructured data or data that is composed of JSON primitives (composite values, etc.), you could still use OAS to describe your types. For example, OAS won't help much with XML but if you are sending back strings that follow a particular pattern or regex, OAS might still be able to help.

[jroper]

@darrelmiller while yes OpenAPI is limited in its ability to describe non-JSON payloads, note that the SSE event type is quite a different problem to say, wanting to describe an xml or protobuf payload, since the payload of an SSE event will generally still be JSON, it just has this other attribute outside of the JSON that impacts the interpretation of the data. So I don't think that fixing Swaggers ability to describe non-JSON payloads will necessarily gain us a useful solution to providing support for idiomatic SSE usage.

For example, one way to support SSE's event type would be to augment the discriminator property in a schema, where you could give it a special value, say discriminator: $sseEvent, rather than discriminator: petType, and that will use the SSE event field, rather than the petType field in the JSON, to discriminate between different types of events. So then on the wire, an SSE stream of pets (from the pet store example) might look like this:

event: Cat
data: {"name":"Mittens"}

event: Dog
data: {"name":"Fido"}

And the first event would be deserialized as a Cat, the second as a Dog.

[fizxmike]

My use case (and maybe others') is REST + SSE updates. So, given an API endpoint (say, get all X or get X by id), I would like to also subscribe to the X subject/topic. Maybe this can help in your design.

Should we just be moving to REST over WebSockets? I think OpenAPISpec is still compatible so long as "data" is JSON. It seems that http(s) GET,POST, etc. could just be replaced with "event" types... Thoughts?

[stefano-xy]

As mentioned above, I think it's possible to do it if we generalize the support for non-JSON data. As simple types like integer and strings are the same when encoded in "text" instead of JSON, the only difference is visible in case of object and array. We could add format also to them, enabling things like:

/events:
  get:
    responses:
      200:
        content:
          text/event-stream:
            schema:
              type: array # eventually also stream as new type, to mean unboundness, or a new parameter of array
              format: chunked # or whatever other keyword to mean chunks separated by empty lines
                              # another use could be jsonlines, one line per JSON object, with no [ , ]
              items:
                type: object
                format: text # assuming this means "key: value", one per line, for each key 
                             # or we may introduce yaml !! 🥇
                required:
                  - id # not strictly required by SSE, but it's an example
                  - event # not strictly required by SSE, but it's an example
                  - data
                properties: # also possible to model with oneOf+discriminator
                  id:
                    type: integer
                    description: my event id
                  event:
                    type: string
                    description: my event type
                  data:
                    type: object # object in case we have a complex structure, otherwise string, integer, ..., as usual
                    format: json # JSON is the default, but we may have different format as proposed above for the array
                    required:
                      - a
                      - b
                    properties: # as usual
                      a: # ...
                      b: # ...
                      c: # ...

Note that this is not strictly limited to SSE. Enabling format for array and object may have this and other applications. SSE protocol is modeled manually.

[barrycaceres]

This thread is a few years old now and I see it was referenced in August of 2019 and I have not seen anything in the Open API spec that does support SSE (unless I am missing it).

For me, I use SSE as an option when the client is uploading a large file to be processed by the server. The caller controls this with the Accept header.

If the caller sends in no Accept, or Accept: */* or Accept: application/json (or similar) then the operation works the same as Swagger/Open API would normally expect.

• on a 200 it would return something like a FileProcessedResponse object (as defined in my schema) to describe the results of processing the file that was uploaded.

• on a 4xx or 5xx it would return an ErrorResponse to describing why the file could not be processed

Typically if the file could only be partially processed (i.e.: it contains both good records and bad records) the response is a 200 with the FileProcessedResponse object describing how many succeeded and what failed.

That said... in the event the client sends Accept: text/event-stream instead, then I produce the following events....

progress -- which contains the FileProcessedResponse in the data segment but indicating values for what has been done "so far"

completed -- same as the 200 response above, this is the final resultant FileProcessedResponse object

aborted -- same as the 4xx or 5xx response above -- the data segment of the event is my ErrorResponse object indicating why things ultimately failed.

So the SSE events for the end point very much correlate to the normal the Swagger/OpenAPI definition of the 200, 4xx and 5xx responses. So I am wondering if following this model would make it easier to describe things in OpenAPI. Something as simple as a providesSSE flag which can be set to true or false (defaults to false) and then for each response code correlate it to an SSE event type instead... so the SSE response is always a 200 but the actual events indicate the success and failure.

[dkirrane]

Duplicate https://github.com/OAI/OpenAPI-Specification/issues/770

[davix]

I think you can kind of represent it (without tooling support though) like this

paths:
  /:
    get:
      produces: ["text/event-stream"]
      responses:
        default:
          description: example of defining events for sse 
          schema:
            type: object
            x-events:
              added:
                type: object
                properties:
                  message:
                    type: string
                  createdAt:
                    type: string
                    format: date-time
              updated:
                type: object
                  properties:
                    previous:
                      type: string
                    newValue:
                      type: string
                    updatedAt:
                      type: string
                      format: date-time

Hi, it seems still uncertain whether or when it will be supported, but we may need some workaround right now.

Does anybody know workaround how to describe SSE with swagger file?

I can give up its functionalities (like code generation) and just use swagger file as a document.

I tried the method in earlier comment, but the generated output doesn't look like a SSE output.

Here's generated:

Here's expected:

event: Cat
data: {"name":"Mittens"}

event: Dog
data: {"name":"Fido"}

[ihakh]

This thread is a few years old now and I see it was referenced in August of 2019 and I have not seen anything in the Open API spec that does support SSE (unless I am missing it).

For me, I use SSE as an option when the client is uploading a large file to be processed by the server. The caller controls this with the Accept header.

If the caller sends in no Accept, or Accept: */* or Accept: application/json (or similar) then the operation works the same as Swagger/Open API would normally expect.

• on a 200 it would return something like a FileProcessedResponse object (as defined in my schema) to describe the results of processing the file that was uploaded.
• on a 4xx or 5xx it would return an ErrorResponse to describing why the file could not be processed

Typically if the file could only be partially processed (i.e.: it contains both good records and bad records) the response is a 200 with the FileProcessedResponse object describing how many succeeded and what failed.

That said... in the event the client sends Accept: text/event-stream instead, then I produce the following events....

progress -- which contains the FileProcessedResponse in the data segment but indicating values for what has been done "so far"

completed -- same as the 200 response above, this is the final resultant FileProcessedResponse object

aborted -- same as the 4xx or 5xx response above -- the data segment of the event is my ErrorResponse object indicating why things ultimately failed.

So the SSE events for the end point very much correlate to the normal the Swagger/OpenAPI definition of the 200, 4xx and 5xx responses. So I am wondering if following this model would make it easier to describe things in OpenAPI. Something as simple as a providesSSE flag which can be set to true or false (defaults to false) and then for each response code correlate it to an SSE event type instead... so the SSE response is always a 200 but the actual events indicate the success and failure.

I think this is a very good point to start three type of JSON OK event for start and progress and complete and one or more type of ERROR event

step 1, make the proposal to start the discussion

the proposal of @barrycaceres seems ok to implement

[davide1995]

Any update on this? I don't want always to use EventSource.addEventListener('message') manually. I would like to expect that when I call this.service.myObservable().subscribe(), it works in the right manner even with text/event-stream

[philsturgeon]

Now we have callbacks and websockets I'd call this done.

[jhpratt]

@philsturgeon Callbacks and websockets are completely different than SSE.

[philsturgeon]

I meant webhooks but yep there's a bunch of server emitted things supported and there's plenty of other issues describing other forms #523 https://github.com/OAI/OpenAPI-Specification/issues/770

Let's talk about Server Push on the Server Push issue and anything else can have a more specific issue.

[jhpratt]

Server sent events also isn't the same thing as webhooks or server push. Not to mention the fact that this thread is where all discussion has taken place.

In all seriousness, are you familiar with these various technologies? They have very little in common, and use case is not one of them.

[philsturgeon]

There’s a lot of duplicates floating around so I’ve closed #1499 instead. We can give this another 5 years to see if it makes any progress. ;)

[philsturgeon]

The point isn’t if they’re the exact same use case or not, more that trying to support every single type of everything makes OAS a kitchen sink spec, and there’s clearly not much progress or interest. It’s never come up at work either, and we get every type of request you could imagine.

As you’re clearly really interested in it, I’d suggest making a proposal and seeing how far you can get it. Nobody will do it, as can be seen from the inactivity so far.

[enhsaihan-d]

I'm quite baffled how this hasn't been implemented yet with how SSE is becoming more popular with rise of reactive systems

[philsturgeon]

I'm quite baffled how this hasn't been implemented yet with how SSE is becoming more popular with rise of reactive systems

It’s not been added because you haven’t added it.

https://github.com/OAI/OpenAPI-Specification#participation

[ahopkins]

+1

[darrelmiller]

The best way to get this formalized in the OpenAPI specification is for the users of SSE to get some agreement on a standard way of describing the event stream. There are a couple of reasonable suggestions so far, but let me add a slight variant to the one proposed by @ragazzojp .

/events:
  get:
    responses:
      200:
        content:
          text/event-stream:
            schema:
              type: array 
              format: event-stream
              items:
                oneOf:
                    - type: object
                      properties:
                         id:
                           type: string
                         event: 
                           const: cat
                         data: 
                           type: object
                           properties: 
                             name:
                               type: string
                    - type: object
                      properties:
                         id:
                           type: string
                         event: 
                           const: dog
                         data: 
                           type: object
                           properties: 
                             name:
                               type: string

The OpenAPI specification has precedent for using JSON schema to model formats that are not JSON (e.g. form -url-encoded). I don't see it as outside the realms of possibility to use JSON schema to describe the shape of each event. A constant value can be used for the event property if events are being sent. Any complex structure described by the data field would need to be be serialized on a single line. These types of rules could be defined in the OpenAPI specification for mapping JSON Schema to the text/event-stream media type. What would be event better is if this mapping were done in a separate specification and then the format keyword used to indicate the use of a special serialization. Although, I'm not sure how enthusiastic the JSON Schema folks would be about this.

I think the community who are interested in SSE can get 90% of the way towards a solution that describes server sent events using OAS as it currently stands. Once there is agreement on the best way to do that, standardizing that approach in the OAS spec may help with adoption and tooling.

[ahopkins]

Why closed?

[darrelmiller]

There have a been a couple of suggestions of how to model SSE in OpenAPI. SSE is just a long polling HTTP GET with a payload that supports sending discrete elements of data.

[twhittock]

https://github.com/OAI/OpenAPI-Specification/issues/396#issuecomment-894718960 is this the canonical way to represent SSE, then? And we bypass any tooling to write the handler manually?

[philsturgeon]

After years of inactivity I invited the community interested in SSE to get involved in May 2021. Nobody did.

Then in August 2021 @darrelmiller invited the community interested in SSE to participate. Nobody did.

Be the change you want to see. 🙌🏻

https://github.com/OAI/OpenAPI-Specification#participation

[twhittock]

I'm not sure what it means for me to be the change I want to see @philsturgeon - is there a specific issue with considering the proposal in https://github.com/OAI/OpenAPI-Specification/issues/396#issuecomment-894718960 canonical for SSE, as I suggested?

The participation section you link to states that we should "Check the issues and pull requests to see if someone has already documented your idea or feedback on the specification", which surely is what is happening when people contribute to this issue, but it keeps getting closed by members of the project.

Perhaps this falls under "Not all feedback can be accommodated and there may be solid arguments for or against a change being appropriate for the specification" - if so, why not explain that Open API will not support SSE and give those solid arguments against it?

I genuinely don't know what you expect to see.

[philsturgeon]

I was pointing you to the weekly calls:

The TSC holds weekly web conferences to review open pull requests and discuss open issues related to the evolving OpenAPI Specification. Participation in weekly calls and scheduled working sessions is open to the community. You can view the entire OpenAPI technical meeting calendar online.

Swing by, talk about what you want to do, and disucss a potential proposa.

There are examples above of how it could be done to start from, so perhaps that can be the basis of a x-foo draft proposal. We can talk that through on the weekly call.

[giowe]

any update?

[twhittock]

No, I attended one of the calls which was cut off early due to key people not attending and no proposals were entertained. I get the feeling it's really out of scope for this project, so I just gave up and rolled my own.