Performance concern on ODataOutputFormatter

[anhduc130]

Assemblies affected ASP.NET Core OData 8.0.12

Description Hi, we have noticed a high latency (200ms - 400ms) spent in ODataOutputFormatter that serializes and writes to response body after the results (around 350 KB) have been returned from the controller.

Our schema is similar like this

[
  {
    "id": "string",
    "rating": {
      "id": "string",
      "count": 0,
      "average": 0,
      "breakdown": {
        "oneStar": 0,
        "twoStars": 0,
        "threeStars": 0,
        "fourStars": 0,
        "fiveStars": 0
      }
    },
    "scores": [
      {
        "type": "string",
        "value": 0
      },
      {
        "type": "string",
        "value": 0
      }
    ]
  }
  ...
]

Here is the obtained profiler

Could you please help us understand what we could do to improve?

[julealgon]

@anhduc130 out of curiosity, do you know how much time it takes to serialize that same payload using Newtonsoft.Json and System.Text.Json?

Also, does it go down significantly if you paginate the results (either on client with $top or on server with server-driven paging)?

[senioroman4uk]

Hi @julealgon, @anhduc130 and I working on the issue together I did some testing on a warmed up application

odata endpoint median response time is ~ 400ms same endpoint without odata (just regular dotnet 6 api controller ) median response time is ~150 ms

I did not test newtonsoft since we don't use it for response serialization

Requests are not using $top, page size is set on a server side (3000)

[anhduc130]

Hi @julealgon, here's the difference when we turn OData endpoint into a regular one which uses the System.Text.Json. It takes around 15ms.

[julealgon]

Thanks @anhduc130 and @senioroman4uk for the detail. Looks like a very valid concern then, hopefully someone on the OData team will take a look.

I'd still suggest, if you can, to try and create the simplest possible repro that still showcases the vast gap in performance. It could be some specific properties in your payloads that are triggering the behavior.

But yeah... regardless, it doesn't look normal at all.

[habbes]

@anhduc130 The OData serializer does more work to ensure the response adheres to OData standards, which contributes to some of the performance difference. That said there are ongoing efforts to improve OData serialization performance.

Here are some things that you can try:

• Mark your IEdmModel as immutable using the model.MarkAsImmutable() extension method. This enables caching of some expensive operations in the serialization hot path.
• Enable Utf8JsonWriter integration in the serializer (which performs less async and I/O operations internally) (see this blog post: https://devblogs.microsoft.com/odata/using-the-new-json-writer-in-odata/. In our internal benchmarks, this improves async serialization perf by ~47% and also reduce memory usage. Note that this writer currently does not support streaming of large binary payloads and buffers them in memory instead. This should not be a concern if you don’t use the streaming API and I don’t think it matters in AspNetCore.OData anyway (@Sam Xu correct me if I’m wrong). Do let us know if you use this option.
• Disable validations if you don’t need them. OData performs validation of the payload (e.g. ensuring the properties match the schema, no duplicate properties, etc.), but if you’re sure the payload is writing is valid you can disable all validations (or selectively enable them):

AddOData(options => {

    options.AddRouteComponents("/", edmModel, services =>

    {

        services.AddScoped<ODataMessageWriterSettings>(_ => new ODataMessageWriterSettings

        {

            Validations = ValidationKinds.None

        })

    });

})

You can selectively enable validations by combining different flags:

Validations = ValidationKinds.ThrowOnDuplicatePropertyNames & ValidationKinds.ThrowIfTypeConflictsWithMetadata

I’d suggest trying the different options independently and confirm the impact they have on your service and sharing the results with us.

[houbi56]

@habbes All these performance optimizations would be nice to have documented outside of an issue. The current state of documentation - reading blog posts and searching for issues isn't suitable for a lib this complex. Ogling the EF core docs pages..

[senioroman4uk]

@habbes here are our performance testing results with different options

case	50%	75%.	90%.	95%.	99%.	std dev	number of data points
BaseLine (our current odata setup)	366.8	392.49	426.84	449.5	509.04	0.21	300
utf8StreamSerializer	324.55	357.05	380.6	412.47	454.66	0.2	301
Immutable edm model	361.14	391.13	413.66	432.12	511.98	0.18	301
Disabled model validation	341.31	375.23	406.42	429.08	478.2	0.25	301

[senioroman4uk]

is there anything else we can do to lower the response time?