error While subscribing to raw message from a web api microservice using Dapr

[mpprasanth]

Ask your question here

Hello, I am getting below error While subscribing to raw message from a web api microservice (Microservice1) hosted in docker container (Refer the attached diagram) The error message logged in Docker desktop (Microservice1 sidecar docker container log). “time="2022-12-02T06:13:56.234572354Z" level=warning msg="retriable error returned from app while processing pub/sub event 3cbeee4d-03dd-47f7-91fe-42fd9fb7df5d, topic: rawDataTestTopic, body: {\"type\":\"https://tools.ietf.org/html/rfc7231#section-6.5.13\",\"title\":\"Unsupported Media Type\",\"status\":415,\"traceId\":\"00-b3cff34b70d01da2c7c58f4da4f1d35b-77f9e3a244d74e32-00\"}. status code returned: 415" app_id=DBAgent instance=12a64ef3bd7d scope=dapr.runtime type=log ver=1.8.4

Publishing to Kafka pub/sub is done using a curl command: curl -X "POST" http://localhost:3500/v1.0/publish/pubsub/rawDataTestTopic?metadata.rawPayload=true -H "Content-Type: application/json" -d '{"ordernumber": "1234"}'

Program.cs code: app.MapSubscribeHandler(new Dapr.SubscribeOptions() { EnableRawPayload = true });

Microservice1 subscribe to Kafka topic using below code: [Topic(DAPR_PUBSUB_NAME, "rawDataTestTopic",true)] [HttpPost()] public async Task OnIFMoEvent([FromBody] Root data) { string s = data.ToString(); return Ok(); }

…..

public class Root { public string ordernumber { get; set; } }

Please Note:- Both Microservice1 & its side car are running without any error. If I remove EnableRawPayload configuration => app.MapSubscribeHandler(), [Topic(DAPR_PUBSUB_NAME, "rawDataTestTopic")] & Curl command .. metadata.rawPayload=false It is Woking fine

I could see an issue related to this at https://github.com/dapr/dotnet-sdk/issues/709 looks like raw data is supported now

Can someone help ? did I miss any configuration ?

[mpprasanth]

@rynowak @sapinderpalsingh, can you please help ?

[TimSalomons]

I encountered a similar problem when trying to deserialize back into a typed object again. After debugging i found that the body of a rawPayload is presented to the Controller method with the application/octet-stream content type. Per default there is no inputformatter for converting this back into a complex type such as Root.

I resolved this issue by adding a custom InputFormatter to the options MvcOptions Action inside AddControllers()/AddMvc(). Resulting for me in something like this: builder.Services.AddControllers(options => { options.InputFormatters.Add(new CustomFormatter()); }).AddDapr();

This formatter then uses the UTF8 encoding to parse the byte[] to a json string. As a final step it tries to Deserialize the json string into a typed object and return it.

[fgheysels]

I ran into the same issue when using Dapr in ASP.NET Core to listen to raw payloads (so no cloud-events). Just like @TimSalomons , I've solved this by writing a custom formatter.

The custom formatter looks like this:

public class DaprRawPayloadInputFormatter : InputFormatter
{
    public DaprRawPayloadInputFormatter()
    {
            SupportedMediaTypes.Add("application/octet-stream");
    }

public override async Task<InputFormatterResult> ReadRequestBodyAsync(InputFormatterContext context)
{
    using (MemoryStream str = new MemoryStream())
    {
        try
        {
            await context.HttpContext.Request.Body.CopyToAsync(str);

            var jsonString = System.Text.Encoding.UTF8.GetString(str.ToArray());

            var deserializedModel = System.Text.Json.JsonSerializer.Deserialize(jsonString, context.ModelType);

            return InputFormatterResult.Success(deserializedModel);
        }
        catch
        {
            return InputFormatterResult.Failure();
        }
    }
}

It is used like this:

builder.Services.AddControllers(options => options.InputFormatters.Add(new DaprRawPayloadInputFormatter()))
                .AddDapr();

[zabidin901]

This can also be fixed by removing app.UseCloudEvents();

[fvandillen]

This can also be fixed by removing app.UseCloudEvents();

This is our experience too. That begs the question: why? The middleware should do nothing when the content type is not application/cloudevents+json.

[zabidin901]

This can also be fixed by removing app.UseCloudEvents();

This is our experience too. That begs the question: why? The middleware should do nothing when the content type is not application/cloudevents+json.

Looking through the SDK it doesn't look like it ignores content types that are not application/cloudevents+json

https://github.com/dapr/dotnet-sdk/blob/master/src/Dapr.AspNetCore/DaprApplicationBuilderExtensions.cs

So if that middleware is active it will expect cloud events.