PubSub with large message does not trigger subscriber

brunoshine commented 7 months ago

Expected Behavior

Publishing a large message that, for instance, has a base64 image string should trigger the subscriber.

Publisher

import { DaprClient, CommunicationProtocolEnum } from '@dapr/dapr';
import * as image from './image.js'

const client = new DaprClient({
    communicationProtocol: CommunicationProtocolEnum.GRPC,
    maxBodySizeMb: 10,
});

console.log("Publishing...")
setInterval(async () => {
    await client.pubsub.publish("pubsub", "test", {
        payload: image.data.image, //base64 image string

    }, {
        metadata:{
            ttlInSeconds:"1"
        }
    });
    console.log("published")
}, 3000);

Subscriber:

import { DaprServer, CommunicationProtocolEnum, DaprPubSubStatusEnum } from '@dapr/dapr';
const server = new DaprServer({
    communicationProtocol: CommunicationProtocolEnum.GRPC,
    clientOptions: {
        communicationProtocol: CommunicationProtocolEnum.GRPC
    }
});

const configs = await server.client.configuration.get("configuration", ["key"])

server.pubsub.subscribe("pubsub", "test", async (data, headers) => {
    console.log("Subscriber received: " + JSON.stringify(data));
    return DaprPubSubStatusEnum.SUCCESS;
}
);

await server.start();

command:

"scripts": {
    "client": "dapr run --app-id checkout --dapr-http-port 3502   --app-protocol grpc --resources-path .dapr/components/ -- node client.js",
    "server": "dapr run --app-id order-processing --app-port 5002 --dapr-http-port 3501   --app-protocol grpc --resources-path .dapr/components/ -- node server.js"
  },

Redis PubSub Component Definition

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: localhost:6379
  - name: redisPassword
    value: ""

Actual Behavior

Although the message get published to Redis stream the subscriber does not get triggered. In this test we are using a base64 image that is around 150kb.

If I just set the payload to payload: "" the subscriber gets triggered.

Steps to Reproduce the Problem

Convert an image to a base64 string and pass the string into the payload on the publisher.
Execute both the client and the server npm scripts.
Published messages should be on the redis stream named test.
Subscriber is not triggered.
Stop both client and server.
Update the payload to be empty payload:""
Execute both the client and the server npm scripts.
Published messages should be on the redis stream named test.
Subscriber is triggered.

brunoshine commented 7 months ago

It seems that this also happens with bindings. I have a MQTT Input binding where I have placed a 132kb image base64 string on the body of message topic, and with this the binding does not get triggered. If I remove the base64 string then the binding gets called.

Any thoughts?

thanks

shubham1172 commented 5 months ago

I am able to reproduce this locally. Tried with both gRPC and HTTP. The message does gets published, and even comes to Dapr (as part of debug logs):

DEBU[0009] Processing Redis message 1709193127018-0      app_id=subscriber component="pubsub (pubsub.redis/v1)" instance=Shubhams-MacBook-Pro-3.local scope=dapr.contrib type=log ver=1.12.5

However, it never reaches the application. I am using a 1MB payload.

Increasing the maxBodySizeMb does not work too.
I also tried using a Go app like this, it can't receive the large payload either.
Then I set to try this with vanilla JavaScript, and it works with the right bodyParser settings

import express from 'express';
import bodyParser from 'body-parser';

const APP_PORT = process.env.APP_PORT ?? '8080';

const app = express();
app.use(bodyParser.json({ type: 'application/*+json', limit: "50mb" }));
// app.use(bodyParser.urlencoded({ limit: "50mb" }));

app.get('/dapr/subscribe', (_req, res) => {
    res.json([
        {
            pubsubname: "pubsub",
            topic: "test",
            route: "/events"
        }
    ]);
});

// Dapr subscription routes orders topic to this route
app.post('/events', (req, res) => {
    console.log("Subscriber received:", req.body.data);
    res.sendStatus(200);
});

app.listen(APP_PORT);

We also have the same configuration in JS-SDK for bodyParser, so this needs further investigation why HTTP (and then also gRPC for some reason) is not working.

brunoshine commented 5 months ago

Hi @shubham1172 is there any temporary workaround? Thanks

shubham1172 commented 5 months ago

@brunoshine I don't have a workaround for now, but I will further investigate this and update this issue. Thanks for your patience.

dapr-bot commented 3 months ago

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

dapr-bot commented 1 month ago

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

dapr-bot commented 3 weeks ago

This issue has been automatically closed because it has not had activity in the last 67 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

dapr / js-sdk