Allow sending message data as a string, not just as an array

[shantamg]

We want to be able to send data as a string

Since we are using laravel's event facade, we are limited to the payload being an array. Looking at the getPayloadFromEvent function, it's clear that broadcastWith() must return an array, because it's calling array_merge on the payload.

Our use case

The reason we would like to send data as a string, which the ably-php library supports, is because we want to send a json-encoded string without it being bloated by double encoding.

This all started because we had some messages that were over the 64k limit. When I reached out to Ably support, it was suggested that we chunk the data, send it as multiple messages, and then put it together on the client side. That worked great until I realized that each chunk of serialized data was around 10k bigger after being serialized again as a part of the payload (which included a UUID to group the chunks, the total number of chunks, and the current chunk index).

So the idea I had was to reserve a fixed number of characters at the beginning for the metadata and then just send the whole thing as a concatenated string. Then I realized the limitation of laravel events and that became a dead end.

Our current solution

I have a solution that involves taking each chunk, calling json_encode on it to determine how many bytes over my max chunk size it becomes, and then reducing the chunk by that amount. It works, and is fast, but it's a shame that we have to sacrifice that message size for the double encoding.

Any ideas? Could we extend laravel to support sending strings? Or just bypass the laravel event facade? That would also be a shame because our unit tests use Event::assertDispatched.

┆Issue is synchronized with this Jira Task by Unito

[shantamg]

Here is our current implementation of sending messages in chunks in case anyone is interested.

[sacOO7]

@shantamg ably allows you to send message size bigger than 64k. Currently, it depends on your account type. Seems, 64k is default for PAYG account. You might like to check with support if limit can be increased. WDYT?

[shantamg]

@shantamg ably allows you to send message size bigger than 64k. Currently, it depends on your account type. Seems, 64k is default for PAYG account. You might like to check with support if limit can be increased. WDYT?

Yes, the docs do say that it can be increased to 256k for "Committed Use". I suppose I could reach out to support and see what that entails and if there is much of a cost.

For us, we are nowhere near exceeding our message count limit. I believe there are others who would rather send more messages than upgrade their plan for the sake of an increased message size limit.

[sacOO7]

Since ably-laravel-broadcaster is closely integrated with laravel broadcasting interface, it will be tricky to modify public interface that sends string as a payload. You might like to check https://github.com/ably/ably-php-laravel. You can install it as a service-provider and can access all ably-php APIs.

[sacOO7]

While it's true $payload is an array/map, it's basically just a key and value pair. I think only overhead would be of the provided key.

You can see resulting payload to be bloated because by default ably-sdks use msgpack as a default encoding. You can check ably-php batch publish. msgpack generally reduces the amount of data that needs to be transferred between clients and servers -> https://faqs.ably.com/do-you-binary-encode-your-messages-for-greater-efficiency. Ideally, you shouldn't be facing msg size limit issues with default msgpack encoding.

If you still want to send payload as json, you might like to enable following configuration in config/broadcasting.php

        'ably' => [
            'driver' => 'ably',
            'key' => env('ABLY_KEY'),
            'useBinaryProtocol' => false
        ],

So, while broadcasting, you should be able to send message data as string!

[shantamg]

Oh interesting!

But hang on, I don't understand how that helps.

laravel-broadcaster uses ably-php so it has to go through this:

        if ( is_array( $this->data ) || $this->data instanceof \stdClass ) {
            $encodings[] = 'json';
            $msg->data = json_encode($this->data);

I am sending an array of data regardless so it's going to be encoded as json regardless, right?

The only reason I wanted to send the data as a string, which you can see is possible in the next few lines, is because if I know my message is going to need to be split up, I need to encode it and chunk it, and that $msg->data = json_encode($this->data) above is going to encode it again (encoding the containing array and thus bloating my json string with more escape characters).

So to be clear, I usually send data as an array (and it's clear that it gets serialized into a json string by ably-php on the way out. But to avoid overages, I am chunking the data and sending the chunked data (along with metadata about the chunk) in an array, which unfortunately has to be serialized on the way out. Being able to send a string that doesn't need to be escaped would allow me to avoid shrinking the chunks, which is my current workaround.

Thanks again for your interest in this question. I do agree that it may not be worth trying to get laravel-broadcaster, which is so closely coupled to laravel's events, to be able to send a string the event data.