search

dunglas / mercure

🪽 An open, easy, fast, reliable and battery-efficient solution for real-time communications
https://mercure.rocks
GNU Affero General Public License v3.0
3.83k stars 278 forks source link

400 Bad Request on publishing more than 10 MB #878

Closed jcdirks closed 3 months ago

jcdirks commented 3 months ago

When publishing via the same methods, but different data sizes, we are successful with smaller data sizes like 1.1 MB, but fail on posting 10.1 MB.

We use Mercure in Caddy 2.7.6, built with https://github.com/dunglas/symfony-docker, not using FrankenPHP yet. Our code is based on Symfony 6.4.4, symfony/mercure-bundle 0.3.8. We push updates via the #[Broadcast]attribute.

Some data sets, i.e. Turbo Broadcast templates, are smaller, some are larger. As only the size differs, I suspect the cause in Mercure Hub, not in the Symfony component.

We nailed it down to a null returning getContent() in vendor/symfony/mercure/src/Hub.php:99 which throws a "Failed to send an update.". exception. Looking into it with XDebug, a 400 Bad request is returned from the request.

Is there a size limit for the application/x-www-form-urlencoded-encoded POST payload?

dunglas commented 3 months ago

Not by default. Are you using some kind of proxy such as NGINX?

jcdirks commented 3 months ago

No, it's a direct communication between the local (developer notebook) PHP and Caddy docker containers through the URL http://projectname-caddy/.well-known/mercure, that should be the internal docker network. The URL is the same for successful and failed requests.

After tuning the test data, we have narrowed it down to a body size of 9.9 MB to succeed, but 10.1 MB to fail.

jcdirks commented 3 months ago

Getting closer.

The only 400 Response with no custom text is emitted at:https://github.com/dunglas/mercure/blob/0bddcaeb69351654d052c85e6696b50d65874d0b/publish.go#L26

http.Request.parsForm() is limited to 10 MB by default: https://pkg.go.dev/net/http#Request.ParseForm.

I'm now in a state of 'learning Go by guessing'. Is there a way to tune the MaxByteReader of http.Request?

dunglas commented 3 months ago

I wasn't aware of this check, sorry. Anyway, handling 10MB updates with Mercure will likely cause bottlenecks. Can't you reduce the size of the payload?

jcdirks commented 3 months ago

We‘d have to then. User interaction with one row of the dataset will most likely lead to updates, additions and deletions throughout other rows, so we had opted to just render the whole dataset.

But you are right, especially the twig rendering part is slow. If you have any idea how to tune the request function, any tip is greatly appreciated.

Aside of this, big kudos for thinking along and for your contributions to this amazing ecosystem!