Replay "message size too large" error

[pauldambra]

Bug Description

We see these for multiple reasons We can't ingest them When we can't ingest a recording snapshot the chance of an unplayable recording is high We need to minimise these

Kafka checks the size of a message before it compresses it so we need to be under 10MB un compressed to be ingestible

We're gathering samples of these messages so that we can identify improvements

file size	lib version	reason	things to try	tried in
67MB	1.146.1	426k attribute adds	* split incrementals in the SDK	v1.148.0
36MB	1.146.0	massive data image urls	* redact large image urls
34MB	1.146.1	250k attribute adds	* split incrementals in the SDK	v1.148.0
15MB	1.145.0	lots of inlined css	compress the text send inlined css a different way
11MB	1.86.1	many small incremental snapshots	split the snapshots in capture get customer to upgrade

problems seen	count
super large rrweb data structure	5
large data urls	8
lots of inlined CSS	3
lots of small snapshots	1

TODO

• [ ] redact large data urls in the SDK
• [x] split large arrays of rrweb data in the SDK
• [ ] split large incremental snapshots in the SDK

[daibhin]

The new asset stuff from rrweb might help a lot with sending the inlined CSS in a different way https://github.com/rrweb-io/rrweb/pull/1475

[sudo-eugene]

Does this issue have anything to do with maskAllInputs: true or maskTextSelector: "*"? I've tried masking everything I possibly can but I'm still getting the error with unplayable recording

[pauldambra]

Hey @sudo-eugene,

Normally that's not the issue here. The best bet is to report the problem using the in-app support flow since that lets us look into you recordings specifically to see what's best. Ideally if you can include example recordings that'd be awesome

[sudo-eugene]

@pauldambra this doesn't seem to be an issue in my PH cloud instance. It's only on my Self-Hosted instance that this happens. The issue report is, for reasons I understand, not available in the self-hosted version, hence my post here.

[pauldambra]

Ah, it's really hard to debug self-hosted since everyone's setup can vary so much. You can see in this issue the reasons we see this trigger. With large data urls and large amounts of CSS being the next largest un-addressed items.

[sudo-eugene]

Yes I suspect the CSS could be an issue our side. Could this potentially be due to inline CSS, or css files, or both?

Is there a way to disable/exclude CSS that I know won't be of value in our recordings?

[daibhin]

@sudo-eugene we mostly recommend to people with very large CSS bundles that they skip inlining it. This can be done when initializing the SDK:

posthog.init("API_KEY", {
    session_recording: {
        inlineStylesheet: false,
    }
})

Just so you know, this will mean that the files will be fetched during playback. If there are no longer available or have changed since capture your recordings might be unstyled

[lessless]

Just FYI: Sentry doesn't have any issues capturing same sessions