Make network API performance metrics queryable

[pauldambra]

The replay team is currently considering improving PostHog by adding the ability to query network requests response times

We already capture network calls that are made and their performance data using the browsers' built-in PerformanceObserver. But we store these in blob storage as part of the source recording so we can't query them. If these were stored in ClickHouse we could allow you to query them using HogQL.

🗂 Considered Goals

• [ ] split ingestion so we also ingest performance data into ClickHouse
• [ ] add the PerformanceEntry table into HogQL
  • [ ] allow search for sessions
    • [ ] by network status code
    • [ ] network response time
  • [ ] allow finding network requests
    • [ ] by their parent page
    • [ ] by their target url
    • [ ] by their resource type
    • [ ] by their network status
• [ ] update insights (once they are converted to use HogQL) so that we can query this data in insights
  • [ ] allow graphing
    • [ ] request/response size
    • [ ] speed
    • [ ] status
• [ ] build a performance waterfall view on top of this data per session
• [ ] build a "lighthouse" performance view on top of this data per page
• [ ] show "page score" in the toolbar as you navigate around your site

These goals are being considered by the Replay Team and we'd love to hear user feedback, as well as ideas for stretch goals or implementation. Please add to this issue!

If you like this idea, please leave a 👍 or ❤️ reaction on this post to vote for it -- your votes and feedback help us prioritize what to work on next!

[peauc]

This performance feature would really be the cherry on the cake to an already great Posthog :)

[pauldambra]

As a step towards this... so that we can understand what data would be useful to store we've added a feature preview for a waterfall view on top of session replay data.

It doesn't make anything queryable yet 😅 but when you've an interesting session it lets you dig in to the network data gathered with it. It has a few sharp edges but we'd love feedback on what would improve it

• turn on network performance capture
• visit feature previews and enable "replay network waterfall"

[privateJiangyaokai]

Could you also consider graphql support? Based on OperationName for example.

[pauldambra]

hey @privateJiangyaokai,

What would that unlock for you? What does OperationName map to for you, it's not a name we have in PostHog right now.

thanks!

[privateJiangyaokai]

Typical graphql APIs have the same path. e.g. /api/graphql. curls typically look like this: curl 'https://zionbackend.functorz.com/api/graphql' \ -H 'sec-ch-ua: "Google Chrome";v="125", "Chromium";v="125", "Not.A/Brand";v="24"' \ -H 'user-locale: EN' \ -H 'x-zed-version: 1.9.1' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'authorization: Bearer xxx' \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36' \ -H 'content-type: application/json' \ -H 'accept: */*' \ -H 'Referer: https://zion.functorz.com/' \ -H 'x-session-id: f43e2335-d357-41df-aacf-98dc804551fe' \ -H 'sec-ch-ua-platform: "macOS"' \ --data-raw '{"operationName":"UserQuestionsV3","variables":{},"query":"query UserQuestionsV3 {\n userQuestionsV3 {\n ...UserQuestion\n }\n}\n\nfragment UserQuestion on UserQuestion {\n id\n chineseText\n englishText\n tipContent {\n ...TipContent\n }\n type\n placeholderI18nKey\n choices {\n id\n chineseText\n englishText\n nextQuestionId\n questionId\n tipContent {\n ...TipContent\n }\n }\n}\n\nfragment TipContent on TipContent {\n chineseContent\n englishContent\n hyperLink\n}\n"}'

curl 'https://zionbackend.functorz.com/api/graphql' \ -H 'sec-ch-ua: "Google Chrome";v="125", "Chromium";v="125", "Not.A/Brand";v="24"' \ -H 'user-locale: EN' \ -H 'x-zed-version: 1.9.1' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'authorization: Bearer xxx' \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36' \ -H 'content-type: application/json' \ -H 'accept: */*' \ -H 'Referer: https://zion.functorz.com/' \ -H 'x-session-id: f43e2335-d357-41df-aacf-98dc804551fe' \ -H 'sec-ch-ua-platform: "macOS"' \ --data-raw '{"operationName":"UnreadMessageCount","variables":{"popupArea":"USER_CENTER"},"query":"query UnreadMessageCount($popupArea: SiteArea) {\n unreadMessageCount(popupArea: $popupArea) {\n category\n count\n }\n}\n"}'

[privateJiangyaokai]

"operationName":"UserQuestionsV3" is almost equivalent to RESTful API's path.

[pauldambra]

sure, i've used graphql 😊 why would graphql help you here? what would you expect the operation name to be for a network request? why use that as an example?

full disclosure, this would be exposed via HogQL and our existing restful APIs, we don't have graphql in the stack right now, so just trying to understand why it helps you :)

[privateJiangyaokai]

Ah, I meant for teams who use graphql as part of their tech stack, such as us, it would greatly benefit us if we could gather network statistics grouped by graphql's operationName. A resultant table should look like so

[pauldambra]

Ah ha! I see 🙈 Cool, thanks for sticking with me - that's why we ask until we understand 💖

It's not something we'd considered since we'd like to avoid parsing these payloads as much as we can. I guess it'd be a followup item for us but I'll definitely bear it in mind

[privateJiangyaokai]

how much would that parsing cost? I know the payload is going to be unpredictable. But in terms of cycles/ns per kbyte? an alternative would be capturing and storing the payload, which again is going to be unpredictable which I understand. How much is the cost for properly billing your customers (like us)?

lastly, I'd like to understand (but not necessarily challenge) your rationale behind trying to void parsing the payloads. Thanks!

[daibhin]

We have a whole host of Replay data that we should look to expose beyond just API performance metrics. Questioned in https://posthoghelp.zendesk.com/agent/tickets/16210 I wonder if we should expand the scope of this issue to include events like window visible and window hidden

[abigailbramble]

+1 https://posthoghelp.zendesk.com/agent/tickets/16335

[lukyncze]

Are we gonna be able to "monitor" network requests made? Specifically it would really help me, if I could somehow know, how many requests X were fired in specific period. Additionally it would help me to have a monitoring, that could send me a slack message, when there are more same URL requests made in the UI. Thank you in advance!

[daibhin]

@lukyncze when we make this data "queryable" it should be accessible within Product Analytics like any other property.

We do have a tracking issue for a separate Threshold Based Alerting feature: https://github.com/PostHog/posthog/issues/14331. I think that is what you will need to "monitor" any insights you create once this data is available.

One question, could you clarify what you meant by "when there are more same URL requests made in the UI". It would be great to understand that use case a little bit better

[lukyncze]

@daibhin thank you for your answer.

Yep - I would like to monitor the requests with same request URL. For example when user requests https://lukyn.cz/books/:XXX/edit x times in y seconds, I would like to know it. Is it more clear now?

[daibhin]

@lukyncze totally, makes sense. Thanks :)

[CodyMorris]

+1 , this would be amazing