App bridge is not measuring First Input Delay

[daviareias]

Describe the bug

AppBridge is not correctly measuring the 'First Input Delay' (FID) metric for web vital sessions. This discrepancy in the web vitals measurement is affecting our app's 'Built for Shopify' status, as the platform requires at least 100 web vital sessions to be recorded.

It's been observed that Shopify is capturing up to 7 times more 'Largest Contentful Paint' (LCP) sessions compared to FID, which suggests a potential bug in the FID tracking or a misconfiguration.

This is inconsistent with the 'Built for Shopify' status requirement, which stipulates achieving 50 active installations at any given time. The current discrepancy suggests that to meet the erroneous 100 session requirement, an app would need to maintain a much higher installation rate, potentially around 100 installations per month. This rate is disproportionately high and approximately 20 times greater than the actual 'Built for Shopify' requirement.

To Reproduce

Steps to reproduce the behaviour are not applicable as the issue lies in AppBridge and Shopify Partner dashboard's internal metrics collection mechanism.

Attached is a screenshot showing the disparity in the 'Largest Contentful Paint' and 'First Input Delay' measurements.

Here's the warning in the Shopify partner's dashboard saying we don't meet requirements because there are not enough sessions, even though our FID was never below the required measure (if we were we would never get the badge in the first place):

Expected behaviour

We expect AppBridge to accurately measure all web vital metrics, including 'First Input Delay', to ensure that our app meets the necessary performance standards for maintaining our 'Built for Shopify' status.

Contextual information

Packages and versions

• @shopify/app-bridge cdn version (https://cdn.shopify.com/shopifycloud/app-bridge.js)
• @shopify/polaris @ ^12.0.0

Platform

• OS: MacOS, Windows, iOS, Android
• OS Version: N/A
• App: Shopify Mobile, Shopify POS, Chrome, Safari, Firefox

Additional context

The app’s configuration is standard and follows Shopify's recommended setup with Node and Next.js for the backend. This measurement inconsistency has first been noticed 6 months ago and it already affected one of our apps, it will soon affect another app within around 10 days, unless we pump up those numbers.

### Tasks

[Stef733]

I second that, are we expected to have 100 loads daily to get a measurement?

[daviareias]

Our app is still way within the required FID measures but because of this bug, if app bridge can't measure 100 FID sessions in a 28 day period, we'll keep getting warnings that we'll lose BFS status:

The messages above refer to the same app and all it takes to make the warning go away is having 100 sessions measure by App Bridge, which on average takes at least 10x that amount.

[daviareias]

Third time this happened this year with just one app that had 194 INSTALLATIONS in one month, but App bridge won't measure the FID:

[henrytao-me]

@Stef733 @daviareias can you let me know your appId? I can look into it.

[henrytao-me]

FID relies on user input. I confirmed that it is reported properly. So, I close this ticket for now. Feel free to reopen or comment your appId so I can take a look.

Note that: there is an ongoing issue about missing all metrics in partner dashboard due to data pipeline issue which is being investigated here https://github.com/Shopify/shopify-app-bridge/issues/253

[daviareias]

It seems that there's no fix for this. We're just going to tell the team to keep hitting the refresh button, until it gets to 100 loads.

[daviareias]

Not sure how to solve this. Should we keep emailing the users to keep loading the app for no reason?

Should we abandon "app extensions" to force users to load the app instead to hit this metric?

[dev-coderise]

Note that: there is an ongoing issue about missing all metrics in partner dashboard due to data pipeline issue which is being investigated here #253

We noticed that our FID is missing some days as well. For example, you can see it skipped May 24 in screenshot below:

but CLS included May 24 data and also reports almost 2x more loads i.e.

Due to the 100 FID load requirement, we are not able to submit the app for BFS badge even though we fulfill other requirements i.e.

Can someone explain why CLS loads are different than FID loads?

If Shopify is missing some metrics, shouldn't this 100 FID load requirement be waived?

Also, this requirement seems like an anti-pattern. Ideally, in today's automation world, the best apps are those that you set once and they automatically run in the background forever without any further manual interactions. So, on one hand, the BLS badge is trying to follow best standards, but at the same time, penalizing app developers by imposing some load requirement which does not make too much sense to me as a developer.