New category for MVT solutions

[rockeynebhwani]

I would like to create a new category for MVT solutions like Optimizely, ABTasty, OmniConvert. Currently all MVT solutions are classified under "Analytics". There are lot of 3rd parties in this space with poor reputation. Even the likes of Optimizely adds significant overhead due to client side JS implementation (even after all the optimizations they have been making in recent years).

It will be good to add a different category to create awareness added due to MVT solutions.

[patrickhulce]

Thanks for the suggestion @rockeynebhwani do you have a sample proposal for all the entities that would be moved? I've been hesitant to create new categories for fear of dilution. If a new potential category has only a very small overall percentage, I'd prefer to leave them as they are. If it'd be significant though then I'm on board!

[rockeynebhwani]

@patrickhulce - I will have to look at the full list but quick look suggests that we can use this as starting point

Entities to be moved

Optimizely VWO abtasty Monetate Qubit Bunchbox / Peaks & Pies Certona Maxymiser (Now called Oracle Maxymiser) Optimost inspectlet

Entities to be added convertize Google Optimize Adobe Test and Target Sitespect convert (www.convert.com) Evolv (www.evolv.ai)

[jitendravyas]

"Entities to be moved." but moved where?

[rockeynebhwani]

@jitendravyas - Proposal is to move to a new category (we can call it "MVT Providers")

[patrickhulce]

Thanks for the proposal @rockeynebhwani but those entities represent a very small overall proportion that I don't believe they deserve their own toplevel category. Perhaps we need a subcategory or "use case" tag to represent this information within analytics?

[rockeynebhwani]

@patrickhulce - These are the number of entities I could find with quick glance (total 16 in number). If you look at other categories like "tag manager", it has only 15 entities. "Video" category has just 5 entities... So, I hope you didn't mean to say that number of entities should determine if we need a category or not...

From my experience, performance of client side MVT solutions is a real problem. I have been in situations where we had to switch MVT solutions because some third parties will just not put effort in improving performance of their tags. At the same time, I have seen recently that companies like Optimizely are going extra mile and doing everything possible to optimize their tags (e.g. using self-hosting for their snippet and syncing any changes using Cloudflare edge workers to keep it up to date).

There are lot of businesses running on SAAS solutions where they can't move to server side MVT and only option is to use client side JS based solution and lot of time, these solutions do more harm instead of improving customer experience (due to poor performance of tags or not following best practices) so I still think that this deserves it's own category and will bring focus on third parties which are lagging behind otherwise it's get lost in huge list of analytics vendors..

I can't articulate this better than this tweet - https://twitter.com/yoavweiss/status/1123587262942523392

It's a necessary evil on some platform due to limitations imposed by SAAS platform but at least with this data and focus on this category, it will provide people a way to make more informed decision.

Let me know what you think.

[patrickhulce]

I hope you didn't mean to say that number of entities should determine if we need a category or not

It's not the number of entities, it's the volume of their total occurrences. From my quick count they added up to only a couple dozen thousand. Video is a prime example of a type of category I don't think should exist but for historical reasons I'm now somewhat bound to continue tracking it. Because of that regret I'm hesitant to repeat this mistake with yet another toplevel category.

Regarding Yoav's tweet and surrounding text, I'm failing to see why you're so opposed to a subcategory. As you point out analytics is large and comparison is difficult so subcategorizing would be a nice way around this while preserving the toplevel analysis of "analytics" third-parties.

but at least with this data and focus on this category, it will provide people a way to make more informed decision.

I obviously care about the performance impact of third-party providers, that's why I maintain this repo. But the primary performance impact of MVT is not in the time spent in their JS execution but the fact that they the typically delay and mask the true dependencies of the page which radically impacts early paint metrics like FCP and speed index. This isn't really captured in the metrics measured by this repo and, if anything, I think this makes a strong case for not splitting them out for fear of understating their performance impact.

[rockeynebhwani]

@patrickhulce - @andydavies tagged me today on an issue and that reminded me about htis...

• I think concept of sub-category is fine under wider 'analytics' category.

Also, when it comes to JS execution of MVT solutions, I didn't have any data last year but see this graph which I have been tracking and working with Optimizely team to improve these timings so as per this, it feels that JS long tasks and total CPU time is a problem. (I have another version of this screenshot with annotation of various optimizations made to explain drops)....

[patrickhulce]

Thanks for sharing that data @rockeynebhwani ! CPU time is definitely a consideration here, I agree I didn't mean to trivialize it that it's not a problem just that it's a part of the problem MVT solutions add to webperf and my concern was highlighting them separately at the toplevel would make too strong a statement about the total impact they have. It's mostly orthogonal to the discussion at hand here though. Either way this and #108 make clear the need for finer grained categories and a new taxonomy should be made. It will take time though.