Union function does not de-duplicate list-valued group property values in persisted storage

[marcopesani]

When using the Mixpanel SDK's union function to merge a given list with a list-valued group property, for example, when tracking the products seen by the customer, the SDK adds the values to a cookie or local storage without de-duplicating them. This can cause the cookie to become very large, surpassing the 4096 bytes limit.

Can be observed here: Bitrefill website I enabled localStorage as persistence storage to not crash NGINX

Steps to Reproduce:

1. Initialize the Mixpanel SDK using cookies as a persistence storage.
2. Use the union function to merge a list of values with a list-valued group property, e.g. mixpanel.people.union('pages_visited', 'homepage');
3. Observe that the cookie or local storage containing the group property's values becomes very large, surpassing the 4096 bytes limit.

Expected Behavior:

• The union function should de-duplicate the values before adding them to the cookie or local storage, or
• The union function should not store people's values in the browser at all or flush them after writing

Actual Behavior:

• The union function does not de-duplicate the values before adding them to the cookie or local storage.

Notes:

• This issue has been reproduced on different browsers and devices.
• This could be a big problem for sites with high traffic or big lists.
• Solution could be de-duplicating before sending to mixpanel or de-duplicating in the SDK.

Environment:

Mixpanel SDK version: 2.45.0 Browser and version: Google Chrome 108.0.5359.124 (arm64) Operating system: Mac OS Ventura 13.0.1 (22A400)

[chrisdeely]

This issue poses a significant problem when leveraging the cookie persistence as the growing size of the cookie eventually causes failed requests to application servers with 431 Request Header Fields Too Large errors.

Once a user's cookie gets to this size it can be exceedingly difficult to remedy the situation and restore their access to the application.