Changing the way attributes are persisted / aggregated

[dkarlovi]

Current behavior

You need to define all your attributes in the root of your Elasticsearch document, like so

"name": "Something",
"attribute1": 123,
"attribute2": 938,

In the case you're using ES < 7 (one index for everything) or if you have a lot of attributes (in our case - 903, not all attributes for all products of course), you'll run into the issue of Elasticsearch disallowing more than 1000 fields per index by default, changing that limit can lead to performance and memory usage problems.

Expected behavior

Allow storing arbitrary number of attributes, for example using an array instead:

"attributes": [
    {"name": "attribute1", "value": 123},
    {"name": "attribute2", "value": 938}
],

Additional benefit of this approach is: it doesn't clutter the mapping, but also makes mapping attributes possible with libs like ONGR Elasticsearch (used in Coreshop bridge), which maps only explicitly annotated properties by default, you cannot add them on the fly.

Steps to reproduce the issue

Not sure how I'd do that, just create a 100 products, each with 10 different attributes attached,

Version of Vue Storefront

• [x] Vue Storefront
• [ ] Vue Storefront Next

Can you handle fixing this bug by yourself?

• [ ] YES
• [x] NO

Which Release Cycle state this refers to? Info for developer. (doesn't apply to Next)

Pick one option.

• [ ] This is a bug report for test version on https://test.storefrontcloud.io - In this case Developer should create branch from develop branch and create Pull Request 2. Feature / Improvement back to develop.
• [ ] This is a bug report for current Release Candidate version on https://next.storefrontcloud.io - In this case Developer should create branch from release branch and create Pull Request 3. Stabilisation fix back to release.
• [x] This is a bug report for current Stable version on https://demo.storefrontcloud.io and should be placed in next stable version hotfix - In this case Developer should create branch from hotfix or master branch and create Pull Request 4. Hotfix back to hotfix.

Additional information

Somewhat related to vuestorefront/vue-storefront-1#254, doing this would require changing the aggregations, so it's a good direction to not require naming the filters at all, since you can construct filters from what the search matched (and enrich it with attribute metadata, which is done already).

[dkarlovi]

Followup here: we need about 450 filters added, this is currently too much because the URL generated toward VSF API is too large (getting HTTP error 414).

We've worked around that by adding the filters in server-side, just before building the query. It means the filters don't always need to travel from the client to the server (being static, configured in the config file). The issue we're still getting all the aggregations back, even the empty ones, that one can be patched out.

What would really help is:

1. removing the attribute values from the root object, keep the attribute values in an array instead (as noted above)
2. aggregate on that field alone, meaning the buckets will be created naturally by Elasticsearch, without enumerating anything
3. postprocessing can remove attributes which are not to be exported, if required

This would fix all the issues:

1. Dynamic filters in vuestorefront/vue-storefront-1#254
2. too much fields issue reported here

[PawFV]

Any update on this? @gibkigonzo