Sorting on attributes of type "dropdown" or "swatch" use option_id in frontend product collection not the sorting order used everywhere else

[iphigenie]

EDIT: changed the chosen products (not some next to each other) and chosen name so it is more different than default category order and there is no confusion. screenshots made on April 14 on an instance supplied here

If you have an order to options in a "single select" attribute that isn't the order they were created in, sorting by this attribute in listings will not produce the order you expect. The order will not be the order as set in attribute configuration.

Note: the attribute option values will be in the correct order in admin, including sorting products in catalog admin product grid. they also will be in the correct orders in the layered navigation selector. But on the store frontend, they will be in the wrong order, not using the sort order established for the attribute options.

I have been banging my head on my site on a "model year" sorting but now see it is a core bug, possibly linked to the change to elasticsearch as I think this used to work (surely someone would have noticed before if it didnt?)

Steps to reproduce (*)

done it on 2.4.2 with demo content

1. add an options to the color attribute, in my case I added "fluorescent" - it is the last by option_id

2. change order, in my case I moved fluorescent to the proper alphabetical spot before gray

3. add colors to products, in my case I chose the bag section and gave the green bag the color "fluorescent" and give a few other bags "black", "gray" and "white". I have chosen products not next to each other so it is obvious when sorting by color or position

4. make sure the category has the attribute as a "sort by" option (bags has "all" by default)

5. Flush cache (if command line access, can also reindex etc etc it makes no difference)

6. go to the front end to that category and sort by color

In short, in the "default" category order the grey back is in front of the fluorescent one and the white after. If sorting by color works they will both be after. If sorting by color sticks to the wrong (option_id) order, they will both be before.

for reference, in default category order it looks like this

Expected result (*)

1. results should be in the attribute order value as set in the position list of the attribute (i.e. in database the field "sort_order")

in my example you'd expect the product assigned as fluorescent should come before the products assigned as gray and white here we see it in admin things are sorted right:

Actual result (*)

1. results are in the the order options were created (i.e. the field option_id) not the order set in attribute options management

in my example the product assigned as fluorescent is after the product assigned as white and gray

note: all products without color come after the products with a color no matter in which direction you sort.

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• [ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.

• [x

  ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.

• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant[bot]]

Hi @iphigenie. Thank you for your report. To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

  :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[iphigenie]

This has been reproduced on 2.4.2 release vanilla instance

[m2-assistant[bot]]

Hi @engcom-Oscar. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Oscar]

Hi @iphigenie ! Did you flush the cache after making changes with product's attribute? Because for me product's sorting working well - I tried to reproduce this issue on 2.4-develop branch.

[iphigenie]

definitely cache flushed, reindexed - done a few times. I still get white before turquoise.

[iphigenie]

@magento give me 2.4 instance

[magento-deployment-service[bot]]

Hi @iphigenie. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @iphigenie, here is your Magento Instance: https://ad1bf489fc81b71d7ceb70130152bac4-2-4.instances.magento-community.engineering Admin access: https://ad1bf489fc81b71d7ceb70130152bac4-2-4.instances.magento-community.engineering/admin_3f2a Login: a562d948 Password: 1b5480cebe41

[iphigenie]

so yes, the issue happens on 2.4.2 release https://ad1bf489fc81b71d7ceb70130152bac4-2-4.instances.magento-community.engineering/gear/bags.html?product_list_order=color I did clear the caches but no command line access reindex.

[iphigenie]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @iphigenie. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @iphigenie, here is your Magento Instance: https://ad1bf489fc81b71d7ceb70130152bac4-2-4-develop.instances.magento-community.engineering Admin access: https://ad1bf489fc81b71d7ceb70130152bac4-2-4-develop.instances.magento-community.engineering/admin_0659 Login: 71f37a00 Password: edd7250ff6e4

[iphigenie]

Hi @engcom-Oscar are you sure it is sorting by color in your example? Because there's another bug if it is, since they are not sorted by color properly either. the with-a-color products should not be in the middle (I should have picked 2 products further apart so it's more obvious when sorted, sorry, have updated description with a clearer choice of color name and products)

Magento 2.4.2 release instance requested here today

cache flushed via admin button

https://ad1bf489fc81b71d7ceb70130152bac4-2-4.instances.magento-community.engineering/gear/bags.html?product_list_order=color

Magento 2.4.2-dev instance requested here today

cache flushed via admin button

https://ad1bf489fc81b71d7ceb70130152bac4-2-4-develop.instances.magento-community.engineering/gear/bags.html?product_list_order=color

Maybe I'm having a brain blip and missing some obvious step, I'd love that!

[iphigenie]

Hi @iphigenie ! Did you flush the cache after making changes with product's attribute? Because for me product's sorting working well - I tried to reproduce this issue on 2.4-develop branch.

Hi again @engcom-Oscar I think your screenshot is just the default category order, not sorted by color? If sorted by colour the colored items move before the ones without color

I have updated my steps with a slightly different choice of color and products so it is easier to spot the difference between "sort by position" and the correct or incorrect "sort by color" it was done with the instance requested below .

[iphigenie]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @iphigenie. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @iphigenie, here is your Magento Instance: https://ad1bf489fc81b71d7ceb70130152bac4-2-4-develop.instances.magento-community.engineering Admin access: https://ad1bf489fc81b71d7ceb70130152bac4-2-4-develop.instances.magento-community.engineering/admin_b928 Login: 93adc41e Password: 92482baf30ec

[iphigenie]

I have now reproduced this 3 times with deployments supplied here can someone please tag it?

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Oscar Thank you for verifying the issue. Based on the provided information internal tickets MC-41919 were created

Issue Available: @engcom-Oscar, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[gabrieldagama]

Hi @iphigenie, after speaking with the Product Owner, we don't believe this is an issue. The sort order of the attributes on the admin area has no relationship with the sort order that products will show on the listing page.

That is only relevant to how the options are displayed on the product page.

Hope that makes sense. A feature request may be created for such a case, but it would have to be discussed with the Product organization to decide how it should work.

Thanks!

[magento-engcom-team]

Unfortunately, we are archiving this ticket now as it did not get much attention from both Magento Community and Core developers for an extended period. This is done in an effort to create a quality, community-driven backlog which will allow us to allocate the required attention more easily.

Please feel free to comment or reopen according to the Issue reporting guidelines the ticket if you are still facing this issue on the latest 2.x-develop branch. Thank you for collaboration.

[iphigenie]

Hi @iphigenie, after speaking with the Product Owner, we don't believe this is an issue. The sort order of the attributes on the admin area has no relationship with the sort order that products will show on the listing page.

Hi @gabrieldagama

I am going to resubmit and re-describe because it does not make sense that you think this is not a bug. I'll use a different attribute so I can illustrate the issues in something that seems less mundane. Maybe use "brand" or something where the issue with lack of controllable sorting is obvious, like "season" or "model year"

I suspect most people building sites on magento is going to get this reported by clients once they add new options over time as a bug.

But first, it is not true that the order is only relevant on the product page, IT IS USED IN THE LAYERED NAVIGATION - as it should.

Here's the layered navigation showing a particular order for colors, and same for brand. It uses, as everyone would expect, the order as set by the merchant in admin. In my example, Turquoise before White

Now I would expect that if I now sort the product listing by "color" it is going to use the same order.

I think that's what everyone would expect.

But it doesn't. I think it was expected to, it was just missed

This is extremely problematic as a site ages, as it is extremely likely that new manufacturers, or colors get added, and a merchant would absolutely expect to be able to move them in the correct place and the sort order to be what makes sense for the merchant - important brands first, or alphabetical, so that a customer can sort and it is intuitive and useful to the shopping logic.

Because otherwise sorting by these attributes is confusing - and confusion hurts trust, and trust is everything in online shopping.

Apple ending up between Motorola and Samsung, because that's when it was added, is never going to make sense

[gabrieldagama]

Hi @iphigenie, thanks for sharing your thoughts.

What I meant is that this is not a behavior that the product organization expects and the code was never built to produce such behavior, so from our perspective that is not a bug.

Now, I've spoken with @engcom-Oscar and he has brought up a valid point, it seems that the attribute sorting (color) is actually being done by option id (as you described). In this case, I believe it can be considered a bug, I believe that should be alphabetical order as an example.

If you believe the behavior should be based on attribute order on the backend, that will have to be a feature request instead.

Hope that makes sense, Thanks.

[iphigenie]

Hi @gabrieldagama

Thanks for the update - this is exactly what my ticket said, that they are being sorted by "option id" and that made no sense. It's obviously something you'd only start to notice after you've run a site a while and added new options.

Alphabetical I would have understood, or the order as set in admin (which is the order used for the swatches in layered navigation). Both of these a customer would understand. I'd be OK with either

I suspect it is going to be no more efficient to sort by value name (different per store view) than to sort in admin-set-order (same across store views) and it might be worth finding out what the community expects of such a sorting.

There are a few oddities in the admin vs frontend handling of options, I guess most people don't use 100 options in some like we ended up having

[iphigenie]

I shall just make my point one more time: the "attribute option admin order" is used by the system everywhere else @gabrieldagama (also: since the order-by-id is a bug, is there a ticket number for it or should we create one?)

• (admin) on the product admin page when setting the attribute
• (front end) in the layered navigation
• (front end) on the advanced search screen if the attribute is made a search option
• (admin) in the attribute values displayed in admin product grids if the attribute is shown
• (admin) in the attribute filter drop down for admin product grids if the attribute is made a filter

All of the above use the option order as set in the attribute admin screen, not alphabetical order. In Magento1 too it was everywhere.

So it is strange that product owner said this was not intended. They misunderstood the context maybe?

It still seems to me the logical thing to expect is that the sort order for that attribute should use the same as everywhere else, rather than being alphabetical (alphabetical would change the order per store view/language too!). But if it is decided this should be alphabetical on the front end, then please may I suggest the "advanced search" display and the layered navigation display should use that order too, so things are consistent.

[kestraly]

Argh, I was hoping for some resolve on this with ElasticSearch.

In the past I was able to sort by the sort_order of the attribute by using the following code

## \Magento\Eav\Model\ResourceModel\Entity\Attribute $eavAttribute,
## $this->_eavAttribute = $eavAttribute;

$attributeId = $this->_eavAttribute->getIdByCode('catalog_product', $subject->getCurrentOrder());

$collection->clear()->getSelect()
                            ->reset(\Zend_Db_Select::ORDER)
                            ->joinLeft(array('cpei' => 'catalog_product_entity_int'), 'e.entity_id = cpei.entity_id AND cpei.attribute_id = ' . $attributeId . ' AND cpei.store_id = 0', 'cpei.value as product_on_top')
                            ->joinLeft(array('eao' => 'eav_attribute_option'), 'eao.option_id = cpei.value', 'eao.sort_order as sort_order')
                            ->order('sort_order' . ' ' . 'asc');

                        if (count($products) > 0) {
                            foreach ($products as $position) {
                                $productarray[] = $position->getId();
                            }

                    if (!empty($products)) {
                        $result->getCollection()->getSelect()->order(
                            new \Zend_Db_Expr('FIELD(`e`.`entity_id`,' . implode(
                                    ',', array_unique($productarray)
                                ) . ') '
                            ));
                    }

[kestraly]

Can we reopen this?

It would be good if the sortby toolbar with ElasticSearch would sort by either the sort_order / position you've dragged the attribute to, or at the very least the attribute value name.

At the moment it sorts by attribute value id - this relies entirely on you adding the attribute to the catalog in one go and not adding additional attributes later on, which will go to the end of the sort, making the sort absolutely useless.

[nkarthickannan]

@gabrieldagama Hope you are doing well !! As far as I can see, you have mentioned that one issue and one feature request is found in this discussion. Let's ignore the feature request. We should atleast find a solution to the agreed issue before closing it.

Agreeing that an issue is there and ignoring does not seem to be logical. If you have any other ticket for this, please mention so that everyone can understand that this is being taken care in another ticket/issue.

BTW, I'm not sure if you will ever read the comments in a closed issue or not.