Database Error Updating Magento to version 2.4.3 on setup:upgrade

[burgh8wp]

After updating Magento from the version. 2.4.2-p1 to 2.4.3 during setup:upgrade command we see below error:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '106852' for key 'PRIMARY', query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id)

The problem is this table holds around 20,000 records, and looks to be hundreds, potentially over 1000 instances of a duplicate ID for url_rewrite_id which stems from Magento 2.4.3 looking to force a primary key onto the database table catalogu_url_rewrite_product_category

Preconditions (*)

1. Update Magento to version 2.4.3
2. Error appears when running php bin/magento setup:upgrade command

Steps to reproduce (*)

1. Error appears when running php bin/magento setup:upgrade command after changing composer.json to Magento 2.4.3 version

Expected result (*)

1. Upgrade Magento to version 2.4.3 and complete setup:upgrade query

Actual result (*)

php bin/magento setup:upgrade
Cache types config flushed successfully
Cache cleared successfully
File system cleanup:
/home/storm/sites/sitename/public/generated/code/Laminas
/home/storm/sites/sitename/public/generated/code/Magento
/home/storm/sites/sitename/public/generated/code/Psr
/home/storm/sites/sitename/public/generated/code/Symfony
The directory '/home/storm/sites/sitename/public/generated/metadata/' doesn't exist - skipping cleanup
Updating modules:
Cache cleared successfully
Schema creation/updates:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '106852' for key 'PRIMARY', query was: ALTER TABLE `catalog_url_rewrite_product_category` ADD CONSTRAINT  PRIMARY KEY (`url_rewrite_id`)

[ ] Severity: S0 _- Affects critical data or functionality and leaves users without workaround.

** Changing back to 2.4.2-p1 does not give this error output and setup:upgrade completes fine.

[m2-assistant[bot]]

Hi @burgh8wp. Thank you for your report. To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

  :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[Thundar]

I had the same issue and tried to remove the duplicated entries (same versions for update). I have a similar issue again after saving a category with different url in store scope: eg: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '174976976' for key 'PRIMARY', query was: INSERT INTOcatalog_url_rewrite_product_category(url_rewrite_id,category_id,product_id) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?) I think this should be solved in an upgrade. Introduced by https://github.com/magento/magento2/commit/ece2528a6f09da1486d63ddd794f746501984413 , issue was https://github.com/magento/magento2/issues/9581

@ihor-sviziev predicted it https://github.com/magento/magento2/pull/25746#pullrequestreview-359746861 for backward compatibility, I'm adding the issue for different scoped attribute.

[ihor-sviziev]

Hi @burgh8wp @Thundar, You should remove duplications in the catalog_url_rewrite_product_category table before doing an upgrade to 2.4.3. Unfortunately, I don't have such an issue, so I can't provide you a good solution now.

[burgh8wp]

Hi @ihor-sviziev appreciate your reply. But in this instance, they have been there for years with no issue. What has changed in version 2.4.3 specifically where this is no longer acceptable? There are literally thousands of duplications in this table on this 1 site, where there are circa 20,000 individual records. So going through to manually delete is not an option. Also the data is there for a reason, so deleting will break the data on site somewhere. So a better solution is definitely needed. Even a script to run manually once to change/update the ID number? But is this ID number linked from other database tables?

[Thundar]

@ihor-sviziev done, but that's not enough (unfortunately). I could upgrade removing all the duplicates (manually, they were 70K, I don't think this can be a solution for everybody), but I couldn't update the category url in a store scope later, as from the INSERT query above.

[Thundar]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @Thundar. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @Thundar, here is your Magento Instance: https://85b60892c241a3d87d464e439dff008f-2-4-develop.instances.magento-community.engineering Admin access: https://85b60892c241a3d87d464e439dff008f-2-4-develop.instances.magento-community.engineering/admin_81f4 Login: 5f67abc3 Password: 3759b6fe684d

[Thundar]

I'm investigating the second problem (the insert) as I'm not being able to reproduce it in a fresh installation.

[burgh8wp]

@ihor-sviziev done, but that's not enough (unfortunately). I could upgrade removing all the duplicates (manually, they were 70K, I don't think this can be a solution for everybody), but I couldn't update the category url in a store scope later, as from the INSERT query above.

Did you create a script to remove the duplicates? If so, how did you handle the ID number change or did you just delete them?

[ihor-sviziev]

@burgh8wp, sorry, I don't have this issue, can't help you with fixing that :(

[Thundar]

@burgh8wp i strongly suggest NOT to do this in a production environment. START TRANSACTION; CREATE TABLE catalog_url_rewrite_product_category_temp SELECT DISTINCT * FROM catalog_url_rewrite_product_category; ALTER TABLE catalog_url_rewrite_product_category RENAME catalog_url_rewrite_product_category_backup; ALTER TABLE catalog_url_rewrite_product_category_temp RENAME catalog_url_rewrite_product_category; DROP TABLE catalog_url_rewrite_product_category_backup; COMMIT;

Edit: this query solved also the INSERT issue I quoted before. Edit2: backup table dropped as for https://github.com/magento/magento2/issues/33770#issuecomment-908245196

[MajorKuprich]

@Thundar fix unfortunately didn't resolve issue for me. But I've found solution which rollback changes from https://github.com/magento/magento2/commit/ece2528a6f09da1486d63ddd794f746501984413 by remove primary constraint.

etc/db_schema.xml

<?xml version="1.0"?>
<schema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Setup/Declaration/Schema/etc/schema.xsd">
    <table name="catalog_url_rewrite_product_category">
        <constraint xsi:type="primary" referenceId="PRIMARY" disabled="true">
            <column name="url_rewrite_id"/>
        </constraint>
    </table>
</schema>

Next generate whitelist: bin/magento setup:db-declaration:generate-whitelist Finally command bin/magento setup:db-data:upgrade should be executed without errors. :)

Reference: https://magento.stackexchange.com/questions/294038/magento-2-3-remove-constraint-from-db-schema-xml-file

[Thundar]

@MajorKuprich i suggest you not to remove the constraint. Instead, you should find out what is blocking your table, and the cause of the ids duplicated. My query removes duplicates only where a rewrite_id is associated to the same category_id and product_id in different rows. If it doesn't work for you, maybe you have one rewrite_id related to two different categories and/or products: that could be risky for your project. Or, you could have something that is adding duplicates over and over: that would be interesting to investigate.

[Munktells]

I can confirm this issue too, trying to upgrade from 2.4.2-p1 to 2.4.3. I'm using MariaDB in a multistore setup.

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '3181' for key 'PRIMARY', query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id)

@Thundar trick didn't work for me either, after running the SQL statement, I got a new error when running the command php bin/magento setup:upgrade

SQLSTATE[HY000]: General error: 1823 Failed to add the foreign key constraint dev/CAT_URL_REWRITE_PRD_CTGR_PRD_ID_CAT_PRD_ENTT_ENTT_ID to system tables, query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id), ADD CONSTRAINT CAT_URL_REWRITE_PRD_CTGR_PRD_ID_CAT_PRD_ENTT_ENTT_ID FOREIGN KEY (product_id) REFERENCES catalog_product_entity (entity_id) ON DELETE CASCADE, ADD CONSTRAINT FK_BB79E64705D7F17FE181F23144528FC8 FOREIGN KEY (url_rewrite_id) REFERENCES url_rewrite (url_rewrite_id) ON DELETE CASCADE, ADD CONSTRAINT CAT_URL_REWRITE_PRD_CTGR_CTGR_ID_CAT_CTGR_ENTT_ENTT_ID FOREIGN KEY (category_id) REFERENCES catalog_category_entity (entity_id) ON DELETE CASCADE, ADD INDEX CATALOG_URL_REWRITE_PRODUCT_CATEGORY_CATEGORY_ID_PRODUCT_ID (category_id,product_id), COMMENT=url_rewrite_relation, DEFAULT CHARSET=utf8, DEFAULT COLLATE=utf8_general_ci

[mahmoudbeikhit]

I have the Same Error after upgrade Magento form 2.4.1 to Magento 2.4.3 SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '17369' for key 'PRIMARY', query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id) And i removed all Duplicated and it work

[ihor-sviziev]

@mahmoudbeikhit you can use solution like this https://github.com/magento/magento2/issues/33770#issuecomment-898520221

[ihor-sviziev]

BTW, it feels like the declarative schema missing the feature removing duplicates like in regular MySQL connection during adding unique index

https://github.com/magento/magento2/blob/7c6b6365a3c099509d6f6e6c306cb1821910aab0/lib/internal/Magento/Framework/DB/Adapter/Pdo/Mysql.php#L2800-L2816

@maghamed what do you think about it?

[jsyvanne]

I can confirm this issue too, trying to upgrade from 2.4.2-p1 to 2.4.3. I'm using MariaDB in a multistore setup.

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '3181' for key 'PRIMARY', query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id)

@Thundar trick didn't work for me either, after running the SQL statement, I got a new error when running the command php bin/magento setup:upgrade

SQLSTATE[HY000]: General error: 1823 Failed to add the foreign key constraint dev/CAT_URL_REWRITE_PRD_CTGR_PRD_ID_CAT_PRD_ENTT_ENTT_ID to system tables, query was: ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id), ADD CONSTRAINT CAT_URL_REWRITE_PRD_CTGR_PRD_ID_CAT_PRD_ENTT_ENTT_ID FOREIGN KEY (product_id) REFERENCES catalog_product_entity (entity_id) ON DELETE CASCADE, ADD CONSTRAINT FK_BB79E64705D7F17FE181F23144528FC8 FOREIGN KEY (url_rewrite_id) REFERENCES url_rewrite (url_rewrite_id) ON DELETE CASCADE, ADD CONSTRAINT CAT_URL_REWRITE_PRD_CTGR_CTGR_ID_CAT_CTGR_ENTT_ENTT_ID FOREIGN KEY (category_id) REFERENCES catalog_category_entity (entity_id) ON DELETE CASCADE, ADD INDEX CATALOG_URL_REWRITE_PRODUCT_CATEGORY_CATEGORY_ID_PRODUCT_ID (category_id,product_id), COMMENT=url_rewrite_relation, DEFAULT CHARSET=utf8, DEFAULT COLLATE=utf8_general_ci

This is probably due catalog_url_rewrite_product_category_backup table left over from the transaction in Thundar's trick. It has foreign keys set with the same names as the magento update tries to add.

I ended up dropping the backup table and got setup to go through.

[m2-assistant[bot]]

Hi @engcom-Delta. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[Thundar]

@jsyvanne you are right. I'll edit my snippet to drop that table then.

[magenest-dev]

This issue can happen if the instance previously used the Data migration tool to import M1 data. Duplicate url_rewrite_id rows under catalog_url_rewrite_product_category but only one row has matching product_id with entity_id on url_rewrite table.

Irrelevant duplicated rows can be removed from catalog_url_rewrite_product_category with the following (backup this table first) :

DELETE c FROM catalog_url_rewrite_product_category c INNER JOIN (SELECT * FROM url_rewrite WHERE entity_type = 'product' ) s WHERE c.url_rewrite_id = s.url_rewrite_id AND c.product_id <> s.entity_id;

[WebHostingAce]

I simply emptied this table catalog_url_rewrite_product_category .

Then you can simply re-generate this table by disabling and enabling Anchor for the category.

If you have many categories. This might not be the solution.

[hostep]

@victor-v-rad: could you have a look at the remark from @magenest-dev above? Thanks! 🙂

[Hani-token]

Hi, (sorry for my frenglish)

I've solved the problem in 2 step.
Step one remove all the duplicate. Step two : overide the db_shema.xml.
<column xsi:type="int" name="url_rewrite_id" unsigned="true" nullable="false" identity="false" comment="url_rewrite_id"/>
into <column xsi:type="int" name="url_rewrite_id" unsigned="true" nullable="false" identity="true" comment="url_rewrite_id"/>

This way there is duplicate id_product id_category (like before) but no more sql duplicate entry failure. (identity=true set the autoincrement for the primary key).

I've made some test with categories configuration, mass import product. I'ven't got any error.

I'm not that expert in magento to know if there is some sides effects with how the url_rewrite_id is used in magento but for now it works for me.

[Thundar]

I have yet the same issue in production. When adding or removing products from certain categories, using csv import or category smart rules, it gives me Integrity constraint violation: 1062 Duplicate entry 'xxxxx' for key 'PRIMARY

This patch for me, after upgrading Magento with this query https://github.com/magento/magento2/issues/33770#issuecomment-898520221 (run in maintenance mode).

Patch file on magento/module-catalog-url-rewrite:

--- a/Model/ResourceModel/Category/Product.php  2021-09-27 09:49:30.000000000 +0200
+++ b/Model/ResourceModel/Category/Product.php  2021-09-27 09:49:30.000000000 +0200
@@ -53,12 +53,12 @@
     {
         $connection = $this->getConnection();
         if (count($insertData) <= self::CHUNK_SIZE) {
-            return $connection->insertMultiple($this->getTable(self::TABLE_NAME), $insertData);
+            return $connection->insertOnDuplicate($this->getTable(self::TABLE_NAME), $insertData);
         }
         $data = array_chunk($insertData, self::CHUNK_SIZE);
         $totalCount = 0;
         foreach ($data as $insertData) {
-            $totalCount += $connection->insertMultiple($this->getTable(self::TABLE_NAME), $insertData);
+            $totalCount += $connection->insertOnDuplicate($this->getTable(self::TABLE_NAME), $insertData);
         }
         return $totalCount;
     }

[RowanBurgess-zz]

I just came here to say I'm having the same issue. We have our Magento store migrated from M1 to M2 before, then went through Magento 2 upgrade using composer properly. The last upgrade we had was Magento 2.4.2-p2 to Magento 2.4.3 and.

I can confirm there is no duplicates in my table catalog_url_rewrite_product_category, and whenever there is a pair of same product_id & category_id (but different url_rewrite_id), it happens because of same url_rewrite data for 2 different stores.

I noticed the Magento 2.4.3 set url_rewrite_id to be Primary (which it wasn't in previous Magento version). vendor/magento/module-catalog-url-rewrite/etc/db_schema.xml:18

`

</constraint`

By comparing table structure between 2 versions, I found out that there is a redundant Index still existing in the table in 2.4.2 version (even after the upgrade to 2.4.3) but doesn't existing in fresh Magento 2.4.3. FK_BB79E64705D7F17FE181F23144528FC8 for url_rewrite_id col.

Simply delete this Index and it will fix your issue.

But I wonder if that's a proper way to fix this. And should Magento team have it deleted through the upgrade or not.

[davirs]

By comparing table structure between 2 versions, I found out that there is a redundant Index still existing in the table in 2.4.2 version (even after the upgrade to 2.4.3) but doesn't existing in fresh Magento 2.4.3. FK_BB79E64705D7F17FE181F23144528FC8 for url_rewrite_id col.

Simply delete this Index and it will fix your issue.

@RowanBurgess This is exactly my problem! Please, can you help me: how can I found and delete the FK_BB79E64705D7F17FE181F23144528FC8? thank you very much

[phuongleduy]

I empty table catalog_url_rewrite_product_category after that I update again by admin category. It works for me

[aelmizeb]

main.CRITICAL: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '30307912' for key 'PRIMARY', query was: INSERT INTOcatalog_url_rewrite_product_category(url_rewrite_id,category_id,product_id) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?) {"exception":"[object] (Magento\\Framework\\DB\\Adapter\\DuplicateException(code: 1062) The error occur when updating the url_key only for already existing categories and not for the new ones, so you can create a duplicate one for those you want change the url.

[m2-assistant[bot]]

Hi @engcom-Hotel. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Hotel]

Hello @burgh8wp,

Thanks for posting this issue!

We have tried to reproduce the issue after upgrading from 2.4.2-P1 to 2.4.3. Below are my findings:

• Previously in Magento 2.4.2-P1, in table catalog_url_rewrite_product_category there was no primary key. So rows can be duplicated. Please refer to the screenshot of the table structure in Magento 2.4.2-P1:
• But after upgrading in Magento 2.4.3, url_rewrite_id column is a primary key. Please find the below screenshot for reference:

So if the duplicated entries are in the url_rewrite_id column, then the issue is reproducible. But for me, I haven't faced any issue due to no duplicate entry.

Can someone please tell me the scenario where it can be duplicated without manual intervention?

Thanks

[Thundar]

Hello @engcom-Hotel , I found rows are duplicated by the use of this function: https://github.com/magento/magento2/issues/33770#issuecomment-924837177

It can be replicated by the use of Smart Rules for categories, feature from the Adobe Commerce version.

[leonhelmus]

Hey @engcom-Hotel,

Could this change be implemented that @Thundar https://github.com/magento/magento2/issues/33770#issuecomment-924837177 has written. it resolved our issues with duplicate entries. Please at this to the core. This only happens when you make use of visual merchandiser.

[alexvegas-conceptsoft]

@burgh8wp i strongly suggest NOT to do this in a production environment. START TRANSACTION; CREATE TABLE catalog_url_rewrite_product_category_temp SELECT DISTINCT * FROM catalog_url_rewrite_product_category; ALTER TABLE catalog_url_rewrite_product_category RENAME catalog_url_rewrite_product_category_backup; ALTER TABLE catalog_url_rewrite_product_category_temp RENAME catalog_url_rewrite_product_category; DROP TABLE catalog_url_rewrite_product_category_backup; COMMIT;

Edit: this query solved also the INSERT issue I quoted before. Edit2: backup table dropped as for #33770 (comment)

Hey @Thundar

in this way new table does not contains foreign keys, and than we get next error: "SQLSTATE[HY000]: General error: 1823 Failed to add the foreign key constraint dev/CAT_URL_REWRITE_PRD_CTGR_PRD_ID_CAT_PRD_ENTT_ENTT_ID to system tables,"

to prevent this just use following solution:

START TRANSACTION; CREATE TABLE catalog_url_rewrite_product_category_backup SELECT DISTINCT * FROM catalog_url_rewrite_product_category; TRUNCATE TABLE catalog_url_rewrite_product_category; INSERT INTO catalog_url_rewrite_product_category SELECT * FROM catalog_url_rewrite_product_category_backup; COMMIT;

[burgh8wp]

@alexvegas-conceptsoft How would you resolve the issue in production environment then? Our dev was direct replica of live site, and we need to get past this error to be able to update Magento to latest version. Thanks.

[engcom-Hotel]

Hello @burgh8wp,

As per my analysis in this comment, if the url_rewrite_id column has duplicate values in it, then during upgrade the issue will occur. Hence confirming this issue.

Thanks

[github-jira-sync-bot]

:white_check_mark: Jira issue https://jira.corp.magento.com/browse/AC-1947 is successfully created for this GitHub issue.

[m2-assistant[bot]]

:white_check_mark: Confirmed by @engcom-Hotel. Thank you for verifying the issue.
Issue Available: @engcom-Hotel, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[ihor-sviziev]

@sidolov, I added P2 priority, as this issue prevents Magento upgrade.

[amenk]

We tried to workaround this using

alter ignore table catalog_url_rewrite_product_category add unique index tmp_remove_key(url_rewrite_id, category_id, product_id);
alter table catalog_url_rewrite_product_category drop index tmp_remove_key

the update runs through then - when saving category - we get

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '51541850' for key 'PRIMARY', query was: INSERT  INTO `catalog_url_rewrite_product_category` (`url_rewrite_id`,`category_id`,`product_id`) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?)

So I am wondering if there is another concerning this table issue, after the update ran through?

[amenk]

Maybe related pull request: https://github.com/magento/magento2/pull/34791

[amenk]

Okay, we solved or problem here by the following workaround:

Before the upgrade we ran

alter ignore table catalog_url_rewrite_product_category add unique index tmp_remove_key(url_rewrite_id, category_id, product_id);
alter table catalog_url_rewrite_product_category drop index tmp_remove_key;

Then we were still having problems on saving categories such as @Thundar reported in https://github.com/magento/magento2/issues/33770#issuecomment-898210019

Then after the upgrade we actually run @Thundar's query:

START TRANSACTION;
CREATE TABLE catalog_url_rewrite_product_category_temp
SELECT DISTINCT *
FROM catalog_url_rewrite_product_category;
ALTER TABLE catalog_url_rewrite_product_category RENAME catalog_url_rewrite_product_category_backup;
ALTER TABLE catalog_url_rewrite_product_category_temp RENAME catalog_url_rewrite_product_category;
DROP TABLE catalog_url_rewrite_product_category_backup;
COMMIT;

Now we can save categories / change URL keys again, even without the change in my pull request #34791

I also could not reproduce it in a clean installation, but it seems to be possible to have some left-overs from old Magento versions.

Now the question I have for a better understanding: Doesn't the SELCT DISCTINCT * essentially to the same filtering as the temporary key does? So why was the issue still surfacing?

[ant0x64]

I faced the same issue in 2.4.3 during catalog importing for multi-stores. The duplication appears in \Magento\CatalogUrlRewrite\Model\ResourceModel\Category\Product::saveMultiple method on insertMultiple step (56 and 61 rows).

I tried to find out the root of the problem but have not enough time for now, so as a temporary solution I just replaced the insert function with insertOnDuplicate one and it get working.

[ihor-sviziev]

Hi @antondkv, Could you please create a new PR explaining the issue you have and the suggested changes? Thank you!

[amenk]

@ihor-sviziev My Pullrequest https://github.com/magento/magento2/pull/34791 does something similar like @antondkv approach - no clue what's better...

I realized today, that https://github.com/magento/magento2/issues/33770#issuecomment-898520221 is just dropping all the contraints as a side-effect, so it is no suprise that they key violation vanish after that.

[Thundar]

@amenk The constraints are not present before https://github.com/magento/magento2/issues/33770#issuecomment-898520221 . It is meant to solve the error before this query is run in setup:upgrade. ALTER TABLE catalog_url_rewrite_product_category ADD CONSTRAINT PRIMARY KEY (url_rewrite_id); So you should have the constraints after the setup:upgrade.

[amenk]

@thundar I actually was running it after the upgrade because I still got unique key constraint errors on category save. That problems then disappeared, because the unique-key-constraints were not there any more and I wondered why.

I still think it is safer to filter the table using

alter ignore table catalog_url_rewrite_product_category add unique index tmp_remove_key(url_rewrite_id, category_id, product_id);
alter table catalog_url_rewrite_product_category drop index tmp_remove_key;

Because recreating the table not only removes unique-key-constraints but also other indices and foreign key constraints.

But at the end of the day, with the constraints there, Magento still tries to insert duplicates, which needs to be fixed.

[Thundar]

@amenk I use this to prevent duplicates from both category save and csv import: https://github.com/magento/magento2/issues/33770#issuecomment-924837177

[ihor-sviziev]

It looks like this issue was caused by https://github.com/magento/magento2/issues/34210 + in 2.4.3 was added a unique key on this table.

The correct fix will be following:

1. Execute the following SQL before running php bin/magento setup:upgrade command:

   START TRANSACTION; 
   CREATE TABLE catalog_url_rewrite_product_category_backup SELECT * FROM catalog_url_rewrite_product_category;
   TRUNCATE TABLE catalog_url_rewrite_product_category;
   INSERT INTO catalog_url_rewrite_product_category SELECT DISTINCT * FROM catalog_url_rewrite_product_category_backup; 
   COMMIT;

After the successful upgrade and making sure everything works fine, remove the backup table

DROP TABLE catalog_url_rewrite_product_category_backup;

2. Apply as a patch changes from the following PR: https://github.com/magento/magento2/pull/34791