search

PrestaShop / pscleaner

This module allows you to check and fix functional integrity constraints and remove default data
http://www.prestashop.com
Academic Free License v3.0
36 stars 107 forks source link

Product Import does not clean Accessory and Attribute tables, if requested to clean Product data. #60

Open k0lv opened 4 years ago

k0lv commented 4 years ago

Describe the bug I was importing Products with Accessories, and in the results, have seen, that Products are linked incorrectly(not 100% as I defined). Some part were as I described in my .xlsx, but there were also some random relations. I thought, that the error is is import functionality, or in the source data, but finally found out two reasons: 1) I was importing data to PS several times, and every time used PrestaShop Cleaner v2.0.0 to clean the Product DB. I have found out, that 3 tables, are not cleaned by the Module: a) ps_accessory b) ps_attribute c) ps_tagcount // (I was not able to find out how to register issue for the module, sorry.)_ 2) Before importing, I was also setting parameter "Delete all products before import" to YES, but this also does not cleaned data.

Results (Issue): As a result I receive new data-base of products, but with partially old Accessories. Here is my issue.

Additional observations: As for ps_attribute and ps_tag_count, I do not know, if old data, that remains in tables can cause the issues for the future clean imports. I think it was worth to mention here. Maybe you can comment on that. Thank you.

Edit by Khouloud: steps to reproduce the issue

  1. install PS1.7.6.2

  2. Go to the Import page => import products Products_1.xlsx 2.1 Delete all products before import 2.2 Use product reference as a key

  3. check the ps_accessory table => ok image

  4. Go to the Import page => import products Products_2.xlsx 4.1 Delete all products before import 4.2 Use product reference as a key

  5. check the ps_accessory table => NOK image

To Reproduce Steps to reproduce the behavior:

  1. Import 1st portion of Products with Accessories.
  2. Use PS Cleaner to delete all Product Catalog.
  3. Import 2nd portion of Products with Accessories with "Delete all products before import" set to YES. As a result you will get mixture of New and Old product relationships.

Additional information PrestaShop version: 1.7.6.2 PHP version: 7.0.33

image

khouloudbelguith commented 4 years ago

Hi @k0lv,

I did not manage to reproduce the issue with PS1.7.6.2 & pscleaner v2.0.0. I used this csv file product.csv.zip I attached a screen record https://drive.google.com/file/d/1Cq1y1skIqeVt4T94T12u50htvC-ebgHv/view Thanks to check & feedback.

k0lv commented 4 years ago

Hi @k0lv,

I did not manage to reproduce the issue with PS1.7.6.2 & pscleaner v2.0.0. I used this csv file product.csv.zip I attached a screen record https://drive.google.com/file/d/1Cq1y1skIqeVt4T94T12u50htvC-ebgHv/view Thanks to check & feedback.

Hello, @khouloudbelguith. I think, that the main difference for yours and mine scenario is, that you are using Product ID, instead of Reference#. The trick is that ps_accessory table is working with ID's. So, if after cleaning, ps_attribute table has data for the old Product ID's (relationships), all the data will be corrupted, because in my case new portion of import will generate new ID's for the products (second import file, in my case had different sequence of the products, and as a results different ID's generated). You can easily have an evidence if you will check ps_accessory, ps_attribute and ps_tag_count tables in phpMyAdmin, after PS Cleaner, or Clean Product Import (with deletion) activity. Thank you for the fast response and Happy New Year! 🎄

1) Importing first portion to the fully clean DB: image image RESULTS: image

2) Delete the Catalogue: image RESULTS: Accessory table unchanged: image

3) Import new product list: image image

RESULTS: image

Issue: Old relationships applied to new product list.

P.S. I have found, that ps_attribute and ps_tag_count tables also retain historical data. Yet I do not know if that fact can cause any data integrity issues in the future.

k0lv commented 4 years ago

Workaround: Found funny fact, that PS Cleaner "FUNCTIONAL INTEGRITY CONSTRAINTS" option, actually cleans the Accessory table. :) Wondering, why this is not done as a part of "CATALOG DELETION"... image

khouloudbelguith commented 4 years ago

@k0lv, could you please provide me the two xlsx file used in your case to test it. First, I used this file product.csv.zip using this configuration image Second, I used this file product2.csv.zip using this configuration image I checked the ps_accessory table => it is empty, but I checked in the BO & FO => related products well added. https://drive.google.com/file/d/1qfHLLRoV6bQBHMUSX-z5wfJG8tj6VY6r/view

Thanks!

k0lv commented 4 years ago

Products_1.xlsx Products_2.xlsx @khouloudbelguith, adding both files, that I have used to replicate the issue.

  1. I have used "Delete all products..." for all the imports.
  2. Now I am very confused. How can you explain, that ps_accessory is blank, but products are related. This sounds like a magic for me, but I am not so familiar with the PS data structure, maybe I am wrong.

Update: If you use Product Deletion function in Catalog, Accessory table is cleaned out: A) image B) image

khouloudbelguith commented 4 years ago

@k0lv, thanks for these clarifications. I manage to reproduce the issue with PS1.7.6.2 & PS1.7.5.2 following these steps:

  1. install PS1.7.6.2

  2. Go to the Import page => import products Products_1.xlsx 2.1 Delete all products before import 2.2 Use product reference as a key

  3. check the ps_accessory table => ok image

  4. Go to the Import page => import products Products_2.xlsx 4.1 Delete all products before import 4.2 Use product reference as a key

  5. check the ps_accessory table => NOK image

I’ll add this to the debug roadmap so that it’s fixed. If you have already fixed it on your end or if you think you can do it, please do send us a pull request! Thanks!