search

WeareJH / m2-module-import

Import module for Magento 2 - provides commands, utilities and abstractions for building imports for Magento 2 projects
MIT License
19 stars 8 forks source link

Skip unchanged records filter #22

Closed AydinHassan closed 4 years ago

AydinHassan commented 4 years ago

Introduce a new import filter, which, at the beginning of an import, will load the file from the very last import of the same name/type as the one processing. It will hash each row of the previous import and keep a copy of the data in memory.

For each row in the new import, we hash it and check if the hash exists already (in the previous dataset) if it does, we skip it. We essentially produce a delta, on the fly.

codecov-commenter commented 4 years ago

Codecov Report

Merging #22 into master will increase coverage by 0.07%. The diff coverage is 64.78%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master      #22      +/-   ##
============================================
+ Coverage     62.00%   62.08%   +0.07%     
- Complexity      478      497      +19     
============================================
  Files            76       79       +3     
  Lines          1966     2036      +70     
============================================
+ Hits           1219     1264      +45     
- Misses          747      772      +25
Impacted Files Coverage Δ Complexity Δ
src/Archiver/Csv/Entity/ArchiveResource.php 0.00% <0.00%> (ø) 3.00 <3.00> (?)
src/Entity/ImportHistoryResource.php 0.00% <0.00%> (ø) 4.00 <3.00> (+3.00)
src/Filter/SkipUnchangedRecordsFromLastImport.php 97.36% <97.36%> (ø) 8.00 <8.00> (?)
src/Archiver/Csv/Entity/Archive.php 100.00% <100.00%> (ø) 5.00 <5.00> (?)
src/Source/SourceConsumer.php 100.00% <100.00%> (ø) 1.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 110c55a...481f4d1. Read the comment docs.