search

genecommerce / module-encryption-key-manager

Tooling to help generate and invalidate magento encryption keys
GNU Lesser General Public License v3.0
54 stars 14 forks source link

Improve performance with large data sets #36

Open convenient opened 1 month ago

convenient commented 1 month ago

resolves https://github.com/genecommerce/module-encryption-key-manager/issues/9

To improve pefromance

TODO

Summary

Baseline build

This was generated by making a branch with the original ChangeEncryptionKey.php file and seeing how it peformed in the same way.

For posterity https://github.com/genecommerce/module-encryption-key-manager/pull/41

Generating a new encryption key

real    8m46.067s
user    1m57.702s
sys     0m54.372s
PASS
A new key will be generated for re-encryption, use "--key" to specify a custom key.
The system currently has 1 keys
Generating a new encryption key using the magento core class
_reEncryptSystemConfigurationValues - start
_reEncryptSystemConfigurationValues - end
_reEncryptCreditCardNumbers - start
_reEncryptCreditCardNumbers - end
reEncryptEnvConfigurationValues - start
reEncryptEnvConfigurationValues - end
Cleaning cache
Done
cc_number_enc
1:3:Y+Vfqd/P2VFws/1d1l3KUm7DefiEK+8UuPXsjZLPy5yiniWFWUbV4S7YiMQ9VQWV
1:3:ObbdarSNVDDDlbp+Wzs8lkrUNGGSXiEnKNa6J2qztFchM5w8ewKArbyitG95spU1
1:3:dVCvUsULbR5MThPjiZupC96cH/BsIzAKI9Xt3kEDA3PHAktee1wSnLq5WUlVwfJ3
1:3:f42nrfoDfbZxOah560YcrdaD964ClS98QDv5sIi9pM87rcxpamt84r8Y1YS8Ngaa
1:3:pmMR3kjiuo9Ffqrx+a5I3VCyGtjCHeRc6hVQgz+m4zcQvf7QW7I9csNT86un/eUH

New build

https://app.circleci.com/pipelines/github/genecommerce/module-encryption-key-manager/100/workflows/31a8735c-5a08-4add-a5c2-13e1d6c452f9/jobs/94

Generating a new encryption key

real    1m4.381s
user    0m11.750s
sys     0m1.129s
PASS
A new key will be generated for re-encryption, use "--key" to specify a custom key.
The system currently has 1 keys
Generating a new encryption key using the magento core class
_reEncryptSystemConfigurationValues - start
_reEncryptSystemConfigurationValues - end
_reEncryptCreditCardNumbers - start
_reEncryptCreditCardNumbers - total possible records: 2500000
_reEncryptCreditCardNumbers - batch size:             10000
_reEncryptCreditCardNumbers - batch count:            250
_reEncryptCreditCardNumbers - total records updated:  2500000
_reEncryptCreditCardNumbers - end
reEncryptEnvConfigurationValues - start
reEncryptEnvConfigurationValues - end
Cleaning cache
Done
cc_number_enc
1:3:pU72eUmaUyl8zt/+6urRRwXZkQOhOLH5f46wXQvnsIpeX7wmL1R88+scz+rw4utf
1:3:pDFpTbclXYOekn0qhbVmlLYkBWdpksAKe8/2azFXxbaNwpCmiSscdqn30PdTWv1j
1:3:IY0C8vwgATh4xa6Tlz1f7SBgQ77Db9jityxA3JAWSdR/Fhm0aPxWVDiogINr/mIa
1:3:CUbE6GMFaAgOm0Hn3y1FsofhCsnwYh+q9nJX7alYzbgkXkg7zhDqZ9sjjuRIhP+t
1:3:HUG3gXyIXJs7MvdaJqbT3cg9+5qR3up9lb3k5MoxF7VwSpBRYtXzArwjO1qlG6nU

Extra large data set

25 mil rows

Generating a new encryption key

real    11m0.911s
user    1m58.354s
sys     0m10.758s
PASS
A new key will be generated for re-encryption, use "--key" to specify a custom key.
The system currently has 1 keys
Generating a new encryption key using the magento core class
_reEncryptSystemConfigurationValues - start
_reEncryptSystemConfigurationValues - end
_reEncryptCreditCardNumbers - start
_reEncryptCreditCardNumbers - total possible records: 25000000
_reEncryptCreditCardNumbers - batch size:             10000
_reEncryptCreditCardNumbers - batch count:            2500
_reEncryptCreditCardNumbers - total records updated:  25000000
_reEncryptCreditCardNumbers - end
reEncryptEnvConfigurationValues - start
reEncryptEnvConfigurationValues - end
Cleaning cache
Done
cc_number_enc
1:3:8hhO5NotWUFLfPAzybRUpojMCZoinhlvWjwQz7h5fY7kWqg4XEgou4IoRtcT50Gj
1:3:9/MFBMZzd/keueHRSYLaB3+WKnbHpdJiATswS8C+xqjdjk2X4TvS71fTPwWRUAoS
1:3:FwMeT8GBvLoeqSN59cvxfCKhmfB/LvgJoX4DoIComQ7hFVdn/GVDsnrRfzgfGtz9
1:3:UYp/qBO3yS/QropCcwsh3VCSlKUHUT0Nhbe79vTMM6FvG99wV27yje6yqhHRxL0a
1:3:O6K5ZOgzFDdg2B5tlAN+Gh9B9dXg1Kn8zyIp5BD5bsAdYYdUAOFLMWp0y88qh3+c