Closed pgmassari closed 7 years ago
Hi there,
I had a very similar issue some time ago when I was trying to import 60K entities reading from a XLS file. Although I'm not 100% sure, I think it was caused by a PDO's memory leak issue, using find()
on every iteration before save()
causes somehow a memory leak (query results cache or something).
For every EAV entity being saved, EAV plugin triggers a find()
& save()
. So I think what you are experiencing is because EAV plugin is contributing to bloat a bit faster something that would occur anyway.
I think we should try some different scenarios in order to detect if this is actually being caused by EAV plugin or PDO:
Table with no EAV plugin:
save()
perform a random find()
(get the first entity on the table for instance)(All this within a single loop.)
PS: In my case I solved this problem by processing every XLS's row on a different php process.
Hi, thanks for your considerations.
I can't test it in this very moment, and I didn't save the memory_get_usage
logs, but I recall clearly it was something along the following:
Situation A:
transactional()
(I don't think this is significant, though).find()
to see if it's a brand new entity or needs to be patched, then use newEntity()
or patchEntity()
, then save()
.Products
stuff, with more or less 20-30 fields.Situation B:
Starting memory_get_usage
of both situations: more or less 8-9 MB
Situation A: memory_get_usage
increasing by less than 0.5 MB every loop (this memory leak is probably due to my setup, no big deal) after each loop, ending at about 14-15 MB.
Situation B: memory_get_usage
increasing several MB every loop (scaling up each step), exhausting allowd memory of 256 MB more or less after 5-6 iterations.
Anyway, I will try to perform the tests and keep you posted.
This two patches (bbc26000837286ec1e4bfcacaa8e6945e8aecc0e & 65923fad4264589ff644383327e43f07249709c7) should solve the problem.
It was easy to reproduce but very difficult to fix. Here is my testing environment:
Conslusion: memory stay always below 6.5MB
Please let me know if it is now working for you.
I write here since I can't open an issue at: https://github.com/quickapps-plugins/eav
In my cakephp 3.2.5 app I'm using the EAV plugin, and it's working as expected.
Problem: I also need to cycle through several thousand rows of CSV, each row describing an entity with possible EAVs, to import them into the app, all within a cakephp
transactional()
call.Within the cycle I need to call
find()
on my table model, before deciding whether to callnewEntity()
orpatchEntity()
, and then callsave()
.I noticed a severe memory leak when cycling and, thanks to logging
memory_get_usage
results, I pin-pointed the cause to thefind()
on the table model that has the EAV behavior loaded.If I repeat the cycle disabling just the EAV behavior on the same table, no memory leak happens (and my import procedure succeds, except for not patching what should be an EAV, of course).