Memory leak in EAV plugin

[pgmassari]

I write here since I can't open an issue at: https://github.com/quickapps-plugins/eav

In my cakephp 3.2.5 app I'm using the EAV plugin, and it's working as expected.

Problem: I also need to cycle through several thousand rows of CSV, each row describing an entity with possible EAVs, to import them into the app, all within a cakephp transactional() call.

Within the cycle I need to call find() on my table model, before deciding whether to call newEntity()or patchEntity(), and then call save().

I noticed a severe memory leak when cycling and, thanks to logging memory_get_usage results, I pin-pointed the cause to the find() on the table model that has the EAV behavior loaded.

If I repeat the cycle disabling just the EAV behavior on the same table, no memory leak happens (and my import procedure succeds, except for not patching what should be an EAV, of course).

[botchris]

Hi there,

I had a very similar issue some time ago when I was trying to import 60K entities reading from a XLS file. Although I'm not 100% sure, I think it was caused by a PDO's memory leak issue, using find() on every iteration before save() causes somehow a memory leak (query results cache or something).

For every EAV entity being saved, EAV plugin triggers a find() & save(). So I think what you are experiencing is because EAV plugin is contributing to bloat a bit faster something that would occur anyway.

---

I think we should try some different scenarios in order to detect if this is actually being caused by EAV plugin or PDO:

Table with no EAV plugin:

• Import x4 more entities from CSV.
• Import x4 more entities from CSV, and after every save() perform a random find() (get the first entity on the table for instance)

(All this within a single loop.)

---

PS: In my case I solved this problem by processing every XLS's row on a different php process.

[pgmassari]

Hi, thanks for your considerations.

I can't test it in this very moment, and I didn't save the memory_get_usage logs, but I recall clearly it was something along the following:

Situation A:

• EAV behavior not loaded.
• Importing 10k rows, looping in 10 of 1k rows at the time, within a transactional() (I don't think this is significant, though).
• For each row: parse csv, then use find() to see if it's a brand new entity or needs to be patched, then use newEntity() or patchEntity(), then save().
• Entities are quite normal typical Products stuff, with more or less 20-30 fields.

Situation B:

• EAV behavior loaded.
• same as A, but with 10 EAV fields more. The file is exactly the same. In situation A i just ignore EAV fields when i read them.

Starting memory_get_usage of both situations: more or less 8-9 MB

Situation A: memory_get_usage increasing by less than 0.5 MB every loop (this memory leak is probably due to my setup, no big deal) after each loop, ending at about 14-15 MB.

Situation B: memory_get_usage increasing several MB every loop (scaling up each step), exhausting allowd memory of 256 MB more or less after 5-6 iterations.

Anyway, I will try to perform the tests and keep you posted.

[botchris]

This two patches (bbc26000837286ec1e4bfcacaa8e6945e8aecc0e & 65923fad4264589ff644383327e43f07249709c7) should solve the problem.

It was easy to reproduce but very difficult to fix. Here is my testing environment:

• Inserts 1000 new entities with 3 eav columns each, all random values.
• 1000 more iterations to update all previously inserted entities. That is: find(), patch() and save(). Again, patch with all random values

Conslusion: memory stay always below 6.5MB

Please let me know if it is now working for you.