search

ddeboer / data-import

Import data from and export data to a range of different file formats and media
MIT License
567 stars 123 forks source link

DoctrineWriter : Disable truncate if an 'index' is passed to the constructor ? #121

Open LouTerrailloune opened 10 years ago

LouTerrailloune commented 10 years ago

Shall we disable it automatically when something is passed in $index ? (otherwise $index is ignored if we don't call setTruncate(false).

Baachi commented 9 years ago

I would rather change the default value form true to false.

LouTerrailloune commented 9 years ago

Not sure it's a good idea to set 'truncate' to false by default. Some people are certainly using it like that (truncating the table).

Setting truncate to false if $index is passed to the constructor is just a small BC break (because if people are using $index, they already called setTruncate(false)).

Another solution is to remove the test in findOrCreateItem() and check for in memory objects. (duplicate will be avoided)

Baachi commented 9 years ago

@LouTerrailloune You are right, it might not be the best idea. I only want to avoid side effects or some magic. I'm okay with that if you leave a note in the docblock and Readme.

To your second solution: Its a great idea, but i think this will the grown up the memory usage. Doctrine does the same and if you have a large object graph (~10000 Objects) you have a huge memory usage. This is why don't use the DoctrineWriter only for rare cases.

LouTerrailloune commented 9 years ago

For second solution: the batch size is here to limit memory usage. We need to keep a reference to the objects in the batch only (+ the key), and we do the query too. When we flush a batch, we remove our reference, the object can be removed from memory since they are not referenced anymore.