[QUESTION] What is the point of having chunkSize for ToCollection?

[ghost]

Prerequisites

• [X] Have read and understood: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/.github/SUPPORT.md
• [X] Checked if your Laravel Excel version is still supported: https://docs.laravel-excel.com/3.1/getting-started/support.html#supported-versions
• [X] Checked that your question isn't already asked before.
• [X] Filled in the entire issue template

Versions

• PHP version: 7.2
• Laravel version: 6 (LTS)
• Package version: 3.1

Description

Hello guys,

I'm currently having a hard time understanding why is useful to use chunkSize when making a import to a collection. The point of the chunk, as I understand, is to not have to deal with the whole file at once on main memory, but if you're chunking to a collection in the end you're going to have the whole file on primary memory anyway, even if you do it by parts.

For example, if you have a .xlsx file that you want to open, make some kind of processing with collections and save it again as a .csv, making it by chunks wouldn't really make any difference because in the end you would need to save the file as whole anyway. In that case, what would be the best way (performance wise) to achieve the task?

I don't know if I'm understanding something fundamentally wrong, so any clarification would be well appreciated :))

Cheers!

[patrickbrouwers]

Hey @cegafe3, if you use chunkSize with collection it won't load the entire file into memory. It will call the collection() method multiple times each time with another batch of rows.

You could perhaps store it in chunks in a temporary database (myql, redis, ...) and then insert it back into a file in chunks.