rap2hpoutre / fast-excel

🦉 Fast Excel import/export for Laravel
MIT License
2.08k stars 246 forks source link

Export large collections with chunk and relation #319

Open zotopteam opened 1 year ago

zotopteam commented 1 year ago

This is described in the documentation

Export large collections with chunk

Export rows one by one to avoid memory_limit issues [using yield](https://www.php.net/manual/en/language.generators.syntax.php):

function usersGenerator() {
    foreach (User::cursor() as $user) {
        yield $user;
    }
}

// Export consumes only a few MB, even with 10M+ rows.
(new FastExcel(usersGenerator()))->export('test.xlsx');

Problems and solutions

There is no problem with this code, but the problem occurs when exporting complex data. If an export has relation data, there will be problems using cursors (cursors do not support relation, see laravel official documents for details). This will lead to a large number of queries. At this time A better way is to use lazy (lazy also uses yield),

For example, if we want to export the user's group name and school name at the same time, using cursor, each school and group will have to be queried separately. with does not work. There is no problem with using lazy instead, and the speed is much faster.

Export rows one by one to avoid memory_limit issues [using yield](https://www.php.net/manual/en/language.generators.syntax.php):

function usersGenerator() {
    foreach (User::with(['group','school'])->lazy() as $user) {
        yield $user;
    }
}

// Export consumes only a few MB, even with 10M+ rows.
(new FastExcel(usersGenerator()))->export('test.xlsx');

But what needs to be careful is the coexistence of lazy and limit. When using lazy, limit does not take effect.

The above are the problems I encountered in use and the solutions I found. Please refer to them. If there are any problems, please remind me