Big data download not supported

[phoenixg]

Prerequisites

[Y] Able to reproduce the behaviour outside of your code, the problem is isolated to Laravel Excel. [Y] Checked that your issue isn't already filed. [Y] Checked if no PR was submitted that fixes this problem.

Versions

PHP version: 7.3 Laravel version: 5.8.17 Nova version: 2.0.5 Package version: 3.1.13 and 1.1.4

Description

I've got more than 10, 000 records in my table. When I use Download Excel, it just nothing happens even though I use the chunk method.

 ((new DownloadExcel())->withHeadings()->withChunkCount(300)),

I thought it may work in chunk mode, but seems not.

[patrickbrouwers]

It's either your PHP (or apache/nginx) process timing out (maximum execution time) or memory being exhausted. You should be able to find out why it's not responding by looking in your PHP logs or laravel logs.

[ragingdave]

I've just hit the same issue, attempting to export anything larger than about 10K records results in extremely high memory usage and the queued process timing out. I have horizon configured to at max use 512MB, but this hits that limit at about 9K for xlsx exports and about 10K for csv exports. I'm not quite sure but based on memory usage just building and building there is either a very large performance issue or a memory leak as I would expect chunked jobs to keep increasing memory usage, but rather be more uniform.

I also don't really feel good about increasing memory beyond 512 as that's already alot of memory to alocate per worker, and quite frankly don't understand how this (or probably more specifically the phpspreadsheet package) is using so much memory to just write files.

[ghost]

+1 on this issue. Most of my use cases involve >10k records

[matthewjumpsoffbuildings]

is there any updates on this? some of my use cases have 100's of thousands of records, sometimes even a few million, and im at a loss as to how to handle it

[ragingdave]

I haven't a chance to test it, because I already kind of lost trust in this package to begin with due to how long big data was an issue, but seemingly laravel sort of fixed the issue for them with lazy collections?

https://github.com/Maatwebsite/Laravel-Excel/issues/2209#issuecomment-593596708 Again unsure if it actually fixes anything, because I thought the original issue was related to how php spreadsheet works and not around collection/query handling but I guess it's something

[ragingdave]

@matthewjumpsoffbuildings @MaatwebsiteSupport I take back my potential hope here... Just found out that the latest version removes the version constraint and just marks QueuedExports as "deprecated" since another fix in the main package broke that feature entirely. That's not deprecated....that's broken.

So now, anyone just doing a simple composer update, could find that their entire export system is broken as a result of a point release?? Maybe it's just me, but I'm basically done with this package. After months of no response around big data issues, then just hoping that laravel framework enhancements fixes it, then breaking a core feature of this package and the base package....I have completely lost faith that this package is being maintained in a way that I can depend on it. This maybe just me venting a bit, but to have now twice where the exports just break without any real warning as a result of a minor update??? That's unacceptable to me. It's not even a "oh we should revert that and find another way", it's "oh that broke.....alright well that's no longer a feature anymore...". If anyone sees this....I would honestly just find another way to handle what this does or write your own....at least it won't just randomly break

[patrickbrouwers]

We are trying to fix the performance issues, which means we cannot longer support the old queuing way. I had it locked on an older version, which blocked Laravel 7 upgrades. We are honestly trying our best but appreciation and understanding unfortunately seems far off :(

[matthewjumpsoffbuildings]

so i guess @ragingdave is right, this is not a viable solution, and wont be any time soon :/

[patrickbrouwers]

It’s the only solution that is currently possible. I simply don’t have the time to work on it more than I currently do. It’s open source, I don’t get paid to work on this.

[ragingdave]

I understand that at this time, it's the only solution that you can come up with.

I certainly appreciate open source work and this package specifically was one that saved me a ton of time and I would love to keep using it. However, if you as the maintainer find it acceptable to just break and/or remove features that currently are in use and advertised as a core feature...then I'm sorry but I just don't have time to keep re-testing your or anyone's package that I'm utilizing to ensure that on a minor fix version update, that a core feature hasn't just been removed, or a feature just stops working.

I understand that you don't get paid for this, and I appreciate the time you put in that you can, but you also have to understand that if your package is practically beta level stability where things break all the time....then it just isn't something that I and people like me will end up using.

Frankly, you trying to play the victim here is the part I don't understand. You now are forcing me and others using the features you just randomly removed, to either be unable to upgrade to laravel 7/nova 3 or completely re build out this feature for ourselves. For some it will be easier than others, but it's still completely unacceptable, because while you aren't being paid to work on this, we are being paid to build things using this package and choosing to use this package was supposed to include some amount of trust. That trust for me, has been broken so I won't be using this or pretty much any other package where you are the maintainer because of that.

[patrickbrouwers]

I know it sucks when things break unintentionally, but it can happen. In no way we are changing code to screw people over. The software is open source and 100% optional, we try to support it on best effort and best intention. You are always able to write it yourself, use another piece of software, contribute to this one; nobody (and certainly not me) is forcing you to use this package.

[matthewjumpsoffbuildings]

@patrickbrouwers its been over 2 years. Theres whole sections on your documentation about queued exports, but it seems like it just breaks in my local test environment. I'm assuming its still running into the issue that the file thats being written has to be opened into memory. Whatever it is, it would be great to have some updates.