[BUG] Cannot using multiple servers and import excel file using queue

[mtawil]

Prerequisites

• [x] Checked if your Laravel Excel version is still supported: https://docs.laravel-excel.com/3.1/getting-started/support.html#supported-versions
• [x] Able to reproduce the behaviour outside of your code, the problem is isolated to Laravel Excel.
• [x] Checked that your issue isn't already filled.
• [x] Checked if no PR was submitted that fixes this problem.
• [x] Filled in the entire issue template

Versions

• PHP version: 7.3.9
• Laravel version: 5.8.35
• Package version: 3.1

Description

I have multiple servers running the same application, and all runs queue worker pointing to one Redis cluster as a queue driver, sometimes when trying to import an excel file using queue way, I get this error File not found at path: laravel-excel-**********.

I also set the remote disk option as s3, and I am sure that all the configurations are correct.

Steps to Reproduce

It seems like the following jobs sometimes run on different servers:

• AppendQueryToSheet
• CloseSheet
• StoreQueuedExport

I think that one of the first queues created an excel file locally then upload it remotely. In the next queue, specifically StoreQueuedExport had run on another worker and trying to get the same file locally again, and it fails because it can't find the file.

[mtawil]

Figured out the issue, it's because of AppendQueryToSheet job, it gets dispatched many times based on the chunk count, so when the queue worker is running on multiple servers, the job will be randomly running by another server and the initiated laravel-excel file will be not founded.

[patrickbrouwers]

Best to make sure only 1 worker can pick up those jobs?

[mtawil]

TL;DR:

The issue was coming from the RemoteTemporaryFile class that using a getLocalPath and called the same method on LocalTemporaryFile class: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/Files/RemoteTemporaryFile.php#L51

We should find a way or do something on this method to be entirely remotely.

TS;WM:

Let's reproduce it by installing a Laravel application on five servers, with php artisan queue:work running on all of it, and then we will set:

• exports.chunk_size = 100
• temporary_files.remote_disk = s3

When we request to export, e.g. 10,000 records, the initiated job will be handled by the server 3, which is will be the first who create the excel file.

Let's go to the class who dispatch the AppendQueryToSheet: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/QueuedWriter.php#L164

As we can see, the AppendQueryToSheet job will be pushed 100 times in our case, and it will be distributed to multiple servers in random, including the server three which had initiated the file.

• Expose the handle method of AppendQueryToSheet class: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/Jobs/AppendQueryToSheet.php#L69

• Expose the reopen method of Writer class: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/Writer.php#L111

• Expose the getLocalPath method, which is, in our case, in the RemoteTemporaryFile class: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/src/Files/RemoteTemporaryFile.php#L51

You can see that the other servers are asking for the local file, not the remote one, which is not existing.

That's it.

[patrickbrouwers]

Hm, how it should work is that the remote temporary file copies itself to a local file and after writing, copies itself back to the remote file. That's what the sync() method does.

[moyvera]

same issue here, for now do you have a workaround ?

I replicated this issue using laravel horizon with multi process and multi instances of my app.

[mtawil]

Hello @patrickbrouwers

Hm, how it should work is that the remote temporary file copies itself to a local file and after writing, copies itself back to the remote file. That's what the sync() method does.

That's not true; there is no sync when the queued job spreads in the other servers.

[haihv96]

I have same issue. the problem is worker queue run on another server.

[stale[bot]]

This bug report has been automatically closed because it has not had recent activity. If this is still an active bug, please comment to reopen. Thank you for your contributions.