Long-running export consumer process blocks everything else

[erfanimani]

The point of consumer processes is that they're run asynchronously and independently of each other. This module seems to break that.

We've got a store with around 25K products, producing a CSV of around 50MB. The exportProcessor consumer takes around 5 to 10 minutes to complete an export. During the entire duration, everything else is blocked.

We've also had to increase "Consumers Job Timeout" to 600 seconds (the default of 30 is way too low), and disable "Consumers Govenor" in order to get this anywhere near working.

This looks like a big design flaw. A queue consumer should not be allowed to block cron, causing delayed order confirmation emails for example.

[gnuzealot]

@erfanimani there is a setting for the concurrent jobs that can be run at a time, default is 2. If you have one long running cron it's going to be a limit on the other so you probably just need to update the setting to 3 to allow all jobs to run.

[erfanimani]

@gnuzealot few things that I did that worked:

• Maximum cron processes: 7
• Max load average: 24
• Consumers job timeout: 600
• Exporters job timeout: 3600
• Turned off consumers governor (this seemed to kill consumer processes in the middle of their task)