In SQS FIFO, task consumption temporarily stops when a large number of workers are started

[marvelph]

Checklist

• [ ] I have verified that the issue exists against the master branch of Celery.
• [ ] This has already been asked to the discussion group first.
• [x] I have read the relevant section in the contribution guide on reporting bugs.
• [x] I have checked the issues list for similar or identical bug reports.
• [x] I have checked the pull requests list for existing proposed fixes.
• [ ] I have checked the commit log to find out if the bug was already fixed in the master branch.
• [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• [ ] I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
• [ ] I have verified that the issue exists against the master branch of Celery.
• [x] I have included the contents of pip freeze in the issue.
• [ ] I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

• [ ] I have tried reproducing the issue on more than one Python version and/or implementation.
• [ ] I have tried reproducing the issue on more than one message broker and/or result backend.
• [ ] I have tried reproducing the issue on more than one version of the message broker and/or result backend.
• [ ] I have tried reproducing the issue on more than one operating system.
• [ ] I have tried reproducing the issue on more than one workers pool.
• [ ] I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
• [x] I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version:4.4.7 and 5.1.2

celery report Output:

``` ```

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

``` alabaster==0.7.12 amqp==5.0.6 asgiref==3.3.4 Babel==2.8.0 beautifulsoup4==4.9.3 billiard==3.6.4.0 boto3==1.16.10 botocore==1.19.10 celery==5.1.2 certifi==2020.6.20 chardet==3.0.4 click==7.1.2 click-didyoumean==0.0.3 click-plugins==1.1.1 click-repl==0.2.0 defusedxml==0.6.0 diff-match-patch==20200713 dill==0.3.3 Django==3.2.4 django-amazon-ses==4.0.0 django-appconf==1.0.4 django-bootstrap-breadcrumbs==0.9.2 django-bootstrap-datepicker-plus==3.0.5 django-bootstrap-form==3.4 django-bootstrap4==2.3.1 django-braces==1.14.0 django-chartjs==2.2.1 django-datatables-view==1.19.1 django-debug-toolbar==3.1.1 django-decorator-include==2.1 django-elasticache==1.0.3 django-environ==0.4.5 django-extra-views==0.13.0 django-formtools==2.2 django-imagekit==4.0.2 django-import-export==2.4.0 django-modelsdoc==0.1.11 django-pandas==0.6.4 django-storages==1.10.1 django-summernote==0.8.11.6 django-wkhtmltopdf==3.3.0 docutils==0.14 et-xmlfile==1.0.1 factory-boy==3.1.0 Faker==4.14.0 flower==0.9.5 humanize==3.1.0 idna==2.10 imagesize==1.2.0 jaconv==0.2.4 jdcal==1.4.1 Jinja2==2.11.2 jmespath==0.10.0 klepto==0.2.0 kombu==5.1.0 MarkupPy==1.14 MarkupSafe==1.1.1 numpy==1.18.4 odfpy==1.4.1 openpyxl==3.0.5 packaging==20.4 pandas==1.2.3 pilkit==2.0 Pillow==8.0.1 pox==0.2.9 prometheus-client==0.8.0 prompt-toolkit==3.0.19 psycopg2-binary==2.8.6 pycurl==7.43.0.6 Pygments==2.7.2 pykakasi==2.0.1 pylibmc==1.6.1 pyparsing==2.4.7 python-dateutil==2.8.1 python-magic==0.4.18 pytz==2020.4 PyYAML==5.3.1 redis==3.5.3 requests==2.24.0 s3transfer==0.3.3 six==1.15.0 snowballstemmer==2.0.0 soupsieve==2.0.1 Sphinx==3.2.1 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 sqlparse==0.4.1 tablib==2.0.0 tenacity==6.2.0 text-unidecode==1.3 tornado==6.1 urllib3==1.25.11 uWSGI==2.0.19.1 uwsgi-tools==1.1.1 vine==5.0.0 voluptuous==0.12.0 wcwidth==0.2.5 xlrd==1.2.0 XlsxWriter==1.3.7 xlwt==1.3.0 ```

Other Dependencies

Minimally Reproducible Test Case

```python ```

Expected Behavior

Actual Behavior

When 4 to 8 workers start at the same time, the task will stop consuming. If there are 2 workers, the problem will not occur. Once the Visibility Timeout has passed, task digestion will resume. We use a SQS FIFO queue for brokers. We do not have this problem when using Redis.

[open-collective-bot[bot]]

Hey @marvelph :wave:, Thank you for opening an issue. We will get back to you as soon as we can. Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors. If you require immediate assistance please consider sponsoring us.

[auvipy]

@galCohen88 can you check

[GOGKI]

Was this task anyhow resolved?

[auvipy]

I don't know. can you help to reproduce?