search

celery / celery

Distributed Task Queue (development branch)
https://docs.celeryq.dev
Other
24.66k stars 4.65k forks source link

Dockerfile - Build failed with kombu<5.0,>=4.6.4 installation #5763

Closed naggappan closed 4 years ago

naggappan commented 5 years ago

I am trying to build the celery docker container as mentioned in the docs to setup the dev environment. But docker build fails with below error,

Command: docker-compose build celery docker_Version: Docker version 19.03.2, build 6a30dfc docker_Compse_version: docker-compose version 1.18.0, build 8dd22a9

Error:

    Running setup.py install for unipath: started
    Running setup.py install for unipath: finished with status 'done'
Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 PyYAML-5.1.2 Pygments-2.4.2 Sphinx-1.8.5 alabaster-0.7.12 amqp-2.5.1 argparse-1.4.0 asn1crypto-1.0.1 aspy.yaml-1.3.0 atomicwrites-1.3.0 attrs-19.2.0 aws-xray-sdk-0.95 azure-common-1.1.5 azure-nspkg-3.0.2 azure-storage-0.36.0 azure-storage-common-1.1.0 azure-storage-nspkg-3.1.0 babel-2.7.0 basho-erlastic-2.1.1 billiard-3.6.1.0 boto-2.49.0 boto3-1.9.243 botocore-1.12.243 bumpversion-0.5.3 case-1.5.3 cassandra-driver-3.19.0 certifi-2019.9.11 cffi-1.12.3 cfgv-2.0.1 chardet-3.0.4 codecov-2.0.15 couchbase-2.5.8 coverage-4.5.4 cryptography-2.7 datetime-4.3 dnspython-1.16.0 docker-4.1.0 docutils-0.15.2 ecdsa-0.13.2 elasticsearch-7.0.5 entrypoints-0.3 ephem-3.7.7.0 eventlet-0.25.1 filelock-3.0.12 flake8-3.7.8 flakeplus-1.1.0 future-0.17.1 gevent-1.4.0 greenlet-0.4.15 identify-1.4.7 idna-2.8 imagesize-1.1.0 importlib-metadata-0.23 importlib-resources-1.0.2 jmespath-0.9.4 jsondiff-1.1.1 jsonpickle-1.2 kombu-4.6.5 linecache2-1.0.0 mccabe-0.6.1 mock-3.0.5 monotonic-1.5 more-itertools-7.2.0 moto-1.3.7 msgpack-0.6.2 nodeenv-1.3.3 nose-1.3.7 packaging-19.2 pathlib2-2.3.5 pluggy-0.13.0 pre-commit-1.18.3 py-1.8.0 pyArango-1.3.2 pyaml-19.4.1 pycodestyle-2.5.0 pycouchdb-1.14 pycparser-2.19 pycryptodome-3.9.0 pycurl-7.43.0.3 pydocstyle-1.1.1 pydocumentdb-2.3.2 pyflakes-2.1.1 pylibmc-1.6.1 pymongo-3.9.0 pyparsing-2.4.2 pytest-4.6.5 pytest-cov-2.8.0 pytest-sugar-0.9.2 pytest-travis-fold-1.3.0 python-consul-1.1.0 python-dateutil-2.8.0 python-jose-2.0.2 python-memcached-1.59 python3-protobuf-2.5.0 pytz-2019.2 redis-3.3.8 requests-2.22.0 responses-0.10.6 riak-2.7.0 s3transfer-0.2.1 simplejson-3.16.0 six-1.12.0 snowballstemmer-2.0.0 softlayer-messaging-1.0.3 sphinx-celery-1.4.6 sphinx-testing-0.7.2 sphinx2rst-1.1.0 sphinxcontrib-websupport-1.1.2 sqlalchemy-1.3.9 tblib-1.4.0 termcolor-1.1.0 toml-0.10.0 tox-3.14.0 traceback2-1.4.0 typing-3.7.4.1 unipath-1.1 unittest2-1.1.0 urllib3-1.25.6 vine-1.3.0 virtualenv-16.7.5 wcwidth-0.1.7 websocket-client-0.56.0 werkzeug-0.16.0 wheel-0.33.6 wrapt-1.11.2 xmltodict-0.12.0 zipp-0.6.0 zope.interface-4.6.0
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Ignoring couchbase-cffi: markers 'platform_python_implementation == "PyPy"' don't match your environment
Collecting pytz>dev (from -r requirements/default.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/87/76/46d697698a143e05f77bec5a526bf4e56a0be61d63425b68f4ba553b51f2/pytz-2019.2-py2.py3-none-any.whl
Collecting billiard<4.0,>=3.6.1 (from -r requirements/default.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/9a/bb/2a016ac912fca48e06ff5a662407f3d1681aa47df97fb19feba7cc931ee1/billiard-3.6.1.0-py3-none-any.whl
Collecting kombu<5.0,>=4.6.4 (from -r requirements/default.txt (line 3))
  ERROR: Could not find a version that satisfies the requirement kombu<5.0,>=4.6.4 (from -r requirements/default.txt (line 3)) (from versions: 0.1.0, 0.2.0, 0.9.0, 0.9.1, 0.9.2, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0b4, 1.0.0rc1, 1.0.0rc2, 1.0.0rc3, 1.0.0rc4, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.5.0, 1.5.1, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.2.6, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.5.9, 2.5.10, 2.5.11, 2.5.12, 2.5.13, 2.5.14, 2.5.15, 2.5.16, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.0.8, 3.0.9, 3.0.10, 3.0.11, 3.0.12, 3.0.13, 3.0.14, 3.0.15, 3.0.16, 3.0.17, 3.0.18, 3.0.19, 3.0.20, 3.0.21, 3.0.22, 3.0.23, 3.0.24, 3.0.25, 3.0.26, 3.0.27, 3.0.28, 3.0.29, 3.0.30, 3.0.31, 3.0.32, 3.0.33, 3.0.34, 3.0.35, 3.0.36, 3.0.37, 4.0.0rc3, 4.0.0rc4, 4.0.0rc5, 4.0.0rc6, 4.0.0, 4.0.1, 4.0.2, 4.1.0, 4.2.0, 4.2.1, 4.2.2.post1, 4.3.0, 4.4.0, 4.5.0)
ERROR: No matching distribution found for kombu<5.0,>=4.6.4 (from -r requirements/default.txt (line 3))
WARNING: You are using pip version 19.1.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR: Service 'celery' failed to build: The command '/bin/sh -c pyenv exec python3.7 -m pip install   -r requirements/default.txt   -r requirements/test.txt   -r requirements/test-ci-default.txt   -r requirements/docs.txt   -r requirements/test-integration.txt   -r requirements/pkgutils.txt &&   pyenv exec python3.6 -m pip install   -r requirements/default.txt   -r requirements/test.txt   -r requirements/test-ci-default.txt   -r requirements/docs.txt   -r requirements/test-integration.txt   -r requirements/pkgutils.txt &&   pyenv exec python3.5 -m pip install   -r requirements/default.txt   -r requirements/test.txt   -r requirements/test-ci-default.txt   -r requirements/docs.txt   -r requirements/test-integration.txt   -r requirements/pkgutils.txt &&   pyenv exec python3.4 -m pip install   -r requirements/default.txt   -r requirements/test.txt   -r requirements/test-ci-default.txt   -r requirements/docs.txt   -r requirements/test-integration.txt   -r requirements/pkgutils.txt &&   pyenv exec python2.7 -m pip install   -r requirements/default.txt   -r requirements/test.txt   -r requirements/test-ci-default.txt   -r requirements/docs.txt   -r requirements/test-integration.txt   -r requirements/pkgutils.txt' returned a non-zero code: 1

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Celery version:

celery report Output:

``` ```

Steps to Reproduce

Required Dependencies

Python Packages

pip freeze Output:

``` ```

Other Dependencies

N/A

Minimally Reproducible Test Case

```python ```

Expected Behavior

Actual Behavior

thedrow commented 5 years ago

@auvipy I've asked you multiple times now to stop closing issues without comment. Please stop.

Why did you close this issue?

naggappan commented 5 years ago

@thedrow @auvipy are you able to setup the docker based development environment? is docker build successfully for you? Very basic checkout the master code and build is failing with above error.

dipAch commented 4 years ago

@thedrow Any update on this? Facing the same issue here when building via docker.

image

dipAch commented 4 years ago

Hi! As this has been open for quite some time now, I tried debugging myself.

I could see that for Python-3.4, we don't have version supporting kombu > 4.5.0. As a result, i went ahead and locally changed the kombu version requirement in requirements/default.txt:

From:

To:

With the above change, the issue (docker-compose build celery) seems to be fixed. All other versions of python (2.7, 3.5, 3.6, 3.7) seemed to install kombu==4.6.5, and only Python-3.4 installed kombu==4.5.0. I tried running the unit & integration tests with Pyenv set to Python-3.4 and could see that the unit tests were passing except for one failure. On the other hand, running integration tests resulted in a bunch of fixture related setup error for celery_session_app & celery_parameters (didn't have the time to debug, yet!).

Please find the images / error_file below for the test summary (with Python-3.4), for your ready reference:

Screen Shot 2019-10-29 at 4 12 03 PM

Screen Shot 2019-10-29 at 3 25 12 PM

integration_tests_fixture_setup_error.txt

I am new to Celery Development, so not sure if downgrading the kombu version would have any effect on the python-3.4 installation for Celery development.

Please let me know your thoughts on this, @thedrow @auvipy?

Environment Information:

Docker Engine - 19.03.4
Docker Compose - 1.24.1
MacOS High Sierra - 10.13.5
Container Platform - Linux
Python Version - 3.4.9
Pytest - 4.6.6
Pytest-sugar - 0.9.2
auvipy commented 4 years ago

PR in progress https://github.com/celery/celery/pull/5801

auvipy commented 4 years ago

technically, dev environment should be based on master

dipAch commented 4 years ago

@auvipy Got it. Thanks!

auvipy commented 4 years ago

can you pull the docker and report?

dipAch commented 4 years ago

@auvipy Doing that now, will let you know.

auvipy commented 4 years ago

test failure is normal as master at CI is broken. just check if docker is working properly. if not, show the log here

dipAch commented 4 years ago

@auvipy Getting below error on executing docker-compose build celery:

Building celery
Step 1/26 : FROM debian:buster
 ---> 8e9f8546050d
Step 2/26 : ENV PYTHONIOENCODING UTF-8
 ---> Using cache
 ---> d203357ee575
Step 3/26 : RUN apt update && apt install -y build-essential     libcurl4-openssl-dev     libffi-dev     tk-dev     xz-utils     curl     lsb-release     git     libmemcached-dev     make     liblzma-dev     libreadline-dev     libbz2-dev     llvm     libncurses5-dev     libsqlite3-dev     wget     pypy     python-openssl     libncursesw5-dev     zlib1g-dev     pkg-config     libssl1.0-dev
 ---> Running in 997e518d0344

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://security-cdn.debian.org/debian-security buster/updates InRelease [39.1 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian buster InRelease [122 kB]
Get:4 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [108 kB]
Get:3 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:5 http://cdn-fastly.deb.debian.org/debian buster/main amd64 Packages [7899 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
Fetched 8222 kB in 14s (580 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
Package libssl1.0-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libssl1.0-dev' has no installation candidate
ERROR: Service 'celery' failed to build: The command '/bin/sh -c apt update && apt install -y build-essential     libcurl4-openssl-dev     libffi-dev     tk-dev     xz-utils     curl     lsb-release     git     libmemcached-dev     make     liblzma-dev     libreadline-dev     libbz2-dev     llvm     libncurses5-dev     libsqlite3-dev     wget     pypy     python-openssl     libncursesw5-dev     zlib1g-dev     pkg-config     libssl1.0-dev' returned a non-zero code: 100
auvipy commented 4 years ago

can you try again? I will check it manually tomorrow morning again

dipAch commented 4 years ago

Crashes at below step in Dockerfile:

Step 11/26 : RUN sh install-couchbase.sh

 ---> Running in a29a6ca1238a
--2019-10-30 18:22:03--  http://packages.couchbase.com/clients/c/libcouchbase-2.10.4_buster_amd64.tar
Resolving packages.couchbase.com (packages.couchbase.com)... 54.230.195.161, 54.230.195.18, 54.230.195.155, ...
Connecting to packages.couchbase.com (packages.couchbase.com)|54.230.195.161|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-30 18:22:04 ERROR 404: Not Found.

tar: libcouchbase-2.10.4_buster_amd64.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
dpkg: error: cannot access archive 'libcouchbase-2.10.4_buster_amd64/libcouchbase2-core_2.10.3-1_amd64.deb': No such file or directory
dpkg: error: cannot access archive 'libcouchbase-2.10.4_buster_amd64/libcouchbase-dev_2.10.3-1_amd64.deb': No such file or directory
ERROR: Service 'celery' failed to build: The command '/bin/sh -c sh install-couchbase.sh' returned a non-zero code: 2
dipAch commented 4 years ago

@auvipy I don't see any libcouchbase-* package (for version 2.10.4) listed for debian:buster on the site: https://docs.couchbase.com/c-sdk/2.10/sdk-release-notes.html#latest-release

Could that be why it's failing with above? I see 404 in the response for the wget request.

$ wget -P ~/Desktop/ http://packages.couchbase.com/clients/c/libcouchbase-2.10.4_buster_amd64.tar                 9.4m  Thu Oct 31 12:18:37 2019
--2019-10-31 12:19:16--  http://packages.couchbase.com/clients/c/libcouchbase-2.10.4_buster_amd64.tar
Resolving packages.couchbase.com (packages.couchbase.com)... 54.230.195.155, 54.230.195.161, 54.230.195.18, ...
Connecting to packages.couchbase.com (packages.couchbase.com)|54.230.195.155|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-31 12:19:17 ERROR 404: Not Found.
auvipy commented 4 years ago

hi, can you pull again and build?

dipAch commented 4 years ago

Hi, doing that now. Will let you know.

dipAch commented 4 years ago

Hi @auvipy , sorry for the late response, was AFK.

I was able to build the docker image successfully and spin up the container. But, had to make a few changes to Dockerfile.

One change was to set the build argument to: ARG DEBIAN_FRONTEND=noninteractive With this, I was able to disable the prompt for tzdata, during the build. With this, the timezone will be by default set to UTC however, it can be changed/reconfigured later on to reflect appropriate timezone.

The second change was to add sudo to the apt-get install ... command, as sudo is not included by default. I believe we are using sudo in the script install-couchase.sh. Ref - https://github.com/tianon/docker-brew-ubuntu-core/issues/48

I have raised a PR for this, with the above changes included. Kindly review.

auvipy commented 4 years ago

thanks!

naggappan commented 4 years ago

It works thanks

auvipy commented 4 years ago

thanks for trying it!