Error during installing vendored packages

EmPiFree commented 4 years ago

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version? cf version 6.47.1+0921a21b7.2019-10-18

What version of the buildpack you are using? https://github.com/cloudfoundry/python-buildpack v1.7.1

If you were attempting to accomplish a task, what was it you were attempting to do? Installling vendored dependencies

What did you expect to happen? Successfully install all vendored dependencies

What was the actual behavior? Erorr during installing vendored dependencies:

-----> Python Buildpack version 1.7.1
   -----> Supplying Python
   -----> Installing python 3.8.0
          Copy [/tmp/cache/final/dependencies/cabbf104215db0e7ded4b2d65b3de8abfed4bb146a4cd61eacda7dbcc561925a/python-3.8.0-linux-x64-cflinuxfs3-6af4e503.tgz]
   -----> Installing pip-pop 0.1.3
          Copy [/tmp/cache/final/dependencies/3b86fb2cd9c13268f8858fd0ccb2f9eaf11295a51804b7075fe38d7e2a471191/pip-pop-0.1.4-0a3b0f1b.tar.gz]
          Generating 'requirements.txt' from Pipfile.lock
   -----> Running Pip Install
          Using the pip --no-build-isolation flag since it is available
          Looking in links: file:///tmp/app/vendor
          Collecting attrs==19.3.0 (from -r /tmp/app/requirements.txt (line 2))
              Preparing wheel metadata: started
              Preparing wheel metadata: finished with status 'error'
              ERROR: Command errored out with exit status 1:
               command: /tmp/contents687858878/deps/0/bin/python /tmp/contents687858878/deps/0/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_h2bnk_n
                   cwd: /tmp/pip-install-d90u0avf/attrs
              Complete output (14 lines):
              running dist_info
              creating /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info
              writing /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/PKG-INFO
              writing dependency_links to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/dependency_links.txt
              writing requirements to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/requires.txt
              writing top-level names to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/top_level.txt
              writing manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              reading manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              reading manifest template 'MANIFEST.in'
              no previously-included directories found matching 'docs/_build'
              warning: no previously-included files found matching 'changelog.d/*.rst'
              writing manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              creating '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.dist-info'
              error: invalid command 'bdist_wheel'
              ----------------------------------------
          ERROR: Command errored out with exit status 1: /tmp/contents687858878/deps/0/bin/python /tmp/contents687858878/deps/0/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_h2bnk_n Check the logs for full command output.
          Running pip install without indexes failed. Not all dependencies were vendored. Trying again with indexes.
Collecting attrs==19.3.0 (from -r /tmp/app/requirements.txt (line 9))
            Downloading https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl

So the build first tries to install from file:///tmp/app/vendor after one dependency fails, it reads the requirements.txt and download the package, which then works correctly...

I think it's an error according the pip version:

WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
          You should consider upgrading via the 'pip install --upgrade pip' command.

On my local machine with pip version 19.3.1 it works, when I run pip download -r requirements.txt --no-binary=:all: -d vendor. However it failed when I downgrade to the same version as in the buildpack (19.2.3). Even after I updated wheel to 0.33.6 and setuptoolsto 41.6.0 (was 41.2.0 before).

So the error could be solved if we try to upgrade pipto the newest version. However I don't know if the process with pip download -r requirements.txt --no-binary=:all: -d vendor is the same as the process the python buildpack does, when installing from vendor directory

Please confirm where necessary:

[x] I have included a log output
[x] My log includes an error message
[ ] I have included steps for reproduction

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/169730915

The labels on this github issue will be updated when the story is started.

ryanmoran commented 4 years ago

Hi @EmPiFree. We've spent a bit of time today reproducing this and we think there are a couple of things that might be going wrong.

We are able to reproduce this issue if we include a requirements.txt file that includes wheel and then also use the --no-binary=:all: option when running pip install to vendor our dependencies. We believe that the issue is that wheel itself provides a command to setup_tools called bdist_wheel that is not included in the "binary-less" vendored copy of wheel. This means that when pip goes to install attrs in your application, it finds no bdist_wheel binary on the system and fails. The subsequent installation process falls back to ignoring the vendor directory and re-downloading all of your dependencies which ultimately works, but is not ideal.

We don't think you should be including wheel in your requirements.txt, but we also aren't sure what you have in your application source code. Could you please provide us with a Pipfile, Pipfile.lock, and requirements.txt for your app so that we can do some more specific reproductions?

Cheers!

ForestEckhardt commented 4 years ago

We're gonna close this as inactive. Please feel free to re-open if you have further issues.

Bijaya196 commented 3 years ago

requests 2.21.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.10 which is incompatible. django-image-cropping 1.1.0 has requirement django-appconf==1.0.1, but you'll have django-appconf 1.0.4 which is incompatible. Installing collected packages: appdirs, asn1crypto, soupsieve, beautifulsoup4, certifi, pycparser, cffi, chardet, urllib3, idna, requests, uritemplate, itypes, MarkupSafe, Jinja2, coreschema, coreapi, six, cryptography, defusedxml, distlib, dj-database-url, pytz, Django, django-admin-rangefilter, django-admin-sortable2, django-appconf, django-autocomplete-light, django-autoslug, django-ipware, django-axes, django-cors-headers, django-dynamic-raw-id, elasticsearch, python-dateutil, elasticsearch-dsl, django-elasticsearch-dsl, django-nine, djangorestframework, django-elasticsearch-dsl-drf, typing, django-extensions, django-filter, URLObject, django-spurl, django-folders, django-grappelli, django-hotspots, django-image-cropping, django-mptt, django-multiupload, django-parler, django-parler-rest, django-tinymce, python-magic, django-upload-validator, django-waffle, waitress, WebOb, WebTest, django-webtest, lxml, docx-mailmerge, olefile, Pillow, easy-thumbnails, future, eight, et-xmlfile, text-unidecode, Faker, factory-boy, feincms-cleanse, filelock, maxminddb, geoip2, gunicorn, hsaudiotag3k, webencodings, setuptools, html5lib, http-ece, httplib2, typing-extensions, zipp, importlib-metadata, importlib-resources, jdcal, oauthlib, requests-oauthlib, lti, mimesis, openpyxl, polib, psycopg2, py-vapid, PyJWT, pyOpenSSL, PyPDF2, pywebpush, radar, raven, reportlab, rules, signxml, virtualenv, whitenoise, xhtml2pdf, xmltodict, pyenchant, more-itertools, atomicwrites, pluggy, attrs, py, pytest, pytest-django, inflection, pytest-factoryboy, pytest-mock, pyparsing, packaging, termcolor, pytest-sugar, freezegun The script chardetect is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for itypes: started Running setup.py install for itypes: finished with status 'done' Running setup.py install for coreschema: finished with status 'done' The script django-admin is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for django-admin-sortable2: started Running setup.py install for django-admin-sortable2: finished with status 'done' Running setup.py install for django-autocomplete-light: started Running setup.py install for django-autocomplete-light: finished with status 'done' Running setup.py install for django-ipware: started Running setup.py install for django-ipware: finished with status 'done' Running setup.py install for URLObject: started Running setup.py install for URLObject: finished with status 'done' Running setup.py develop for django-folders Running setup.py install for django-grappelli: started Running setup.py install for django-grappelli: finished with status 'done' Running setup.py develop for django-hotspots Running setup.py install for django-image-cropping: started Running setup.py install for django-image-cropping: finished with status 'done' Running setup.py install for django-multiupload: started Running setup.py install for django-multiupload: finished with status 'done' Running setup.py install for django-upload-validator: started Running setup.py install for django-upload-validator: finished with status 'done' The script waitress-serve is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for olefile: started Running setup.py install for olefile: finished with status 'done' Running setup.py install for easy-thumbnails: started Running setup.py install for easy-thumbnails: finished with status 'done' Running setup.py install for future: started Running setup.py install for future: finished with status 'done' Running setup.py install for et-xmlfile: started Running setup.py install for et-xmlfile: finished with status 'done' The script faker is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for feincms-cleanse: started Running setup.py install for feincms-cleanse: finished with status 'done' Running setup.py install for maxminddb: started Running setup.py install for maxminddb: finished with status 'done' The scripts gunicorn and gunicorn_paster are installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for hsaudiotag3k: started Running setup.py install for hsaudiotag3k: finished with status 'done' Running setup.py install for http-ece: started Running setup.py install for http-ece: finished with status 'done' Running setup.py install for mimesis: started Running setup.py install for mimesis: finished with status 'done' Running setup.py install for openpyxl: started Running setup.py install for openpyxl: finished with status 'done' Running setup.py install for py-vapid: started The script pyjwt is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for PyPDF2: started Running setup.py install for PyPDF2: finished with status 'done' Running setup.py install for pywebpush: started Running setup.py install for pywebpush: finished with status 'done' Running setup.py install for radar: started Running setup.py install for radar: finished with status 'done' The script raven is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for rules: started Running setup.py install for rules: finished with status 'done' Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for xhtml2pdf: started Running setup.py install for xhtml2pdf: finished with status 'done' The scripts py.test and pytest are installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Running setup.py install for pytest-factoryboy: started Running setup.py install for pytest-factoryboy: finished with status 'done' Running setup.py install for termcolor: started Running setup.py install for termcolor: finished with status 'done' Successfully installed Django-1.11.29 Faker-5.0.2 Jinja2-2.11.2 MarkupSafe-1.1.1 Pillow-4.3.0 PyJWT-1.7.1 PyPDF2-1.26.0 URLObject-2.4.3 WebOb-1.8.6 WebTest-2.0.35 appdirs-1.4.4 asn1crypto-1.4.0 atomicwrites-1.4.0 attrs-20.3.0 beautifulsoup4-4.9.3 certifi-2017.11.5 cffi-1.14.4 chardet-3.0.4 coreapi-2.3.3 coreschema-0.0.4 cryptography-2.7 defusedxml-0.6.0 distlib-0.3.1 dj-database-url-0.5.0 django-admin-rangefilter-0.3.6 django-admin-sortable2-0.6.21 django-appconf-1.0.4 django-autocomplete-light-3.2.10 django-autoslug-1.9.8 django-axes-4.5.4 django-cors-headers-2.1.0 django-dynamic-raw-id-2.4 django-elasticsearch-dsl-0.4.5 django-elasticsearch-dsl-drf-0.7.2 django-extensions-1.9.9 django-filter-1.1.0 django-folders django-grappelli-2.10.4 django-hotspots django-image-cropping-1.1.0 django-ipware-2.1.0 django-mptt-0.8.7 django-multiupload-0.5.2 django-nine-0.1.13 django-parler-1.9.2 django-parler-rest-2.0 django-spurl-0.6.7 django-tinymce-2.7.1 django-upload-validator-1.0.2 django-waffle-0.14.0 django-webtest-1.9.7 djangorestframework-3.7.7 docx-mailmerge-0.4.0 easy-thumbnails-2.7.1 eight-0.4.2 elasticsearch-5.5.3 elasticsearch-dsl-5.4.0 et-xmlfile-1.0.1 factory-boy-3.2.0 feincms-cleanse-8 filelock-3.0.12 freezegun-0.3.11 future-0.16.0 geoip2-2.9.0 gunicorn-19.9.0 hsaudiotag3k-1.1.3.post1 html5lib-1.0b10 http-ece-1.1.0 httplib2-0.18.1 idna-2.10 importlib-metadata-3.3.0 importlib-resources-3.3.1 inflection-0.5.1 itypes-1.1.0 jdcal-1.4.1 lti-0.9.5 lxml-4.1.1 maxminddb-2.0.3 mimesis-0.0.10 more-itertools-8.6.0 oauthlib-3.1.0 olefile-0.46 openpyxl-2.5.14 packaging-20.8 pluggy-0.13.1 polib-1.0.8 psycopg2-2.7.3.2 py-1.10.0 py-vapid-1.7.0 pyOpenSSL-17.5.0 pycparser-2.20 pyenchant-3.1.1 pyparsing-2.4.7 pytest-4.0.0 pytest-django-3.4.4 pytest-factoryboy-2.0.2 pytest-mock-1.10.0 pytest-sugar-0.9.2 python-dateutil-2.6.1 python-magic-0.4.20 pytz-2017.3 pywebpush-1.10.0 radar-0.3 raven-6.6.0 reportlab-3.5.57 requests-2.21.0 requests-oauthlib-1.3.0 rules-1.3 setuptools-52.0.0 signxml-2.5.2 six-1.11.0 soupsieve-2.1 termcolor-1.1.0 text-unidecode-1.3 typing-3.6.6 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.21.1 virtualenv-20.2.2 waitress-1.4.4 webencodings-0.5.1 whitenoise-5.1.0 xhtml2pdf-0.2b1 xmltodict-0.11.0 zipp-3.4.0 You are using pip version 18.1, however version 21.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Running python /tmp/app/manage.py collectstatic --noinput --traceback Error staging application global-trainingByBijay: timed out after 15 minute(s) FAILED

Bijaya196 commented 3 years ago

could you please any one help me on this

ryanmoran commented 3 years ago

There's not a lot in these logs to indicate what might be going on here. I can see that the build timed out after 15 minutes, which is rather surprising for a Python app. Can you provide the entirety of the log output so that we might be able to have a better idea of what's happening? It would also help if you could include an example that reproduces this error.

cloudfoundry / python-buildpack

Error during installing vendored packages #152