search

what-digital / divio

A project for tracking divio.com deployment issues
0 stars 0 forks source link

missing pypi packages in Divio wheels proxy mirror #1

Open viktor-yunenko opened 4 years ago

viktor-yunenko commented 4 years ago

pip-tools wheels repository

Resources

Description

The caching can be out of date for 1 week, but eg boto package is 2 years behind pypi.org.

I think the 500 server error during deployment (related to pip-reqs) used to disappear after 1-3 days, but now it doesn't.

In other cases a package building error might be raised only inside docker with custom wheels (django-spurl 0.6.6), or a package version added by pip-reqs won't exist on pypi.org (django-standard-form 1.1.3). This is the case for djangocms-socialshare 1.0.0.0, only an older version is accessible.

Initial debugging log

I've been having an odd issue with deployments probably during the last 2 weeks:

Step 3/7 : RUN pip-reqs resolve && pip install --no-index --no-deps --requirement requirements.urls
---> Running in 582790fc7a53
Traceback (most recent call last):
File "/usr/local/bin/pip-reqs", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/pip_reqs/__main__.py", line 46, in main
args.func(client, args)
File "/usr/local/lib/python3.6/site-packages/pip_reqs/__main__.py", line 76, in resolve
urls = client.resolve(fh.read())
File "/usr/local/lib/python3.6/site-packages/pip_reqs/client.py", line 25, in resolve
r.raise_for_status()
File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://wheels.aldryn.net/v1/aldryn-extras+pypi/aldryn-baseproject-v4-py36/+resolve/

For example the following project had no changes in its dockerfile, but the deployment stopped working this morning (working at midnight) - https://control.divio.com/control/6065/edit/78738/

My dockerfiles can be seen below:

Source of the problem

I just tracked down the problem to a pypi package - https://pypi.org/project/djangocms-aldryn-forms-bootstrap4-templates/#history

The projects that are working use 1.0.0.1, but I need 1.0.0.2. It appears that divio is using a separate pypi repository that has 1.0.0.1, but not the latest version?

Few hours ago we had another problem with pip-reqs on another project, it started failing because one of the package - poetry - didn't work when installed through pip-reqs, standard pip install is working within the docker, we added RUN pip install poetry to Dockerfile.

In the meantime I tried to remove pip-reqs in my base image - https://gitlab.com/what-digital/djangocms-template/-/blob/divio/base.Dockerfile

Now all the package are installed normally. Docker build runs without issues locally. But divio deployment fails without an error message. Are there any ways of removing pip-reqs? I've been resolving issues with it before, would be very nice to utilize pip-tools instead.

Attaching the last deployment logs below:

===== docker migrate release =====
+ start migrate
aldryn-django: running migration commands
----> CACHE_URL="locmem://" python manage.py createcachetable django_dbcache; exit 0
Cache table 'django_dbcache' already exists.
----> python manage.py migrate --noinput
Operations to perform:
Apply all migrations: account, admin, aldryn_forms, aldryn_sso, auth, backend_auth, bootstrap4_alerts, bootstrap4_badge, bootstrap4_card, bootstrap4_carousel, bootstrap4_collapse, bootstrap4_content, bootstrap4_grid, bootstrap4_jumbotron, bootstrap4_link, bootstrap4_listgroup, bootstrap4_media, bootstrap4_picture, bootstrap4_tabs, bootstrap4_utilities, bs4_hiding, bs4_spacer, captcha, cms, contenttypes, cuser, djangocms_blog, djangocms_googlemap, djangocms_history, djangocms_icon, djangocms_link, djangocms_modules, djangocms_page_meta, djangocms_picture, djangocms_redirect, djangocms_snippet, djangocms_text_ckeditor, djangocms_video, easy_thumbnails, email_notifications, filer, heading_element, horizontal_line, light_gallery, menus, robots, section_element, section_with_image_background, sessions, sites, socialaccount, taggit
Running migrations:
No migrations to apply.
----> python manage.py aldryn_update_s3_media_headers
0/115 files updated
----> python manage.py cms fix-tree
fixing page tree
fixing plugin tree
all done

This also happens when the ENV commands in docker are placed after pip install. Again it builds fine locally.

boto python package

On another project boto 2.49 can't be found during divio deployment, while it was pushed to pypi.org back in 2018. The deployment error:

  Could not find a version that satisfies the requirement boto==2.49.0 (from -r requirements.txt (line 33)) (from versions: 0.8d, 0.9d, 1.5d, 1.8d, 0.2a0, 0.3a0, 0.4a0, 0.4b0, 0.5a0, 0.6a0, 0.6b0, 0.6rc0, 0.7a0, 0.8a0, 0.8b0, 0.8rc0, 0.9a0, 0.9b0, 0.9rc0, 1.0a0, 1.1a0, 1.1b0, 1.1rc0, 1.2a0, 1.3a0, 1.4a0, 1.4b0, 1.4rc0, 1.5rc0, 1.6a0, 1.6b0, 1.7a0, 1.8a0, 1.8b0, 1.8rc0, 1.9a0, 1.9b0, 2.0a1, 2.0a2, 2.0b1, 2.0b2, 2.0b3, 2.0b4, 2.0rc1, 2.0, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.7.0, 2.8.0, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.9.6, 2.9.7, 2.9.8, 2.9.9, 2.10.0, 2.11.0, 2.12.0, 2.13.0, 2.13.3, 2.14.0, 2.15.0, 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.20.0, 2.20.1, 2.21.0, 2.21.1, 2.21.2, 2.22.0, 2.22.1, 2.23.0, 2.24.0, 2.25.0, 2.26.0, 2.26.1, 2.27.0, 2.28.0, 2.29.0, 2.29.1, 2.30.0, 2.31.0, 2.31.1, 2.32.0, 2.32.1, 2.33.0, 2.34.0, 2.35.0, 2.35.1, 2.35.2, 2.36.0, 2.37.0, 2.38.0, 2.38.0.1, 2.38.1a1, 2.39.0, 2.40.0, 2.41.0, 2.42.0, 2.43.0, 2.44.0, 2.45.0, 2.46.0, 2.46.1, 2.47.0, 2.48.0)
No matching distribution found for boto==2.49.0 (from -r requirements.txt (line 33))
You are using pip version 19.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR: Service 'base' failed to build: The command '/bin/sh -c pip install         --no-deps         --requirement requirements.txt' returned a non-zero code: 1

django-spurl

Version 0.6.6 isn't installable by divio pip-reqs. Version 0.6.5 works.

viktor-yunenko commented 4 years ago

The issue remains and the situations worsens - I'm finding more packages that aren't installable from pypi every other week.

macolo commented 4 years ago

@GaretJax could you give us a helping hand? We dont know whats hitting us here.

macolo commented 4 years ago

Can we do something to invalidate the cache or add a specific version or package to the cache? Or is the problem completely elsewhere?

macolo commented 4 years ago

if we just use pip-tools in the Dockerfile: HTTPError: 500 Server Error on deployment

GaretJax commented 4 years ago

We need the requirements.txt and requirements.in files to help you out.

GaretJax commented 4 years ago

If you want to stop using the wheelsproxy, it's enough to:

  1. Remove the pip-reqs command from your Dockerfile
  2. Remove and INDEX_URLs env vars you may still have in your dockerfile.

Beside "caching", using the wheelsproxy has a lot of advantages:

  1. Consistent dependency resolution (pip install requirements.in could lead to the installation of packages with conflicting dependencies)
  2. We are able to layer in private indexes while preserving fast lookups (this is part of the issue you have with packages being present which are not on PyPI.
  3. Precompiled wheels for all packages compatible with the docker image being used. This allows you to get rid completely of all dev tools from your base image, such as compilers, dev libraries, headers,...
  4. Obviously, faster deployments...

If there are other packages failing to install from the wheelsproxy, please let me know. To help you out quickly, we need to know the package/version and in case of 500s, also a copy of the requirements.in/requirements.txt files.

In general, the wheelsproxy is a pypi-compatible index and you can browse it yourself at: https://wheels.aldryn.net/v1/INDEXES/IMAGE/+simple/PACKAGE/ where INDEXES is a + separated list of indexes to use (by default aldryn-extras+pypi), IMAGE is the base image in use (e.g. aldryn-baseproject-v4-py36) and PACKAGE is the package name. For example: https://wheels.aldryn.net/v1/aldryn-extras+pypi/aldryn-baseproject-v4-py36/+simple/django/

GaretJax commented 4 years ago

Oh, just to add, usage of the wheelsproxy should be completely optional and if something is not working without it, we shall fix it. ;-)

macolo commented 4 years ago

Thank you @GaretJax for your quick reply, it's much appreciated!

I also had such a problem with a fresh version I released to pypi, it was subsequently not available within 1-2h so I switched to using the gitlab repo URL directly. Is there a delay and if yes what would be a normal delay in hours or days?

@victor-yunenko can you please list any other packages here that cause problems in case there are any remaining?

GaretJax commented 4 years ago

There can be a delay, but it should be in the order of minutes (with the normal case being < 1 minute).

There is from time to time a longer delay caused by PyPI's CDN not picking up new packages quickly enough, but we should have workarounds for that as well.

Feel free to ping us in such cases.

GaretJax commented 4 years ago

Another thing @macolo, we always pick the source distribution (.tar.gz) even for packages with wheels available. There are some packages which don't publish any source distribution at all (only wheels) and these result in the releases not being available via the wheelsproxy.

viktor-yunenko commented 4 years ago

Hi @GaretJax, thank you for following up on this.

It probably became unclear now, but initially this ticket was raised as a summary of my conversations with Daniele.

If you want to stop using the wheelsproxy, it's enough to:

Remove the pip-reqs command from your Dockerfile Remove and INDEX_URLs env vars you may still have in your dockerfile.

Basically that's what we tried with Daniele back then, in various combinations, few notes that I left:

In the meantime I tried to remove pip-reqs in my base image - https://gitlab.com/what-digital/djangocms-template/-/blob/divio/base.Dockerfile

Now all the package are installed normally. Docker build runs without issues locally. But divio deployment fails without an error message. Are there any ways of removing pip-reqs? I've been resolving issues with it before, would be very nice to utilize pip-tools instead.

Attaching the last deployment logs below:

===== docker migrate release =====
[...]
----> python manage.py cms fix-tree
fixing page tree
fixing plugin tree
all done

This also happens when the ENV commands in docker are placed after pip install. Again it builds fine locally.

As an experiment I tried to do that again on another project, but I'm seeing the similar outcome, where the deployment fails without an error after:

fixing page tree
fixing plugin tree
all done

Perhaps there's something I can do to avoid that problem?

viktor-yunenko commented 4 years ago

I would be happy to stay with pip-reqs, but over the last years the amount of problems with it has been rather problematic. Personally I can handle it well, but when I find out that a new to divio developer quitely spent a whole day - or sometimes several days - on trying to install few packages - the impact of those issues on the given project's schedule and budget becomes highly salient.

viktor-yunenko commented 4 years ago

Here are the files from my last attempt of disabling pip-reqs:

Perhaps there's an existing project that was able to disable them successfully?

GaretJax commented 4 years ago

I will have to look into it, but my first guess would be to say that the version of uwsgi from PyPI is not built with support for some required lib. On the wheels proxy we build it with:

printf '[uwsgi]\nmain_plugin=python,gevent,emperor_pg\ninherit = base\nbin_name = uwsgi' >/tmp/profile.ini
export UWSGI_PROFILE=/tmp/profile.ini

pip-reqs is not doing anything at startup, as such if you try to run your images in live mode locally (or look at the runtime logs, not the deployment logs, when your deployment fails without message) you may be able to see the cause of the issue, as it seems that the server fails to start.

macolo commented 4 years ago

@GaretJax we have a couple of packages that are missing in the wheels proxy:

macolo commented 4 years ago

@GaretJax FYI this might be related to the following (from Dennis):

It looks like you are using a python 3.6 package registry on a new python 3.7 image. You have to use https://wheels.aldryn.net/v1/aldryn-extras+pypi/stretch-py37/+simple/ instead of https://wheels.aldryn.net/v1/aldryn-extras+pypi/aldryn-baseproject-v4-py36/+simple/ We are not pulling docker images in regular intervals so it is strongly advised to not pin the base image to latest in the Dockerfile and use a proper version (e.g. 1.1.2 )instead.

macolo commented 3 years ago

Note: about a month ago the error messaging in pip-reqs has been improved by @kinkerl - it should be much more speaking now when there are problems with packages in requirements.txt.