'apt-get install git-all' not making it through to 'pip install' of requirements

dantagg commented 2 years ago

What happened?

I updated local django Dockerfile python-build-stage to include

RUN apt-get update && apt-get install --no-install-recommends -y \
# dependencies for building Python packages
build-essential \
# including git \
git-all \
# psycopg2 dependencies
libpq-dev

and updated python-run-stage to include

RUN apt-get update && apt-get install --no-install-recommends -y \
# psycopg2 dependencies
libpq-dev \
# Translations dependencies
gettext \
# including git \
git-all \
# cleaning up unused files
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*

I did this because one of my pip requirements needs git to install, for example

git+https://github.com/mozilla/django-csp.git

It isn't this, but the one I'm installing is in a private repository.

When I run

docker compose -f local.yml build

it fails with the following error ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

Looking at the build log, there is an error

 => CACHED [django_fabman_local_docs python-build-stage 1/3] RUN apt-get update && apt-get install --no-install-recommends -y   build-essential   libpq-dev   && apt-  0.0s
 => CACHED [django_fabman_local_docs python-build-stage 2/3] COPY ./requirements /requirements                                                                         0.0s
 => ERROR [django_fabman_local_docs python-build-stage 3/3] RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels    -r /requirements/local.txt -r   16.1s

and then further down there is

=> CANCELED [django_fabman_local_django python-build-stage 1/3] RUN apt-get update && apt-get install --no-install-recommends -y   build-essential   git-all   libp  14.3s

This suggests the pip install is running ahead and being attempted before the apt-get installs have finished and the when pip fails, the APT-GET is cancelled because the whole build has borked.

What should've happened instead?

The build should have completed successfully

Additional details

Host system configuration:
- Version of cookiecutter CLI
Cookiecutter 1.7.3
- OS name and version:
ProductName: macOS ProductVersion: 11.6.4 BuildVersion: 20G417 Docker Compose version v2.3.3 Docker version 20.10.13, build a224086 Python on host: Python 3.10.0 (v3.10.0:b494f5935c Python on Docker: Python 3.9.12 (main, Mar 29 2022, 14:20:48) [GCC 10.2.1 20210110] on linux
- Python version, run python3 -V:
- Docker version (if using Docker), run docker --version:
- docker-compose version (if using Docker), run docker-compose --version:
- ...

Options selected and/or replay file: On Linux and MacOS: cat ${HOME}/.cookiecutter_replay/cookiecutter-django.json (Please, take care to remove sensitive information)

{
"cookiecutter": {
"project_name": "Django Fabman",
"project_slug": "django_fabman",
"description": "Service to support the various extensions of Fabman for KWMC Factory",
"author_name": "Daniel Tagg",
"domain_name": "kwmc.factory",
"email": "***@***.com",
"version": "0.1.0",
"open_source_license": "Not open source",
"timezone": "Europe/London",
"windows": "n",
"use_pycharm": "y",
"use_docker": "y",
"postgresql_version": "14.1",
"cloud_provider": "None",
"mail_service": "Other SMTP",
"use_async": "n",
"use_drf": "y",
"frontend_pipeline": "None",
"use_celery": "n",
"use_mailhog": "n",
"use_sentry": "n",
"use_whitenoise": "y",
"use_heroku": "n",
"ci_tool": "Gitlab",
"keep_local_envs_in_vcs": "n",
"debug": "n",
"_template": "gh:cookiecutter/cookiecutter-django"
}

xjlin0 commented 2 years ago

instead of git-all, how about just git? I also install git in my Dockerfile and it works

ARG PYTHON_VERSION=3.9-slim-bullseye

# define an alias for the specfic python version used in this file.
FROM python:${PYTHON_VERSION} as python

# Python build stage
FROM python as python-build-stage
RUN echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.list
ARG BUILD_ENVIRONMENT=local

# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
  # dependencies for building Python packages
  build-essential curl vim wget git    ####### HERE #######

and in requirements/base.py

...
celery==5.2.6  # pyup: < 6.0  # https://github.com/celery/celery
-e git+https://github.com/llazzaro/django-scheduler.git@develop#egg=django-scheduler  ### THIS ###
django-celery-beat==2.2.1  # https://github.com/celery/django-celery-beat
...

luzfcb commented 2 years ago

@xjlin0

Another option is not to use git (and ssh git clone) at all:

The latest version of pip supports this syntax:

<NAME-OF-THE-PACKAGE>@<FULL-URL-TO-THE-COMPRESSED-PYTHON-PACKAGE>.<tar.gz-or-zip>

For GitHub, it can be translated to:

<NAME-OF-THE-PACKAGE>@https://github.com/<USER>/<REPOSITORY-NAME>/archive/<BRANCH_NAME_OR_COMMIT_HASH>.<tar.gz-or-zip>

Sample:

celery==5.2.6  # pyup: < 6.0  # https://github.com/celery/celery
django-schedule@https://github.com/llazzaro/django-scheduler/archive/develop.tar.gz  ### THIS ###
django-celery-beat==2.2.1  # https://github.com/celery/django-celery-beat

firtk commented 10 months ago

This is an old issue but I encountered the same error. Turns out I forgot to update local/docs/Dockerfile. By looking at the error message, it seems the author of the issue also forgot it. If anyone encounters this issue, make sure to update all Dockerfiles.

cookiecutter / cookiecutter-django