Support Pipenv across all existing platforms

ehmatthes commented 1 year ago

Similar to #197, make sure Pipenv is supported across the three currently-supported platforms.

[x] Support Pipenv on Heroku.
[x] Support Pipenv on Fly.io.
[ ] Support Pipenv on Platform.sh. #200

ehmatthes commented 1 year ago

Running integration tests

[x] Heroku: ./integration_tests/test_deploy_process.sh -d pipenv -p heroku
[x] Fly.io: ./integration_tests/test_deploy_process.sh -d pipenv -p fly_io
- Failed; I assume this is because the Dockerfile only refers to requirements.txt.
[ ] Platform.sh: ./integration_tests/test_deploy_process.sh -d pipenv -p platform_sh

ehmatthes commented 1 year ago

Support Pipenv on Fly.io

[x] Pass an argument for the dep man approach to dockerfile template, and generate an appropriate line for installing requirements.
- [x] Consider adding --skip-lock, since there shouldn't be any reason to generate locks on the server. (No, this ignores the existing Pipfile.lock, which we definitely want to use.)
- [x] Failed due to indentation; hard to balanced indentation of if blocks in template with final indentation of generated Dockerfile. Prioritize correct indentation of generated Dockerfile.
- [x] Wrap the entire Run block in an if block, because the readability issue is about the overall block.
- Relevant section of generated Dockerfile:
```
COPY Pipfile /tmp/Pipfile
```

RUN set -ex && \ pip install --upgrade pip && \ pip install pipenv && \ pipenv install --skip-lock && \ rm -rf /root/.cache/

Pipfile:

% cat tmp_django_simple_deploy_test/Pipfile [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi"

[packages] whitenoise = "" # Added by simple_deploy command. dj-database-url = "" # Added by simple_deploy command. psycopg2-binary = "" # Added by simple_deploy command. gunicorn = "" # Added by simple_deploy command. django = "" django-bootstrap5 = "" requests = "" django-simple-deploy = ""

[dev-packages]

[requires] python_version = "3.10"

Relevant output:

=> [4/7] COPY Pipfile /tmp/Pipfile 0.0s => [5/7] RUN set -ex && pip install --upgrade pip && pip install pipenv && pipenv install --skip-lock && rm 10.1s => [6/7] COPY . /code/ 0.0s => ERROR [7/7] RUN ON_FLYIO_SETUP="1" python manage.py collectstatic --noinput 0.3s

[7/7] RUN ON_FLYIO_SETUP="1" python manage.py collectstatic --noinput:

10 0.310 Traceback (most recent call last):

10 0.310 File "/code/manage.py", line 11, in main

10 0.310 from django.core.management import execute_from_command_line

10 0.311 ModuleNotFoundError: No module named 'django'

[x] Run Heroku integration test, and look at generated Pipfile.


% cat tmp_django_simple_deploy_test/Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages] whitenoise = "" # Added by simple_deploy command. dj-database-url = "" # Added by simple_deploy command. psycopg2 = "<2.9" # Added by simple_deploy command. gunicorn = "" # Added by simple_deploy command. django = "" django-bootstrap5 = "" requests = "" django-simple-deploy = "*"

[dev-packages]

[requires] python_version = "3.10"



- [x] Pass integration test.
- [ ] Unit test this approach.

ehmatthes commented 1 year ago

More working notes - fly.io support

[x] Build a local env and install from the generated Pipfile.

Hung on Django for a moment, then installed fine:

% pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile...
An error occurred while installing django! Will try again.
🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 8/8 — 00:00:08
Installing initially failed dependencies...
☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
(p_env) eric@Erics-MacBook-Pro-2 pipenv_trial % pip freeze
asgiref==3.5.2
certifi==2022.12.7
charset-normalizer==2.1.1
distlib==0.3.6
dj-database-url==1.1.0
Django==4.1.4
...

[x] Run integration test, keep resources alive, cd to tmp project, remove --skip-lock, run fly deploy --verbose, record screen to examine fly output that disappears.

Scrolls and disappears:

Locking [dev-packages] dependencies...
Updated Pipfile.lock...
...
To activate this project's virtualenv, run pipenv shell.
Alternatively run a command inside the virutalenv with pipenv run

https://user-images.githubusercontent.com/1886842/207141186-c51fdaac-ffa2-4ea6-81eb-51ec09404797.mov

[x] Do I need to do something in Dockerfile to activate the environment? In fly.toml?
- "You can't run pipenv shell inside Docker" source
- Works for collectstatic: pipenv run python manage.py collectstatic Current Dockerfile, works through collectstatic stage but fails at migrate command:
```
ARG PYTHON_VERSION=3.10-slim-buster
```

FROM python:${PYTHON_VERSION}

ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ENV PIPENV_PIPFILE /tmp/Pipfile

RUN mkdir -p /code

WORKDIR /code

COPY Pipfile /tmp/Pipfile

RUN set -ex && \ pip install --upgrade pip && \ pip install pipenv && \ pipenv install && \ rm -rf /root/.cache/

COPY . /code/

RUN ON_FLYIO_SETUP="1" pipenv run python manage.py collectstatic --noinput

EXPOSE 8000

CMD ["gunicorn", "--bind", ":8000", "--workers", "2", "blog.wsgi"]


- [x] Use pipenv in fly.toml as well, for migrate command. This works.
- [x] May need to tweak final `CMD` in dockerfile?
- [x] Note: Fly may be incompatible with Pipenv, and maybe we should just generate requirements.txt. (nope)

ehmatthes commented 1 year ago

Run gunicorn after pipenv install

Deploy process works through migration, fails at gunicorn command. From Fly dashboard logs:

2022-12-12T20:11:10.474 runner[1b0a7489] sea [info] Starting virtual machine
2022-12-12T20:11:10.758 app[1b0a7489] sea [info] Starting init (commit: f447594)...
2022-12-12T20:11:10.782 app[1b0a7489] sea [info] Preparing to run: `gunicorn --bind :8000 --workers 2 blog.wsgi` as root
2022-12-12T20:11:10.791 app[1b0a7489] sea [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })

[x] Use pipenv run gunicorn?

ehmatthes commented 1 year ago

Works!

I've been tweaking a project that was initially deployed through the integration test. Process: modify Dockerfile and fly.toml, then run fly deploy --verbose. It finally works, and I can make an account, and the site and admin home page have proper css. I do get this message at the release phase:

--> This release will not be available until the release command succeeds.
     Starting instance
     Configuring virtual machine
     Pulling container image
     Unpacking image
     Preparing kernel init
     Configuring firecracker
     Starting virtual machine
     Starting init (commit: f447594)...
     Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
     no label, UUID=bb945d0a-7089-4bf6-ac44-a55bba78ab63
     Preparing to run: `pipenv run python manage.py migrate` as root
     2022/12/12 20:19:54 listening on [fdaa:0:94e0:a7b:bbfb:8717:d0fb:2]:22 (DNS: [fdaa::3]:53)
     System check identified some issues:
     WARNINGS:
     ?: (staticfiles.W004) The directory '/code/static' in the STATICFILES_DIRS setting does not exist.
     Operations to perform:
       Apply all migrations: admin, auth, blogs, contenttypes, sessions
     Running migrations:
       No migrations to apply.
     Starting clean up.

[ ] May need to implement better support of static files. Maybe whitenoise isn't available because it was installed through Pipenv?

Dockerfile:

ARG PYTHON_VERSION=3.10-slim-buster

FROM python:${PYTHON_VERSION}

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV PIPENV_PIPFILE /tmp/Pipfile

RUN mkdir -p /code

WORKDIR /code

COPY Pipfile /tmp/Pipfile

RUN set -ex && \
    pip install --upgrade pip && \
    pip install pipenv && \
    pipenv install && \
    rm -rf /root/.cache/

COPY . /code/

RUN ON_FLYIO_SETUP="1" pipenv run python manage.py collectstatic --noinput

EXPOSE 8000

CMD ["pipenv", "run", "gunicorn", "--bind", ":8000", "--workers", "2", "blog.wsgi"]

fly.toml:

app = "ancient-bird-3980"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  PORT = "8000"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8000
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/static/"

[deploy]
  release_command = "pipenv run python manage.py migrate"

ehmatthes commented 1 year ago

Final testing for Fly.io

[x] Generate Dockerfile that matches above.
[x] Generate fly.toml that matches above.
[x] Run integration test again. Passes?

ehmatthes commented 1 year ago

Cleaning up Fly.io

[x] This will be much more readable if the using_pipenv logic is moved to deploy.py, and we just have two separate, simple templates for the different dependency management systems. It's really not easy to read the current Dockerfile template.
[x] Consider the same approach for fly.toml, but that's much simpler because it's only the single deploy command that's conditional. Still, it's really nice to look at these templates and read them as easily as you can read the final file. (Not a clear enough advantage right now, especially considering the unit test reference files should be better to look at.)

ehmatthes / django-simple-deploy