search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.83k stars 2.12k forks source link

`docker build` on Windows creates an image whose `/start.py` entrypoint has Windows (`\r\n`) line endings, which then fails to start #13691

Open qknight opened 2 years ago

qknight commented 2 years ago

Description

I'm not able to build the docker image and then run it on my windows machine using docker desktop:

git clone https://github.com/matrix-org/synapse.git
git branch -l
* develop

The build:

docker build --no-cache -f docker/Dockerfile -t fred2 .
[+] Building 175.5s (25/25) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                  0.0s
 => => transferring dockerfile: 6.72kB                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                     0.0s
 => => transferring context: 181B                                                                                                                                                                                                     0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                            1.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                                                    0.8s
 => [internal] load build context                                                                                                                                                                                                     1.1s
 => => transferring context: 7.68MB                                                                                                                                                                                                   1.1s
 => CACHED [stage-2 1/5] FROM docker.io/library/python:3.9-slim@sha256:dcf2eafca55558d8b1aa73edd6aa41b7187c5bcb63e533a7b04a0673f81f37fe                                                                                               0.0s
 => [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked     apt-get update -qq && apt-get install -yqq       build-essential cargo git libff  55.7s
 => [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update -qq && apt-get install -yqq     curl     gosu     libjpeg62-turbo     li  65.3s
 => [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update -qq && apt-get install -yqq     build-essential     libffi-dev     libjp  128.9s
 => [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"                         19.4s
 => [requirements 4/6] WORKDIR /synapse                                                                                                                                                                                               0.1s
 => [requirements 5/6] COPY pyproject.toml poetry.lock /synapse/                                                                                                                                                                      0.0s
 => [requirements 6/6] RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then     /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes};   else     to  3.0s
 => [builder 3/7] COPY --from=requirements /synapse/requirements.txt /synapse/                                                                                                                                                        0.0s
 => [builder 4/7] RUN --mount=type=cache,target=/root/.cache/pip   pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt                                                                  34.6s
 => [builder 5/7] COPY synapse /synapse/synapse/                                                                                                                                                                                      0.2s
 => [builder 6/7] COPY pyproject.toml README.rst /synapse/                                                                                                                                                                            0.0s
 => [builder 7/7] RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then     pip install --prefix="/install" --no-deps --no-warn-script-location /synapse[all];   else     pip install --prefix="/install" --no-warn-script-locatio  5.0s
 => [stage-2 3/5] COPY --from=builder /install /usr/local                                                                                                                                                                             2.4s
 => [stage-2 4/5] COPY ./docker/start.py /start.py                                                                                                                                                                                    0.0s
 => [stage-2 5/5] COPY ./docker/conf /conf                                                                                                                                                                                            0.0s
 => exporting to image                                                                                                                                                                                                                1.1s
 => => exporting layers                                                                                                                                                                                                               1.0s
 => => writing image sha256:2508ae439a33c1f95b2c39e939c1499fce6ec6e61864d8e354d464c9dfc70fef                                                                                                                                          0.0s
 => => naming to docker.io/library/fred2

And running the generate

docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host fred2 generate exec /start.py: no such file or directory

Steps to reproduce

git clone https://github.com/matrix-org/synapse.git cd synapse docker build --no-cache -f docker/Dockerfile -t fred2 . docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host fred2 generate

Homeserver

does not apply i think

Synapse Version

v1.66.0 / v1.60.0 / v1.56.0

Installation Method

Other (please mention below)

Platform

My docker desktop configuration:

docker version
Client:
 Cloud integration: v1.0.28
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:09:02 2022
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.11.1 (84025)
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I'm using WSL2 backend in for docker and this is my configuration:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "features": {
    "buildkit": true
  }
}

Relevant log output

docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host fred2 generate
exec /start.py: no such file or directory

Anything else that would be useful to know?

I can reproduce this on a different Windows machine with similar setup:

Update: Seems to be a Docker Windows problem because on my linux host it was able to build&execute properly.

squahtx commented 2 years ago

The issue seems to be that \r characters end up inside files copied into the Docker image:

> docker run -it --entrypoint=bash fred2
root@903d7de32e56:/# ls
bin  boot  conf  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  start.py  sys  tmp  usr  var
root@903d7de32e56:/# /start.py
bash: /start.py: /usr/local/bin/python^M: bad interpreter: No such file or directory

start.py is successfully copied, just with Windows line endings.

squahtx commented 2 years ago

As a workaround, you could try re-saving docker/start.py with Unix line endings and rebuilding the image.

richvdh commented 2 years ago

Presumably docker is just copying the raw file?

As a workaround, you could try re-saving docker/start.py with Unix line endings and rebuilding the image.

If you've got windows line endings on start.py, its unlikely to be the only file so afflicted.

I'm not very familiar with Windows stuff, but this doesn't sound like a problem we can solve?

squahtx commented 2 years ago

Presumably docker is just copying the raw file?

I'm assuming that's what's happening. git clone converts the line endings to \r\n, then docker copies them verbatim into the container.

It is possible to tell Git to leave the line-endings as-is when cloning a repo, via some .gitattributes magic: https://git-scm.com/docs/gitattributes . I can't remember off the top of my head how, and I'm not convinced that's the right solution.

Despite being on Linux, python seems happy to run files that contain Windows line endings:

root@903d7de32e56:/# python start.py
Config file '/data/homeserver.yaml' does not exist. [...]

so once start.py is running, things might just work if you're lucky, assuming we never leave the Python interpreter / try to exec anything else (which may or may not be a valid assumption).

MadLittleMods commented 2 years ago

fwiw, I have the following git config magic on Windows not even related to Synapse to avoid problems.

~/.gitconfig

[core]
    autocrlf = input
    eol = lf

core.autocrlf = true

  1. Text files checked-out from the repository will keep original EOL characters in your working tree.
  2. Text files in your working tree with CRLF characters are normalized to LF when committed back to the repository.

-- https://stackoverflow.com/a/4425433/796832

Also a nice table in https://stackoverflow.com/a/41282375/796832 for how core.autocrlf = true interacts with git commit and git checkout