Closed abrichr closed 3 years ago
I ran the curl
command from an EC2 server and it completed without any issues.
I forked the Poetry repo, replaced the RuntimeError
with a print
, and modified backend.dockerfile
to download my fork:
...
11 RUN curl -sSL https://raw.githubusercontent.com/abrichr/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python && \
...
Unfortunately it appears that the file is indeed corrupt:
#7 5.867 Installing version: 1.1.6
#7 5.867 - Downloading poetry-1.1.6-linux.tar.gz (72.33MB)
#7 5.867 Hashes for poetry-1.1.6-linux.tar.gz do not match: a812d3e0e1ff93b6a69fa04bd2fdd81bd502d8788314387fb554b5807c2628f6 != 0e08ead74244bc593c5c305698e6ab8714f92fe41e65bd48c2737a1a0d73f855
#7 5.872 Traceback (most recent call last):
#7 5.872 File "<stdin>", line 1087, in <module>
#7 5.872 File "<stdin>", line 1083, in main
#7 5.872 File "<stdin>", line 363, in run
#7 5.872 File "<stdin>", line 528, in install
#7 5.872 File "<stdin>", line 548, in make_lib
#7 5.872 File "<stdin>", line 630, in _make_lib
#7 5.872 File "<stdin>", line 636, in extract_lib
#7 5.872 File "/usr/local/lib/python3.7/tarfile.py", line 2000, in extractall
#7 5.872 numeric_owner=numeric_owner)
#7 5.872 File "/usr/local/lib/python3.7/tarfile.py", line 2042, in extract
#7 5.872 numeric_owner=numeric_owner)
#7 5.872 File "/usr/local/lib/python3.7/tarfile.py", line 2112, in _extract_member
#7 5.872 self.makefile(tarinfo, targetpath)
#7 5.872 File "/usr/local/lib/python3.7/tarfile.py", line 2161, in makefile
#7 5.872 copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
#7 5.872 File "/usr/local/lib/python3.7/tarfile.py", line 247, in copyfileobj
#7 5.872 buf = src.read(bufsize)
#7 5.872 File "/usr/local/lib/python3.7/gzip.py", line 287, in read
#7 5.872 return self._buffer.read(size)
#7 5.872 File "/usr/local/lib/python3.7/_compression.py", line 68, in readinto
#7 5.872 data = self.read(len(byte_view))
#7 5.872 File "/usr/local/lib/python3.7/gzip.py", line 493, in read
#7 5.872 raise EOFError("Compressed file ended before the "
#7 5.872 EOFError: Compressed file ended before the end-of-stream marker was reached
Oddly enough this only appears to happen when installing the backend; earlier in the build process it runs successfully for the celeryworker:
...
=> [internal] load build definition from celeryworker.dockerfile
...
=> CACHED [ 4/11] RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python && cd /usr/local/bin && ln -s /opt/poetry/bin/poetry && poetry config virtualenvs.create false 0.0s
=> CACHED [ 5/11] COPY ./app/pyproject.toml ./app/poetry.lock* /app/ 0.0s
=> CACHED [ 6/11] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi" 0.0s
=> CACHED [ 7/11] RUN bash -c "if [ true == 'true' ] ; then pip install jupyterlab ; fi" 0.0s
=> CACHED [ 8/11] COPY ./app /app
...
In my initial description, I neglected to mention that between removing all Docker images and attempting to re-install, I encountered an issue with Docker for Mac that was causing image downloads to fail and hang ( https://github.com/docker/for-mac/issues/5615 ). I updated to the latest version, which resolved that issue. Perhaps this is related...
I tried combining backend.dockerfile
and celeryworker.dockerfile
as per https://github.com/tiangolo/full-stack-fastapi-postgresql/pull/191 , but received the following error:
ERROR: pull access denied for backend, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
I tried downgrading Docker for Mac from 3.3.3 to 2.4.0 ( downloaded from https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2400 ), but am still receiving the same error.
I purged all docker images using this script: https://gist.github.com/beeman/aca41f3ebd2bf5efbd9d7fef09eac54d#gistcomment-3735369
The new error is:
Status: Downloaded newer image for tiangolo/node-frontend:10
---> 46be30c070b7
Step 2/13 : WORKDIR /app
---> Running in 123c8f68f698
Removing intermediate container 123c8f68f698
---> 1d6c29aa7890
Step 3/13 : COPY package*.json /app/
---> 7485faf89dbf
Step 4/13 : RUN npm install
---> Running in 635a0f927ff5
Unhandled rejection Error: spawn /usr/bin/git ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
npm ERR! cb() never called!
npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://npm.community>
It turns out this was a problem with my wired (!) ethernet connection only -- switching to WiFi fixed the issue for me.
It appears I spoke too soon. Switching to WiFi appears to have fixed the corrupt Poetry binary download issue, but the npm error remains.
Here's the full error:
docker-compose up -d
WARNING: Some services (backend, db, flower, frontend, pgadmin, proxy) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Building frontend
Step 1/13 : FROM tiangolo/node-frontend:10 as build-stage
10: Pulling from tiangolo/node-frontend
e79bb959ec00: Pull complete
d4b7902036fe: Pull complete
1b2a72d4e030: Pull complete
d54db43011fd: Pull complete
69d473365bb3: Pull complete
6e2490ee2dc8: Pull complete
6381e1b5feaa: Pull complete
b1efb42c63e2: Pull complete
bccd4a781864: Pull complete
193161fe4092: Pull complete
dd4fcf0b263b: Pull complete
bb90dadea0a2: Pull complete
Digest: sha256:10046f2c98460dc97c490e98ee828d2f64aa568d6e52a0ccedaccdb097b37bc4
Status: Downloaded newer image for tiangolo/node-frontend:10
---> 46be30c070b7
Step 2/13 : WORKDIR /app
---> Running in 094cf5fee62d
Removing intermediate container 094cf5fee62d
---> 411bab064218
Step 3/13 : COPY package*.json /app/
---> d01e6c5e23c0
Step 4/13 : RUN npm install
---> Running in 85e05c9e8bdf
Unhandled rejection Error: spawn /usr/bin/git ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
npm ERR! cb() never called!
npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! <https://npm.community>
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-05-10T20_47_27_104Z-debug.log
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c npm install' returned a non-zero code: 1
To access the last container, I ran:
docker ps -a
docker commit 85e05c9e8bdf
docker run -it 6358d767efd3ae3741c2f0565724e7d3bc3465ef617b1fe934f3fff18d9e4c03 bash -il
Then to inspect the logs:
tail /root/.npm/_logs/2021-05-10T20_47_27_104Z-debug.log
25164 http fetch GET 200 https://registry.npmjs.org/aws-sdk/-/aws-sdk-2.775.0.tgz 17038ms
25165 silly extract aws-sdk@2.775.0 extracted to /app/node_modules/.staging/aws-sdk-5ca7164e (17105ms)
25166 http fetch GET 200 https://registry.npmjs.org/vuetify/-/vuetify-2.3.14.tgz 17753ms
25167 silly extract vuetify@2.3.14 extracted to /app/node_modules/.staging/vuetify-3940eb0d (17811ms)
25168 http fetch GET 200 https://registry.npmjs.org/date-fns/-/date-fns-2.19.0.tgz 17614ms
25169 silly extract date-fns@2.19.0 extracted to /app/node_modules/.staging/date-fns-317fda1d (17852ms)
25170 timing npm Completed in 24171ms
25171 error cb() never called!
25172 error This is an error with npm itself. Please report this error at:
25173 error <https://npm.community>
Not sure where to go from here...
Solved the npm error by removing package-lock.json
.
It appears the solution to the original issue is combining backend.dockerfile
and celeryworker.dockerfile
as per https://github.com/tiangolo/full-stack-fastapi-postgresql/pull/191.
I also had to build with docker-compose up --build --force-recreate -d
.
It appears I once again spoke too soon. The problem appears to be intermittent. I haven't yet found a reliable way to trigger or avoid the npm or poetry errors.
This appears to be related to my network connection. Switching to my mobile connection resolved the issue. Enabling Network Link Conditioner (as per https://nshipster.com/network-link-conditioner/) did not.
Edit: it appears I can avoid the error by building containers separately, i.e. the following produces errors:
docker-compose up --build -d
The following does not:
docker-compose up --build -d backend
...
docker-compose up --build -d celeryworker
Thanks for reporting back and closing the issue 👍
Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.
It all started when I tried to use a
celery.chord
to run two tasks in parallel, followed by another task to consume their result. I received an exception similar to the one described here: https://stackoverflow.com/q/45240564/95989.This issue is not about that. I found this comment https://github.com/tiangolo/full-stack-fastapi-postgresql/issues/293#issuecomment-708665327 , which appeared to resolve a similar issue:
So I decided to try that as well. Unfortunately, upon attempting to rebuild, I encountered this error:
Guessing that it had something to do with Docker caching, I tried appending
?foo=bar
to the URL toget-poetry.py
inbackend.dockerfile
in order to bust the cache:Unfortunately this did not resolve the issue.
Disconcertingly, I'm not able to find any results for this issue online, i.e. https://www.google.com/search?q=%22get-poetry.py%22+%22runtimeerror%22+%22hashes%22+%22do+not+match%22 returns only four results, none of which are helpful.
The source appears to be https://github.com/python-poetry/poetry/blob/master/get-poetry.py#L624 , and that line has been unchanged for three years.
Any suggestions would be appreciated!