fastapi / full-stack-fastapi-template

Full stack, modern web application template. Using FastAPI, React, SQLModel, PostgreSQL, Docker, GitHub Actions, automatic HTTPS and more.
MIT License
27.76k stars 4.95k forks source link

Poetry RuntimeError workaround #377

Closed abrichr closed 3 years ago

abrichr commented 3 years ago

It all started when I tried to use a celery.chord to run two tasks in parallel, followed by another task to consume their result. I received an exception similar to the one described here: https://stackoverflow.com/q/45240564/95989.

This issue is not about that. I found this comment https://github.com/tiangolo/full-stack-fastapi-postgresql/issues/293#issuecomment-708665327 , which appeared to resolve a similar issue:

After a deleting all the volumes and images and rebuilding everything again with docker-compose up -d, it works now!

So I decided to try that as well. Unfortunately, upon attempting to rebuild, I encountered this error:

#7 3.111 Installing version: 1.1.6
#7 3.111   - Downloading poetry-1.1.6-linux.tar.gz (72.33MB)
#7 3.111 Traceback (most recent call last):
#7 3.111   File "<stdin>", line 1086, in <module>
#7 3.111   File "<stdin>", line 1082, in main
#7 3.111   File "<stdin>", line 363, in run
#7 3.111   File "<stdin>", line 528, in install
#7 3.111   File "<stdin>", line 548, in make_lib
#7 3.111   File "<stdin>", line 625, in _make_lib
#7 3.111 RuntimeError: Hashes for poetry-1.1.6-linux.tar.gz do not match: a812d3e0e1ff93b6a69fa04bd2fdd81bd502d8788314387fb554b5807c2628f6 != 763eae856b0eca592c0fecb0247d8486b8e72eb35215be2fe993a32e9c1f1c06

Guessing that it had something to do with Docker caching, I tried appending ?foo=bar to the URL to get-poetry.py in backend.dockerfile in order to bust the cache:

 11 RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py?foo=bar | POETRY_HOME=/opt/poetry python && \
...

Unfortunately this did not resolve the issue.

Disconcertingly, I'm not able to find any results for this issue online, i.e. https://www.google.com/search?q=%22get-poetry.py%22+%22runtimeerror%22+%22hashes%22+%22do+not+match%22 returns only four results, none of which are helpful.

The source appears to be https://github.com/python-poetry/poetry/blob/master/get-poetry.py#L624 , and that line has been unchanged for three years.

Any suggestions would be appreciated!

abrichr commented 3 years ago

I ran the curl command from an EC2 server and it completed without any issues.

I forked the Poetry repo, replaced the RuntimeError with a print, and modified backend.dockerfile to download my fork:

...
 11 RUN curl -sSL https://raw.githubusercontent.com/abrichr/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python && \
...

Unfortunately it appears that the file is indeed corrupt:

#7 5.867 Installing version: 1.1.6
#7 5.867   - Downloading poetry-1.1.6-linux.tar.gz (72.33MB)
#7 5.867 Hashes for poetry-1.1.6-linux.tar.gz do not match: a812d3e0e1ff93b6a69fa04bd2fdd81bd502d8788314387fb554b5807c2628f6 != 0e08ead74244bc593c5c305698e6ab8714f92fe41e65bd48c2737a1a0d73f855
#7 5.872 Traceback (most recent call last):
#7 5.872   File "<stdin>", line 1087, in <module>
#7 5.872   File "<stdin>", line 1083, in main
#7 5.872   File "<stdin>", line 363, in run
#7 5.872   File "<stdin>", line 528, in install
#7 5.872   File "<stdin>", line 548, in make_lib
#7 5.872   File "<stdin>", line 630, in _make_lib
#7 5.872   File "<stdin>", line 636, in extract_lib
#7 5.872   File "/usr/local/lib/python3.7/tarfile.py", line 2000, in extractall
#7 5.872     numeric_owner=numeric_owner)
#7 5.872   File "/usr/local/lib/python3.7/tarfile.py", line 2042, in extract
#7 5.872     numeric_owner=numeric_owner)
#7 5.872   File "/usr/local/lib/python3.7/tarfile.py", line 2112, in _extract_member
#7 5.872     self.makefile(tarinfo, targetpath)
#7 5.872   File "/usr/local/lib/python3.7/tarfile.py", line 2161, in makefile
#7 5.872     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
#7 5.872   File "/usr/local/lib/python3.7/tarfile.py", line 247, in copyfileobj
#7 5.872     buf = src.read(bufsize)
#7 5.872   File "/usr/local/lib/python3.7/gzip.py", line 287, in read
#7 5.872     return self._buffer.read(size)
#7 5.872   File "/usr/local/lib/python3.7/_compression.py", line 68, in readinto
#7 5.872     data = self.read(len(byte_view))
#7 5.872   File "/usr/local/lib/python3.7/gzip.py", line 493, in read
#7 5.872     raise EOFError("Compressed file ended before the "
#7 5.872 EOFError: Compressed file ended before the end-of-stream marker was reached

Oddly enough this only appears to happen when installing the backend; earlier in the build process it runs successfully for the celeryworker:

...
 => [internal] load build definition from celeryworker.dockerfile                                                                                                                                                                                                                                                                                                                                                                  
...
 => CACHED [ 4/11] RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python &&     cd /usr/local/bin &&     ln -s /opt/poetry/bin/poetry &&     poetry config virtualenvs.create false                                                                                                                                                                            0.0s
 => CACHED [ 5/11] COPY ./app/pyproject.toml ./app/poetry.lock* /app/                                                                                                                                                                                                                                                                                                                                                               0.0s
 => CACHED [ 6/11] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi"                                                                                                                                                                                                                                                                                                0.0s
 => CACHED [ 7/11] RUN bash -c "if [ true == 'true' ] ; then pip install jupyterlab ; fi"                                                                                                                                                                                                                                                                                                                                           0.0s
 => CACHED [ 8/11] COPY ./app /app    
...
abrichr commented 3 years ago

In my initial description, I neglected to mention that between removing all Docker images and attempting to re-install, I encountered an issue with Docker for Mac that was causing image downloads to fail and hang ( https://github.com/docker/for-mac/issues/5615 ). I updated to the latest version, which resolved that issue. Perhaps this is related...

abrichr commented 3 years ago

I tried combining backend.dockerfile and celeryworker.dockerfile as per https://github.com/tiangolo/full-stack-fastapi-postgresql/pull/191 , but received the following error:

ERROR: pull access denied for backend, repository does not exist or may require 'docker login': denied: requested access to the resource is denied                                                                  

I tried downgrading Docker for Mac from 3.3.3 to 2.4.0 ( downloaded from https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2400 ), but am still receiving the same error.

abrichr commented 3 years ago

I purged all docker images using this script: https://gist.github.com/beeman/aca41f3ebd2bf5efbd9d7fef09eac54d#gistcomment-3735369

The new error is:

Status: Downloaded newer image for tiangolo/node-frontend:10
 ---> 46be30c070b7
Step 2/13 : WORKDIR /app
 ---> Running in 123c8f68f698
Removing intermediate container 123c8f68f698
 ---> 1d6c29aa7890
Step 3/13 : COPY package*.json /app/
 ---> 7485faf89dbf
Step 4/13 : RUN npm install
 ---> Running in 635a0f927ff5
Unhandled rejection Error: spawn /usr/bin/git ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
    at onErrorNT (internal/child_process.js:415:16)
    at process._tickCallback (internal/process/next_tick.js:63:19)
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>
abrichr commented 3 years ago

It turns out this was a problem with my wired (!) ethernet connection only -- switching to WiFi fixed the issue for me.

abrichr commented 3 years ago

It appears I spoke too soon. Switching to WiFi appears to have fixed the corrupt Poetry binary download issue, but the npm error remains.

Here's the full error:

docker-compose up -d                                                                                                                           
WARNING: Some services (backend, db, flower, frontend, pgadmin, proxy) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.                                                                                                                                               
Building frontend                                                                                                                                                                                                                                                                                                                                                   
Step 1/13 : FROM tiangolo/node-frontend:10 as build-stage                                                                                                                                                                                                                                                                                                           
10: Pulling from tiangolo/node-frontend                                                                                                                                                                                                                                                                                                                             
e79bb959ec00: Pull complete                                                                                                                                                                                                                                                                                                                                         
d4b7902036fe: Pull complete                                                                                                                                                       
1b2a72d4e030: Pull complete                                                                                                                                                                                                                                                                                                                                         
d54db43011fd: Pull complete                                                                                                                                                       
69d473365bb3: Pull complete                                                                                                                                                                                                                                                                                                                                         
6e2490ee2dc8: Pull complete                                                                                                                                                                                                                                                                                                                                         
6381e1b5feaa: Pull complete                                                                                                                                                                                                                                                                                                                                         
b1efb42c63e2: Pull complete                                                                                                                                                                                                                                                                                                                                         
bccd4a781864: Pull complete                                                                                                                                                                                                                                                                                                                                         
193161fe4092: Pull complete                                                                                                                                                                                                                                                                                                                                         
dd4fcf0b263b: Pull complete                                                                                                                                                                                                                                                                                                                                         
bb90dadea0a2: Pull complete                                                                                                                                                                                                                                                                                                                                         
Digest: sha256:10046f2c98460dc97c490e98ee828d2f64aa568d6e52a0ccedaccdb097b37bc4                                                                                                   
Status: Downloaded newer image for tiangolo/node-frontend:10                                                                                                                                                                                                                                                                                                        
 ---> 46be30c070b7                                                                                                                                                                
Step 2/13 : WORKDIR /app                                                                                                                                                                                                                                                                                                                                            
 ---> Running in 094cf5fee62d                                                                                                                                                                                                                                                                                                                                       
Removing intermediate container 094cf5fee62d                                                                                                                                                                                                                                                                                                                        
 ---> 411bab064218                                                                                                                                                                
Step 3/13 : COPY package*.json /app/                                                                                                                                                                                                                                                                                                                                
 ---> d01e6c5e23c0                                                                                                                                                                                                                                                                                                                                                  
Step 4/13 : RUN npm install                                                                                                                                                                                                                                                                                                                                         
 ---> Running in 85e05c9e8bdf                                                                                                                                                                                                                                                                                                                                       
Unhandled rejection Error: spawn /usr/bin/git ENOENT                                                                                                                              
    at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)                                                                                                                                                                                                                                                                                       
    at onErrorNT (internal/child_process.js:415:16)                                                                                                                               
    at process._tickCallback (internal/process/next_tick.js:63:19)                                                                                                                                                                                                                                                                                                  
npm ERR! cb() never called!                                                                                                                                                                                                                                                                                                                                         

npm ERR! This is an error with npm itself. Please report this error at:                                                                                                                                                                                                                                                                                             
npm ERR!     <https://npm.community>                                                                                                                                              

npm ERR! A complete log of this run can be found in:                                                                                                                                                                                                                                                                                                                
npm ERR!     /root/.npm/_logs/2021-05-10T20_47_27_104Z-debug.log                                                                                                                                                                                                                                                                                                    
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c npm install' returned a non-zero code: 1    

To access the last container, I ran:

docker ps -a
docker commit 85e05c9e8bdf 
docker run -it 6358d767efd3ae3741c2f0565724e7d3bc3465ef617b1fe934f3fff18d9e4c03 bash -il

Then to inspect the logs:

tail /root/.npm/_logs/2021-05-10T20_47_27_104Z-debug.log
25164 http fetch GET 200 https://registry.npmjs.org/aws-sdk/-/aws-sdk-2.775.0.tgz 17038ms
25165 silly extract aws-sdk@2.775.0 extracted to /app/node_modules/.staging/aws-sdk-5ca7164e (17105ms)
25166 http fetch GET 200 https://registry.npmjs.org/vuetify/-/vuetify-2.3.14.tgz 17753ms
25167 silly extract vuetify@2.3.14 extracted to /app/node_modules/.staging/vuetify-3940eb0d (17811ms)
25168 http fetch GET 200 https://registry.npmjs.org/date-fns/-/date-fns-2.19.0.tgz 17614ms
25169 silly extract date-fns@2.19.0 extracted to /app/node_modules/.staging/date-fns-317fda1d (17852ms)
25170 timing npm Completed in 24171ms
25171 error cb() never called!
25172 error This is an error with npm itself. Please report this error at:
25173 error <https://npm.community>

Not sure where to go from here...

abrichr commented 3 years ago

Solved the npm error by removing package-lock.json.

It appears the solution to the original issue is combining backend.dockerfile and celeryworker.dockerfile as per https://github.com/tiangolo/full-stack-fastapi-postgresql/pull/191.

I also had to build with docker-compose up --build --force-recreate -d.

abrichr commented 3 years ago

It appears I once again spoke too soon. The problem appears to be intermittent. I haven't yet found a reliable way to trigger or avoid the npm or poetry errors.

abrichr commented 3 years ago

This appears to be related to my network connection. Switching to my mobile connection resolved the issue. Enabling Network Link Conditioner (as per https://nshipster.com/network-link-conditioner/) did not.

Edit: it appears I can avoid the error by building containers separately, i.e. the following produces errors:

docker-compose up --build -d

The following does not:

docker-compose up --build -d backend
...
docker-compose up --build -d celeryworker
tiangolo commented 1 year ago

Thanks for reporting back and closing the issue 👍

Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.