search

google / gcp_scanner

A comprehensive scanner for Google Cloud
Apache License 2.0
311 stars 97 forks source link

docker build fails #114

Closed grumpyp closed 1 year ago

grumpyp commented 1 year ago

Affected Component

Describe the bug

This is a known issue with Python3.7 https://github.com/grpc/grpc/issues/24556

=> [internal] load build context                                                                                                                                                                      0.3s
 => => transferring context: 93.91kB                                                                                                                                                                   0.0s
 => [2/8] RUN mkdir /home/sa_scanner                                                                                                                                                                   0.4s
 => [3/8] COPY src/ /home/sa_scanner/                                                                                                                                                                  0.0s
 => [4/8] COPY pyproject.toml /home/sa_scanner/                                                                                                                                                        0.0s
 => [5/8] COPY README.md /home/sa_scanner                                                                                                                                                              0.0s
 => [6/8] WORKDIR /home/sa_scanner                                                                                                                                                                     0.0s
 => ERROR [7/8] RUN pip install .                                                                                                                                                                     18.9s
------                                                                                                                                                                                                      
 > [7/8] RUN pip install .:                                                                                                                                                                                 
#12 1.139 Processing /home/sa_scanner                                                                                                                                                                       
#12 1.141   Installing build dependencies: started                                                                                                                                                          
#12 5.902   Installing build dependencies: finished with status 'done'                                                                                                                                      
#12 5.905   Getting requirements to build wheel: started                                                                                                                                                    
#12 5.967   Getting requirements to build wheel: finished with status 'done'
#12 5.972   Preparing metadata (pyproject.toml): started
#12 6.100   Preparing metadata (pyproject.toml): finished with status 'done'
#12 6.500 Collecting google-api-python-client==2.80.0
#12 6.662   Downloading google_api_python_client-2.80.0-py2.py3-none-any.whl (11.0 MB)
#12 9.259      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.0/11.0 MB 4.3 MB/s eta 0:00:00
#12 9.344 Collecting google-cloud-container==2.17.4
#12 9.391   Downloading google_cloud_container-2.17.4-py2.py3-none-any.whl (217 kB)
#12 9.441      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 217.7/217.7 kB 4.4 MB/s eta 0:00:00
#12 9.580 Collecting google-cloud-iam==2.11.2
#12 9.630   Downloading google_cloud_iam-2.11.2-py2.py3-none-any.whl (115 kB)
#12 9.652      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.0/115.0 kB 5.3 MB/s eta 0:00:00
#12 9.718 Collecting httplib2==0.21.0
#12 9.760   Downloading httplib2-0.21.0-py3-none-any.whl (96 kB)
#12 9.778      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.8/96.8 kB 5.5 MB/s eta 0:00:00
#12 9.825 Collecting pyu2f==0.1.5
#12 9.874   Downloading pyu2f-0.1.5.tar.gz (27 kB)
#12 9.885   Preparing metadata (setup.py): started
#12 10.28   Preparing metadata (setup.py): finished with status 'done'
#12 10.38 Collecting requests==2.28.2
#12 10.48   Downloading requests-2.28.2-py3-none-any.whl (62 kB)
#12 10.49      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 4.8 MB/s eta 0:00:00
#12 10.65 Collecting google-auth<3.0.0dev,>=1.19.0
#12 10.70   Downloading google_auth-2.16.2-py2.py3-none-any.whl (177 kB)
#12 10.75      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.2/177.2 kB 4.1 MB/s eta 0:00:00
#12 10.80 Collecting google-auth-httplib2>=0.1.0
#12 10.84   Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
#12 10.93 Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
#12 10.98   Downloading google_api_core-2.11.0-py3-none-any.whl (120 kB)
#12 11.01      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.3/120.3 kB 4.2 MB/s eta 0:00:00
#12 11.11 Collecting uritemplate<5,>=3.0.1
#12 11.19   Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
#12 11.31 Collecting proto-plus<2.0.0dev,>=1.22.0
#12 11.40   Downloading proto_plus-1.22.2-py3-none-any.whl (47 kB)
#12 11.49      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.9/47.9 kB 6.2 MB/s eta 0:00:00
#12 11.79 Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
#12 11.83   Downloading protobuf-4.22.1-cp37-abi3-manylinux2014_aarch64.whl (301 kB)
#12 11.90      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.2/301.2 kB 4.7 MB/s eta 0:00:00
#12 11.98 Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
#12 12.03   Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
#12 12.04      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 8.0 MB/s eta 0:00:00
#12 12.10 Collecting six
#12 12.14   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
#12 12.25 Collecting charset-normalizer<4,>=2
#12 12.29   Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (193 kB)
#12 12.32      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.0/194.0 kB 6.9 MB/s eta 0:00:00
#12 12.38 Collecting idna<4,>=2.5
#12 12.42   Downloading idna-3.4-py3-none-any.whl (61 kB)
#12 12.43      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 6.6 MB/s eta 0:00:00
#12 12.50 Collecting urllib3<1.27,>=1.21.1
#12 12.55   Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
#12 12.57      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 7.0 MB/s eta 0:00:00
#12 12.63 Collecting certifi>=2017.4.17
#12 12.68   Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
#12 12.70      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 7.5 MB/s eta 0:00:00
#12 12.76 Collecting googleapis-common-protos<2.0dev,>=1.56.2
#12 12.80   Downloading googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
#12 12.83      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.6/223.6 kB 8.1 MB/s eta 0:00:00
#12 13.46 Collecting grpcio<2.0dev,>=1.33.2
#12 13.51   Downloading grpcio-1.51.3.tar.gz (22.1 MB)
#12 16.95      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 22.1/22.1 MB 6.9 MB/s eta 0:00:00
#12 18.13   Preparing metadata (setup.py): started
#12 18.38   Preparing metadata (setup.py): finished with status 'error'
#12 18.38   error: subprocess-exited-with-error
#12 18.38   
#12 18.38   × python setup.py egg_info did not run successfully.
#12 18.38   │ exit code: 1
#12 18.38   ╰─> [14 lines of output]
#12 18.38       Traceback (most recent call last):
#12 18.38         File "<string>", line 2, in <module>
#12 18.38         File "<pip-setuptools-caller>", line 34, in <module>
#12 18.38         File "/tmp/pip-install-7egp0l84/grpcio_2bc3ade628c0443a83519a9e2c691087/setup.py", line 262, in <module>
#12 18.38           if check_linker_need_libatomic():
#12 18.38              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 18.38         File "/tmp/pip-install-7egp0l84/grpcio_2bc3ade628c0443a83519a9e2c691087/setup.py", line 209, in check_linker_need_libatomic
#12 18.38           cpp_test = subprocess.Popen(cxx + ['-x', 'c++', '-std=c++14', '-'],
#12 18.38                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 18.38         File "/usr/local/lib/python3.11/subprocess.py", line 1024, in __init__
#12 18.38           self._execute_child(args, executable, preexec_fn, close_fds,
#12 18.38         File "/usr/local/lib/python3.11/subprocess.py", line 1901, in _execute_child
#12 18.38           raise child_exception_type(errno_num, err_msg, err_filename)
#12 18.38       FileNotFoundError: [Errno 2] No such file or directory: 'c++'
#12 18.38       [end of output]
#12 18.38   
#12 18.38   note: This error originates from a subprocess, and is likely not a problem with pip.
#12 18.38 error: metadata-generation-failed
#12 18.38 
#12 18.38 × Encountered error while generating package metadata.
#12 18.38 ╰─> See above for output.
#12 18.38 
#12 18.38 note: This is an issue with the package mentioned above, not pip.
#12 18.38 hint: See above for details.
#12 18.40 
#12 18.40 [notice] A new release of pip available: 22.3.1 -> 23.0.1
#12 18.40 [notice] To update, run: pip install --upgrade pip
------
executor failed running [/bin/sh -c pip install .]: exit code: 1

To Reproduce

docker build -t gcp_scan .

I am willing to fix it and open a PR :)

peb-peb commented 1 year ago

I was not able to reproduce this on my machine, i.e., Ubuntu 22.04.2 LTS

My output :arrow_down: image

grumpyp commented 1 year ago

I am on a Macbook with a M1 chip. Someone with a Mac M1 can confirm? In general we should find a universal solution, thats why I suggest using python:3.10-bullseye

ZetaTwo commented 1 year ago

Can you try with:

apt-get install -y build-essential python-dev
grumpyp commented 1 year ago

Can you try with:

apt-get install -y build-essential python-dev

Good morning, same endless wheel unfortunatly.

patrickgerard ~/Documents/GitHub/gcp_scanner [main] $ docker build -t t_gcp .
[+] Building 128.7s (12/13)                                                                            
 => [internal] load build definition from Dockerfile                                              0.0s
 => => transferring dockerfile: 915B                                                              0.0s
 => [internal] load .dockerignore                                                                 0.0s
 => => transferring context: 2B                                                                   0.0s
 => [internal] load metadata for docker.io/library/python:3-slim-buster                           0.6s
 => [1/9] FROM docker.io/library/python:3-slim-buster@sha256:6cc3e51b24c91809e9f7abf8391c8fc56c1  0.0s
 => [internal] load build context                                                                 0.1s
 => => transferring context: 546B                                                                 0.0s
 => CACHED [2/9] RUN mkdir /home/sa_scanner                                                       0.0s
 => CACHED [3/9] COPY src/ /home/sa_scanner/                                                      0.0s
 => CACHED [4/9] COPY pyproject.toml /home/sa_scanner/                                            0.0s
 => CACHED [5/9] COPY README.md /home/sa_scanner                                                  0.0s
 => CACHED [6/9] WORKDIR /home/sa_scanner                                                         0.0s
 => [7/9] RUN apt-get update                                                                      2.6s
 => [8/9] RUN apt-get install -y build-essential python-dev                                      23.0s
 => [9/9] RUN pip install .                                                                     102.3s
 => => #   Created wheel for pyu2f: filename=pyu2f-0.1.5-py3-none-any.whl size=39401 sha256=5453b996b4
 => => # 0ec90179ba804ad4167f63626cbc13dfab9d3f9c7edcc433573507                                       
 => => #   Stored in directory: /root/.cache/pip/wheels/a0/84/f9/1676c313d1bf6c4c30f783a7c5b18648622d5
 => => # f1b2f4caf468a                                                                                
 => => #   Building wheel for grpcio (setup.py): started                                              
 => => #   Building wheel for grpcio (setup.py): still running...

What's your thoughts about my suggestion using python:3.10-bullseye ?

ZetaTwo commented 1 year ago

When you say endless, how long do you mean? If it is compiling some C++ libraries that could take a while.

grumpyp commented 1 year ago

When you say endless, how long do you mean? If it is compiling some C++ libraries that could take a while.

A bit more than 20 mins. But it works.

patrickgerard ~/Documents/GitHub/gcp_scanner [main] $ docker build -t t_gcp .
[+] Building 1293.5s (15/15) FINISHED                                                                  
 => [internal] load build definition from Dockerfile                                              0.1s
 => => transferring dockerfile: 37B                                                               0.1s
 => [internal] load .dockerignore                                                                 0.0s
 => => transferring context: 2B                                                                   0.0s
 => [internal] load metadata for docker.io/library/python:3-slim-buster                           7.1s
 => [auth] library/python:pull token for registry-1.docker.io                                     0.0s
 => [1/9] FROM docker.io/library/python:3-slim-buster@sha256:6d4b1a970e49fc2c37b4ad7b71ea4396a75  0.0s
 => => resolve docker.io/library/python:3-slim-buster@sha256:6d4b1a970e49fc2c37b4ad7b71ea4396a75  0.0s
 => [internal] load build context                                                                 0.2s
 => => transferring context: 546B                                                                 0.0s
 => CACHED [2/9] RUN mkdir /home/sa_scanner                                                       0.0s
 => CACHED [3/9] COPY src/ /home/sa_scanner/                                                      0.0s
 => CACHED [4/9] COPY pyproject.toml /home/sa_scanner/                                            0.0s
 => CACHED [5/9] COPY README.md /home/sa_scanner                                                  0.0s
 => CACHED [6/9] WORKDIR /home/sa_scanner                                                         0.0s
 => CACHED [7/9] RUN apt-get update                                                               0.0s
 => CACHED [8/9] RUN apt-get install -y build-essential python-dev                                0.0s
 => [9/9] RUN pip install .                                                                    1282.0s
 => exporting to image                                                                            4.1s 
 => => exporting layers                                                                           4.0s 
 => => writing image sha256:2348299ecb8c6f3b08a551bf75f85722de45295c3cd12c642c3d72d9d05d942d      0.0s 
 => => naming to docker.io/library/t_gcp                                                          0.0s

Would you rather merge this, or the version I suggested? Personally said, I would cancel the run after a few min if I see it's just a Python dependency. But it works apperently.

Happy to hear any other opinions you have on this matter :)

grumpyp commented 1 year ago

Hi, I'd like to get this one done. How would you like to proceed @ZetaTwo ?

ZetaTwo commented 1 year ago

Basically, I'm not at all a fan of locking the version like this but I also agree that waiting 20 min for a build is very annoying. Do we know what the root cause is of this working for .10 but not .11? Is it because they have a binary distribution for the former but not the latter? I could envision something like locking to .10 for now but then at the same time open an issue to re-evaluate this after X amount of time once .11 works as well as .10.

grumpyp commented 1 year ago

I havn't identified the root cause as of a specific reason why it's not working but it's related to https://pypi.org/project/grpcio/.

I'll open a ticket to re-evaluation like you said. Good idea. Thanks for the head-up.

Martin-Narvar commented 7 months ago

I am experiencing this issue with Python 3.8.18 on Macbook with M1 chip. CircleCI builds this image just fine, so its likely related to Apple Silicon. Reading through this thread, I don't see any clear indication of what I need to do to fix this.