search

PacktPublishing / Data-Engineering-with-Databricks-Cookbook

Data Engineering with Databricks Cookbook, published by Packt
MIT License
46 stars 57 forks source link

Unable to build the docker set up #23

Closed Iravata closed 3 months ago

Iravata commented 4 months ago

build.sh dosent complete successfully. Netcat has no installation candidate

+] Building 54.8s (5/7)                                                                                                                                                                     docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 1.47kB                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/eclipse-temurin:17-jre                                                                                                                                    1.0s
 => [internal] load .dockerignore                                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => CACHED [1/4] FROM docker.io/library/eclipse-temurin:17-jre@sha256:34cc39fcd17383dfbe9b1e1ff29efb89c770913698be71db32e7e4be25bce2e0                                                                       0.0s
 => ERROR [2/4] RUN mkdir -p /opt/workspace/data &&     mkdir -p /usr/share/man/man1  &&     apt-get update -y &&     apt-get install -y --no-install-recommends curl python3 r-base netcat &&     ln -s /  53.8s
------
 > [2/4] RUN mkdir -p /opt/workspace/data &&     mkdir -p /usr/share/man/man1  &&     apt-get update -y &&     apt-get install -y --no-install-recommends curl python3 r-base netcat &&     ln -s /usr/bin/python3 /usr/bin/python &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*:
0.425 Get:1 http://ports.ubuntu.com/ubuntu-ports noble InRelease [256 kB]
1.387 Get:2 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease [126 kB]
1.620 Get:3 http://ports.ubuntu.com/ubuntu-ports noble-backports InRelease [126 kB]
1.855 Get:4 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease [126 kB]
2.088 Get:5 http://ports.ubuntu.com/ubuntu-ports noble/multiverse arm64 Packages [274 kB]
2.495 Get:6 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages [1,776 kB]
6.316 Get:7 http://ports.ubuntu.com/ubuntu-ports noble/restricted arm64 Packages [113 kB]
6.626 Get:8 http://ports.ubuntu.com/ubuntu-ports noble/universe arm64 Packages [19.0 MB]
48.05 Get:9 http://ports.ubuntu.com/ubuntu-ports noble-updates/restricted arm64 Packages [250 kB]
48.85 Get:10 http://ports.ubuntu.com/ubuntu-ports noble-updates/multiverse arm64 Packages [12.2 kB]
48.86 Get:11 http://ports.ubuntu.com/ubuntu-ports noble-updates/universe arm64 Packages [397 kB]
50.04 Get:12 http://ports.ubuntu.com/ubuntu-ports noble-updates/main arm64 Packages [363 kB]
51.24 Get:13 http://ports.ubuntu.com/ubuntu-ports noble-backports/universe arm64 Packages [11.5 kB]
51.25 Get:14 http://ports.ubuntu.com/ubuntu-ports noble-security/restricted arm64 Packages [250 kB]
51.65 Get:15 http://ports.ubuntu.com/ubuntu-ports noble-security/main arm64 Packages [307 kB]
52.34 Get:16 http://ports.ubuntu.com/ubuntu-ports noble-security/universe arm64 Packages [313 kB]
53.04 Get:17 http://ports.ubuntu.com/ubuntu-ports noble-security/multiverse arm64 Packages [12.2 kB]
53.06 Fetched 23.7 MB in 53s (448 kB/s)
53.06 Reading package lists...
53.41 Reading package lists...
53.73 Building dependency tree...
53.80 Reading state information...
53.80 Package netcat is a virtual package provided by:
53.80   netcat-traditional 1.10-48
53.80   netcat-openbsd 1.226-1ubuntu2
53.80
53.81 E: Package 'netcat' has no installation candidate
------
Dockerfile:19
--------------------
  18 |
  19 | >>> RUN mkdir -p ${shared_workspace}/data && \
  20 | >>>     mkdir -p /usr/share/man/man1  && \
  21 | >>>     apt-get update -y && \
  22 | >>>     apt-get install -y --no-install-recommends curl python3 r-base netcat && \
  23 | >>>     ln -s /usr/bin/python3 /usr/bin/python && \
  24 | >>>     apt-get clean && \
  25 | >>>     rm -rf /var/lib/apt/lists/*
  26 |
--------------------
ERROR: failed to solve: process "/bin/sh -c mkdir -p ${shared_workspace}/data &&     mkdir -p /usr/share/man/man1  &&     apt-get update -y &&     apt-get install -y --no-install-recommends curl python3 r-base netcat &&     ln -s /usr/bin/python3 /usr/bin/python &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/ae6g5iz7edmgy07auwyjgahgi
[+] Building 2.1s (3/3) FINISHED                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 1.98kB                                                                                                                                                                       0.0s
 => ERROR [internal] load metadata for docker.io/library/base:latest                                                                                                                                         2.1s
 => [auth] library/base:pull token for registry-1.docker.io                                                                                                                                                  0.0s
------
 > [internal] load metadata for docker.io/library/base:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM base
   2 |
   3 |     # -- Layer: Image Metadata
--------------------
ERROR: failed to solve: base: failed to resolve source metadata for docker.io/library/base:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/8coutzjy1lztg7mk6shmkp8ca
[+] Building 2.4s (3/3) FINISHED                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 436B                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for docker.io/library/spark-base:3.4.1                                                                                                                                    2.4s
 => [auth] library/spark-base:pull token for registry-1.docker.io                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/spark-base:3.4.1:
------
Dockerfile:2
--------------------
   1 |     ARG spark_version
   2 | >>> FROM spark-base:${spark_version}
   3 |
   4 |     # -- Layer: Image Metadata
--------------------
ERROR: failed to solve: spark-base:3.4.1: failed to resolve source metadata for docker.io/library/spark-base:3.4.1: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/cj0rlvzsx821lna66z0wetw8e
[+] Building 1.5s (3/3) FINISHED                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 481B                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for docker.io/library/spark-base:3.4.1                                                                                                                                    1.4s
 => [auth] library/spark-base:pull token for registry-1.docker.io                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/spark-base:3.4.1:
------
Dockerfile:2
--------------------
   1 |     ARG spark_version
   2 | >>> FROM spark-base:${spark_version}
   3 |
   4 |     # -- Layer: Image Metadata
--------------------
ERROR: failed to solve: spark-base:3.4.1: failed to resolve source metadata for docker.io/library/spark-base:3.4.1: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/exkhhx2n4tapx8j41f1dvvzzu
[+] Building 1.5s (3/3) FINISHED                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 995B                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for docker.io/library/base:latest                                                                                                                                         1.5s
 => [auth] library/base:pull token for registry-1.docker.io                                                                                                                                                  0.0s
------
 > [internal] load metadata for docker.io/library/base:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM base
   2 |
   3 |     # -- Layer: Image Metadata
--------------------
ERROR: failed to solve: base: failed to resolve source metadata for docker.io/library/base:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/o9clppmvjiqka01banol8qxi5
bigriobill commented 4 months ago

we resolved with two fixes,

1) in base Dockerfile, change netcat to netcat-traditional. Once this is fixed we encountered the second error.

2) We resolved the "externally-managed-environment" error in our Docker builds by adding the --break-system-packages flag to pip3 install commands in both the base Dockerfile and the JupyterLab Dockerfile. This flag allows pip to install packages system-wide, overriding the default behavior in newer Ubuntu versions that use externally managed environments. The fix enables successful building of our Spark and JupyterLab images for development and testing purposes.