search

apache / datasketches-postgresql

PostgreSQL extension providing approximate algorithms based on apache/datasketches-cpp
https://datasketches.apache.org
Apache License 2.0
85 stars 11 forks source link

Error on Building Dockerfile for Latest Version of PG #52

Closed meysampg closed 2 years ago

meysampg commented 2 years ago

I tried to run the default build command for dockerfile and got this error:

$ docker build . -t datasketch-postgres:latest
Sending build context to Docker daemon  678.4kB
Step 1/14 : ARG BASE_IMAGE_VERSION=latest
Step 2/14 : ARG DATASKETCHES_CPP_HASH=8135b65408947694e13bd131038889e439847aa2
Step 3/14 : ARG DATASKETCHES_CPP_VERSION=3.4.0
Step 4/14 : FROM postgres:$BASE_IMAGE_VERSION
 ---> 07e2ee723e2d
Step 5/14 : MAINTAINER dev@datasketches.apache.org
 ---> Running in 158d8e079fc1
Removing intermediate container 158d8e079fc1
 ---> 44571082f535
Step 6/14 : ENV APACHE_DIST_URLS   https://www.apache.org/dyn/closer.cgi?action=download&filename=   https://www-us.apache.org/dist/   https://www.apache.org/dist/   https://archive.apache.org/dist/
 ---> Running in 47d3ee604c7a
Removing intermediate container 47d3ee604c7a
 ---> 2a4c3b7fa8e2
Step 7/14 : ARG DATASKETCHES_CPP_VERSION
 ---> Running in dedbbd844e8a
Removing intermediate container dedbbd844e8a
 ---> ff9e9408651f
Step 8/14 : ARG DATASKETCHES_CPP_HASH
 ---> Running in fd8993d2c672
Removing intermediate container fd8993d2c672
 ---> 1afa1a4b7d50
Step 9/14 : ENV DS_CPP_VER=$DATASKETCHES_CPP_VERSION
 ---> Running in 099974455ead
Removing intermediate container 099974455ead
 ---> 679e257e74fc
Step 10/14 : ENV DS_CPP_HASH=$DATASKETCHES_CPP_HASH
 ---> Running in 2110648e9155
Removing intermediate container 2110648e9155
 ---> ab685f05fbf7
Step 11/14 : ADD . /datasketches-postgresql
 ---> db324f3359f0
Step 12/14 : WORKDIR /datasketches-postgresql
 ---> Running in 30f6e4f47244
Removing intermediate container 30f6e4f47244
 ---> 61a05ea52c47
Step 13/14 : RUN echo "===> Adding prerequisites..."                      &&     export PG_MAJOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-\([0-9.]*\)\/.*/\1/p"`             &&     export PG_MINOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-$PG_MAJOR\/\S*\s\(\S*\)\s.*/\1/p"` &&     apt-get update -y                                        &&     DEBIAN_FRONTEND=noninteractive                                      apt-get install --no-install-recommends --allow-downgrades -y -q                 ca-certificates                                                 build-essential wget unzip                                      libboost-all-dev                                                postgresql-server-dev-$PG_MAJOR=$PG_MINOR                       libpq-dev=$PG_MINOR                          &&             echo "===> Building datasketches..."                     &&     set -eux;                                                       download_bin() {                                                    local f="$1"; shift;                                            local hash="$1"; shift;                                         local distFile="$1"; shift;                                     local success=;                                                 local distUrl=;                                                 for distUrl in $APACHE_DIST_URLS; do                              if wget -nv -O "$f" "$distUrl$distFile"; then                     success=1;                                                      echo "$hash" "*$f" | sha1sum -c -;                              break;                                                        fi;                                                           done;                                                           [ -n "$success" ];                                          }                                                        &&     download_bin "datasketches-cpp.zip" "$DS_CPP_HASH" "datasketches/cpp/$DS_CPP_VER/apache-datasketches-cpp-$DS_CPP_VER-src.zip" &&     unzip datasketches-cpp.zip                               &&     mv apache-datasketches-cpp-$DS_CPP_VER-src datasketches-cpp  &&     make                                                     &&     make install                                             &&             echo "===> Clean up..."                                  &&     apt-mark hold postgresql-$PG_MAJOR postgresql-client-$PG_MAJOR &&     apt-get -y remove --purge --auto-remove                                 ca-certificates                                                 build-essential wget unzip                                      postgresql-server-dev-$PG_MAJOR libpq-dev        &&     apt-get clean                                            &&     rm -rf /datasketches-postgresql /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Running in f3220886e8ab
===> Adding prerequisites...
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg InRelease [91.7 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [160 kB]
Get:5 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:6 http://deb.debian.org/debian bullseye/main amd64 Packages [8,182 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg/main amd64 Packages [249 kB]
Get:8 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2,592 B]
Fetched 8,885 kB in 11s (782 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '14.1-1.pgdg110+1' for 'postgresql-server-dev-14' was not found
E: Version '14.1-1.pgdg110+1' for 'libpq-dev' was not found
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
The command '/bin/sh -c echo "===> Adding prerequisites..."                      &&     export PG_MAJOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-\([0-9.]*\)\/.*/\1/p"`             &&     export PG_MINOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-$PG_MAJOR\/\S*\s\(\S*\)\s.*/\1/p"` &&     apt-get update -y                                        &&     DEBIAN_FRONTEND=noninteractive                                      apt-get install --no-install-recommends --allow-downgrades -y -q                 ca-certificates                                                 build-essential wget unzip                                      libboost-all-dev                                                postgresql-server-dev-$PG_MAJOR=$PG_MINOR                       libpq-dev=$PG_MINOR                          &&             echo "===> Building datasketches..."                     &&     set -eux;                                                       download_bin() {                                                    local f="$1"; shift;                                            local hash="$1"; shift;                                         local distFile="$1"; shift;                                     local success=;                                                 local distUrl=;                                                 for distUrl in $APACHE_DIST_URLS; do                              if wget -nv -O "$f" "$distUrl$distFile"; then                     success=1;                                                      echo "$hash" "*$f" | sha1sum -c -;                              break;                                                        fi;                                                           done;                                                           [ -n "$success" ];                                          }                                                        &&     download_bin "datasketches-cpp.zip" "$DS_CPP_HASH" "datasketches/cpp/$DS_CPP_VER/apache-datasketches-cpp-$DS_CPP_VER-src.zip" &&     unzip datasketches-cpp.zip                               &&     mv apache-datasketches-cpp-$DS_CPP_VER-src datasketches-cpp  &&     make                                                     &&     make install                                             &&             echo "===> Clean up..."                                  &&     apt-mark hold postgresql-$PG_MAJOR postgresql-client-$PG_MAJOR &&     apt-get -y remove --purge --auto-remove                                 ca-certificates                                                 build-essential wget unzip                                      postgresql-server-dev-$PG_MAJOR libpq-dev        &&     apt-get clean                                            &&     rm -rf /datasketches-postgresql /var/lib/apt/lists/* /tmp/* /var/tmp/*' returned a non-zero code: 1

Based on the log, I searched for libpq-dev on Debian11 (bullseye based on the docker build log) and found that the present version for this library (and probably for postgresql-server-dev-14) is 13, although the latest version of the base image is 14.

So I changed the command to use version 13 of PG and got this error :face_with_head_bandage: 

$ docker build --pull --build-arg BASE_IMAGE_VERSION=13 -t datasketch-postgres:13 .
Sending build context to Docker daemon  678.4kB
Step 1/14 : ARG BASE_IMAGE_VERSION=latest
Step 2/14 : ARG DATASKETCHES_CPP_HASH=8135b65408947694e13bd131038889e439847aa2
Step 3/14 : ARG DATASKETCHES_CPP_VERSION=3.4.0
Step 4/14 : FROM postgres:$BASE_IMAGE_VERSION
13: Pulling from library/postgres
42c077c10790: Pull complete 
3c2843bc3122: Pull complete 
12e1d6a2dd60: Pull complete 
9ae1101c4068: Pull complete 
fb05d2fd4701: Pull complete 
9785a964a677: Pull complete 
16fc798b0e72: Pull complete 
f1a0bfa2327a: Pull complete 
9b94c51b86e5: Pull complete 
281df17c53d0: Pull complete 
86476c34c389: Pull complete 
d8bab0cdd17f: Pull complete 
186eed38aeb6: Pull complete 
Digest: sha256:a598b8679658e155020725d41601813cc8813429494514501e7f65f031774964
Status: Downloaded newer image for postgres:13
 ---> bb3dc2277987
Step 5/14 : MAINTAINER dev@datasketches.apache.org
 ---> Running in 5edaafe8d0a1
Removing intermediate container 5edaafe8d0a1
 ---> 985e7d9927a2
Step 6/14 : ENV APACHE_DIST_URLS   https://www.apache.org/dyn/closer.cgi?action=download&filename=   https://www-us.apache.org/dist/   https://www.apache.org/dist/   https://archive.apache.org/dist/
 ---> Running in 0510521d03e8
Removing intermediate container 0510521d03e8
 ---> 3da018f37781
Step 7/14 : ARG DATASKETCHES_CPP_VERSION
 ---> Running in fe776c07f5b2
Removing intermediate container fe776c07f5b2
 ---> 58e28daa4c1c
Step 8/14 : ARG DATASKETCHES_CPP_HASH
 ---> Running in 9380b94cc7d7
Removing intermediate container 9380b94cc7d7
 ---> c0576af686f2
Step 9/14 : ENV DS_CPP_VER=$DATASKETCHES_CPP_VERSION
 ---> Running in 3522b9625c73
Removing intermediate container 3522b9625c73
 ---> 594070cebb1b
Step 10/14 : ENV DS_CPP_HASH=$DATASKETCHES_CPP_HASH
 ---> Running in cc0767a9012b
Removing intermediate container cc0767a9012b
 ---> f1d468eda982
Step 11/14 : ADD . /datasketches-postgresql
 ---> e3a80b81a5da
Step 12/14 : WORKDIR /datasketches-postgresql
 ---> Running in 626b61f869ff
Removing intermediate container 626b61f869ff
 ---> f667f5d49738
Step 13/14 : RUN echo "===> Adding prerequisites..."                      &&     export PG_MAJOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-\([0-9.]*\)\/.*/\1/p"`             &&     export PG_MINOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-$PG_MAJOR\/\S*\s\(\S*\)\s.*/\1/p"` &&     apt-get update -y                                        &&     DEBIAN_FRONTEND=noninteractive                                      apt-get install --no-install-recommends --allow-downgrades -y -q                 ca-certificates                                                 build-essential wget unzip                                      libboost-all-dev                                                postgresql-server-dev-$PG_MAJOR=$PG_MINOR                       libpq-dev=$PG_MINOR                          &&             echo "===> Building datasketches..."                     &&     set -eux;                                                       download_bin() {                                                    local f="$1"; shift;                                            local hash="$1"; shift;                                         local distFile="$1"; shift;                                     local success=;                                                 local distUrl=;                                                 for distUrl in $APACHE_DIST_URLS; do                              if wget -nv -O "$f" "$distUrl$distFile"; then                     success=1;                                                      echo "$hash" "*$f" | sha1sum -c -;                              break;                                                        fi;                                                           done;                                                           [ -n "$success" ];                                          }                                                        &&     download_bin "datasketches-cpp.zip" "$DS_CPP_HASH" "datasketches/cpp/$DS_CPP_VER/apache-datasketches-cpp-$DS_CPP_VER-src.zip" &&     unzip datasketches-cpp.zip                               &&     mv apache-datasketches-cpp-$DS_CPP_VER-src datasketches-cpp  &&     make                                                     &&     make install                                             &&             echo "===> Clean up..."                                  &&     apt-mark hold postgresql-$PG_MAJOR postgresql-client-$PG_MAJOR &&     apt-get -y remove --purge --auto-remove                                 ca-certificates                                                 build-essential wget unzip                                      postgresql-server-dev-$PG_MAJOR libpq-dev        &&     apt-get clean                                            &&     rm -rf /datasketches-postgresql /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Running in a8339b8d38bb
===> Adding prerequisites...
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:3 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg InRelease [91.7 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [160 kB]
Get:5 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg/main amd64 Packages [249 kB]
Get:6 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:7 http://apt.postgresql.org/pub/repos/apt bullseye-pgdg/13 amd64 Packages [2,575 B]
Get:8 http://deb.debian.org/debian bullseye/main amd64 Packages [8,182 kB]
Get:9 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2,592 B]
Fetched 8,887 kB in 16s (564 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libpq-dev : Depends: libpq5 (= 13.7-1.pgdg110+1) but 14.4-1.pgdg110+1 is to be installed
E: Unable to correct problems, you have held broken packages.
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
The command '/bin/sh -c echo "===> Adding prerequisites..."                      &&     export PG_MAJOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-\([0-9.]*\)\/.*/\1/p"`             &&     export PG_MINOR=`apt list --installed 2>&1 | sed -n "s/^postgresql-$PG_MAJOR\/\S*\s\(\S*\)\s.*/\1/p"` &&     apt-get update -y                                        &&     DEBIAN_FRONTEND=noninteractive                                      apt-get install --no-install-recommends --allow-downgrades -y -q                 ca-certificates                                                 build-essential wget unzip                                      libboost-all-dev                                                postgresql-server-dev-$PG_MAJOR=$PG_MINOR                       libpq-dev=$PG_MINOR                          &&             echo "===> Building datasketches..."                     &&     set -eux;                                                       download_bin() {                                                    local f="$1"; shift;                                            local hash="$1"; shift;                                         local distFile="$1"; shift;                                     local success=;                                                 local distUrl=;                                                 for distUrl in $APACHE_DIST_URLS; do                              if wget -nv -O "$f" "$distUrl$distFile"; then                     success=1;                                                      echo "$hash" "*$f" | sha1sum -c -;                              break;                                                        fi;                                                           done;                                                           [ -n "$success" ];                                          }                                                        &&     download_bin "datasketches-cpp.zip" "$DS_CPP_HASH" "datasketches/cpp/$DS_CPP_VER/apache-datasketches-cpp-$DS_CPP_VER-src.zip" &&     unzip datasketches-cpp.zip                               &&     mv apache-datasketches-cpp-$DS_CPP_VER-src datasketches-cpp  &&     make                                                     &&     make install                                             &&             echo "===> Clean up..."                                  &&     apt-mark hold postgresql-$PG_MAJOR postgresql-client-$PG_MAJOR &&     apt-get -y remove --purge --auto-remove                                 ca-certificates                                                 build-essential wget unzip                                      postgresql-server-dev-$PG_MAJOR libpq-dev        &&     apt-get clean                                            &&     rm -rf /datasketches-postgresql /var/lib/apt/lists/* /tmp/* /var/tmp/*' returned a non-zero code: 1

Is there any missing part in the building process?

Ps. My operating system information:

$ screenfetch             
                          ./+o+-       meysam@smartech
                  yyyyy- -yyyyyy+      OS: Ubuntu 22.04 jammy
               ://+//////-yyyyyyo      Kernel: x86_64 Linux 5.15.0-27-generic
           .++ .:/++++++/-.+sss/`      Uptime: 45d 20m
         .:++o:  /++++++++/:--:/-      Packages: 2219
        o:+o+:++.`..```.-/oo+++++/     Shell: zsh 5.8.1
       .:+o:+o/.          `+sssoo+/    Resolution: 1920x2160
  .++/+:+oo+o:`             /sssooo.   DE: GNOME 41.4
 /+++//+:`oo+o               /::--:.   WM: Mutter
 \+/+o+++`o++o               ++////.   WM Theme: Adwaita
  .++.o+++oo+:`             /dddhhh.   GTK Theme: Yaru-viridian-dark [GTK2/3]
       .+.o+oo:.          `oddhhhh+    Icon Theme: Yaru-viridian
        \+.++o+o``-````.:ohdhhhhh+     Font: Ubuntu 11
         `:o+++ `ohhhhhhhhyo++os:      Disk: 298G / 916G (35%)
           .o:`.syhhhhhhh/.oo++o`      CPU: 11th Gen Intel Core i7-1165G7 @ 8x 4.7GHz [80.0°C]
               /osyyyyyyo++ooo+++/     GPU: Iris Xe Graphics
                   ````` +oo+++o\:     RAM: 13835MiB / 15698MiB
                          `oo++.                                    

$ docker version                                                                   
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:15 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
meysampg commented 2 years ago

Update: Removing minor version =$PG_MINOR from the docker file can solve the problem.

$ gd

Δ Dockerfile
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

────────────────────────────────────────────────────────────────────────┐
• 49: RUN echo "===> Adding prerequisites..."                      && \ │
────────────────────────────────────────────────────────────────────────┘
│ 49 │                ca-certificates                                 \                │ 49 │                ca-certificates                                 \
│ 50 │                build-essential wget unzip                      \                │ 50 │                build-essential wget unzip                      \
│ 51 │                libboost-all-dev                                \                │ 51 │                libboost-all-dev                                \
│ 52 │                postgresql-server-dev-$PG_MAJOR=$PG_MINOR       \                │ 52 │                postgresql-server-dev-$PG_MAJOR       \
│ 53 │                libpq-dev=$PG_MINOR                          && \                │ 53 │                libpq-dev                          && \
│ 54 │    \                                                                            │ 54 │    \
│ 55 │    \                                                                            │ 55 │    \                
│ 56 │    echo "===> Building datasketches..."                     && \                │ 56 │    echo "===> Building datasketches..."                     && \
AlexanderSaydakov commented 2 years ago

Your proposed fix seems to work. I am not sure whether this way of setting up versions is adequate. Perhaps it would be better to run "pg_config --version" instead of parsing the output from "apt list".