search

postgis / docker-postgis

Docker image for PostGIS
https://hub.docker.com/r/postgis/postgis/
MIT License
1.37k stars 462 forks source link

#180 multistage build on master branch #211

Closed smellman closed 3 years ago

smellman commented 3 years ago
smellman commented 3 years ago

@ImreSamu Do you check current source code? Now it will be fail in postgis by upstream both my branch and current master branch. But now, my branch enable to build each modules(if upgrade gdal, it may build gdal only).

ImreSamu commented 3 years ago

@smellman :

Now it will be fail in postgis by upstream both my branch and current master branch.

I see

Run tests: 143
Failed: 1
make: *** [runtest.mk:10: check-regress] Error 1

I will debug in the next few days ..

ImreSamu commented 3 years ago

@smellman

(based on the travis log )

imho: The postgis git hash is very strange.

maybe the git reset --hard FETCH_HEAD .. setting always for the "origin" HEAD ?

+ git init
Initialized empty Git repository in /usr/src/postgis/.git/
+ git remote add origin https://git.osgeo.org/gitea/postgis/postgis.git
+ git fetch --depth 1 origin :a7ff53ff535eae8579a374a11a785ed053acd3c7
From https://git.osgeo.org/gitea/postgis/postgis
 * [new ref]                    -> a7ff53ff535eae8579a374a11a785ed053acd3c7
+ git reset --hard FETCH_HEAD
HEAD is now at 6b5aaea Have berrie bots prepend PG path to PATH

imho ( temporary ) :

smellman commented 3 years ago

maybe the git reset --hard FETCH_HEAD .. setting always for the "origin" HEAD ?

$ git init
$ git remote add origin https://git.osgeo.org/gitea/postgis/postgis.git
$ git fetch --depth 1 origin :13f821e5906e1a2663eb43b410bb6bd1deba88b6
$ git log
fatal: your current branch 'master' does not have any commits yet
$ git reset --hard FETCH_HEAD
HEAD is now at 873d3e5 Do not use abs_path if not strictly needed
$ git log
commit 873d3e5a099dff60614be54c65657eebab184e80 (grafted, HEAD -> master, 13f821e5906e1a2663eb43b410bb6bd1deba88b6)
Author: Sandro Santilli <strk@kbt.io>
Date:   Tue Nov 3 10:56:27 2020 +0100

    Do not use abs_path if not strictly needed

    Hopefully fixes windows builds (winnie).
    See #4781

It only pull specify hashed branch as one commit.

imho ( temporary ) :

* please use the latest ( known built ) hashes https://github.com/postgis/docker-postgis/blob/master/13-master/Dockerfile#L14

I use it but I get same error in build. It seems it raise from geos or other program. I can build it when my first commit so I will check histories.

smellman commented 3 years ago

Sorry I find the bug in my code.

smellman commented 3 years ago

@ImreSamu I try to build current 12-master.

Run tests: 141
Failed: 2
make: *** [runtest.mk:10: check-regress] Error 2

I think postgres:12 image was changed something.

ImreSamu commented 3 years ago

@smellman

It only pull specify hashed branch as one commit.

imho: Please test with the Postgis 2.0.0 version - and test the head NEWS content; it is different for me.

# Checkout the Postgis 2.0.0 version (  Apr 3, 2012 )
# https://github.com/postgis/postgis/releases/tag/2.0.0
# https://github.com/postgis/postgis/commit/731e89549040556a4d693d3c05c1c8221c055157

export POSTGIS_GIT_HASH=731e89549040556a4d693d3c05c1c8221c055157
mkdir postgis_test_fetch_head
cd postgis_test_fetch_head
git init
git remote add origin https://git.osgeo.org/gitea/postgis/postgis.git
git fetch --depth 1 origin :${POSTGIS_GIT_HASH}
git reset --hard FETCH_HEAD
head  NEWS

# Classic
cd ..
git clone --branch master https://git.osgeo.org/gitea/postgis/postgis.git
cd postgis
git checkout ${POSTGIS_GIT_HASH}
head  NEWS

the first head NEWS contains the ( PostGIS 3.1.0alpha3 2020/xx/xx )

root@e9d0ed84e34f:/# export POSTGIS_GIT_HASH=731e89549040556a4d693d3c05c1c8221c055157
root@e9d0ed84e34f:/# mkdir postgis_test_fetch_head
root@e9d0ed84e34f:/# cd postgis_test_fetch_head
root@e9d0ed84e34f:/postgis_test_fetch_head# git init
Initialized empty Git repository in /postgis_test_fetch_head/.git/
root@e9d0ed84e34f:/postgis_test_fetch_head# git remote add origin https://git.osgeo.org/gitea/postgis/postgis.git
root@e9d0ed84e34f:/postgis_test_fetch_head# git fetch --depth 1 origin :${POSTGIS_GIT_HASH}
remote: Counting objects: 2290, done.
remote: Compressing objects: 100% (1957/1957), done.
remote: Total 2290 (delta 454), reused 1454 (delta 242)
Receiving objects: 100% (2290/2290), 12.81 MiB | 772.00 KiB/s, done.
Resolving deltas: 100% (454/454), done.
From https://git.osgeo.org/gitea/postgis/postgis
 * [new ref]                    -> 731e89549040556a4d693d3c05c1c8221c055157
root@e9d0ed84e34f:/postgis_test_fetch_head# git reset --hard FETCH_HEAD
HEAD is now at 9732cf8 Avoid encoding dependencies from subdirs
root@e9d0ed84e34f:/postgis_test_fetch_head# head  NEWS
PostGIS 3.1.0alpha3
2020/xx/xx
Only tickets not included in 3.1.0alpha2

* Breaking changes *
  - #4737, Bump minimum protobuf-c requirement to 1.1.0 (Raúl Marín)
           The configure step will now fail if the requirement isn't
           met or explicitly disabled (--without-protobuf)
  - #4258, Untangle postgis_sfcgal into its own .so (Regina Obe)

expected as in the second classic version

HEAD is now at 731e89549 Update versions for 2.0.0 tag
root@e9d0ed84e34f:/postgis# head  NEWS
PostGIS 2.0.0
2012/04/03

* Important / Breaking Changes  *

  - Upgrading to PostGIS 2.0 REQUIRES a dump and restore. See the
    "Hard Upgrade" section of the documentation.
  - Functions deprecated during the 1.X series (mostly non-ST variants)
    have been removed. Update your application to the new signatures.
    If you cannot update your application, see the legacy.sql file
ImreSamu commented 3 years ago

@smellman

I think postgres:12 image was changed something.

If you modify for the "classic" git clone + git checkout it should work.

diff --git a/12-master/Dockerfile b/12-master/Dockerfile
index 024aa7e..d4cce58 100644
--- a/12-master/Dockerfile
+++ b/12-master/Dockerfile
@@ -65,39 +65,34 @@ RUN set -ex \
       protobuf-c-compiler \
       xsltproc \
     # sfcgal
-    && mkdir -p /usr/src/sfcgal \
-    && cd /usr/src/sfcgal \
-    && git init \
-    && git remote add origin https://gitlab.com/Oslandia/SFCGAL.git \
-    && git fetch --depth 1 origin :${SFCGAL_GIT_HASH} \
-    && git reset --hard FETCH_HEAD \
+    && mkdir -p /usr/src/ \
+    && cd /usr/src/ \
+    && git clone https://gitlab.com/Oslandia/SFCGAL.git \
+    && cd SFCGAL \
+    && git checkout ${SFCGAL_GIT_HASH} \
     && mkdir cmake-build \
     && cd cmake-build \
     && cmake .. \
     && make -j$(nproc) \
     && make install \
     && cd / \
-    && rm -fr /usr/src/sfcgal \
+    && rm -fr /usr/src/SFCGAL \
     # proj4
-    && mkdir -p /usr/src/proj \
-    && cd /usr/src/proj \
-    && git init \
-    && git remote add origin https://github.com/OSGeo/PROJ.git \
-    && git fetch --depth 1 origin :${PROJ_GIT_HASH} \
-    && git reset --hard FETCH_HEAD \
+    && cd /usr/src/ \
+    && git clone https://github.com/OSGeo/PROJ.git \
+    && cd PROJ \
+    && git checkout ${PROJ_GIT_HASH} \
     && ./autogen.sh \
     && ./configure --disable-static \
     && make -j$(nproc) \
     && make install \
     && cd / \
-    && rm -fr /usr/src/proj \
+    && rm -fr /usr/src/PROJ \
     # geos
-    && mkdir -p /usr/src/geos \
-    && cd /usr/src/geos \
-    && git init \
-    && git remote add origin https://github.com/libgeos/geos.git \
-    && git fetch --depth 1 origin :${GEOS_GIT_HASH} \
-    && git reset --hard FETCH_HEAD \
+    && cd /usr/src/ \
+    && git clone https://github.com/libgeos/geos.git \
+    && cd geos \
+    && git checkout ${GEOS_GIT_HASH} \
     && ./autogen.sh \
     && ./configure --disable-static \
     && make -j$(nproc) \
@@ -105,12 +100,10 @@ RUN set -ex \
     && cd / \
     && rm -fr /usr/src/geos \
     # gdal
-    && mkdir -p /usr/src/gdal \
-    && cd /usr/src/gdal \
-    && git init \
-    && git remote add origin https://github.com/OSGeo/gdal.git \
-    && git fetch --depth 1 origin :${GDAL_GIT_HASH} \
-    && git reset --hard FETCH_HEAD \
+    && cd /usr/src/ \
+    && git clone https://github.com/OSGeo/gdal.git \
+    && cd gdal \
+    && git checkout ${GDAL_GIT_HASH} \
     && cd gdal \
     && ./autogen.sh \
     && ./configure --disable-static \
@@ -119,12 +112,10 @@ RUN set -ex \
     && cd / \
     && rm -fr /usr/src/gdal \
     # postgis
-    && mkdir -p /usr/src/postgis \
-    && cd /usr/src/postgis \
-    && git init \
-    && git remote add origin https://git.osgeo.org/gitea/postgis/postgis.git \
-    && git fetch --depth 1 origin :${POSTGIS_GIT_HASH} \
-    && git reset --hard FETCH_HEAD \
+    && cd /usr/src/ \
+    && git clone https://git.osgeo.org/gitea/postgis/postgis.git \
+    && cd postgis \
+    && git checkout ${POSTGIS_GIT_HASH} \
     && ./autogen.sh \
 # configure options taken from:
 # https://anonscm.debian.org/cgit/pkg-grass/postgis.git/tree/debian/rules?h=jessie
smellman commented 3 years ago

@ImreSamu I tries it. Also, I get current errors.

root@f8b2ad8223aa:/tmp/pgis_reg# cat test_103_diff
--- tickets_expected    2020-11-04 04:17:44.308731143 +0000
+++ /tmp/pgis_reg/test_103_out  2020-11-04 04:24:59.908413043 +0000
@@ -195,7 +195,7 @@
 #1543|MULTILINESTRING((0 0,10 0,10 10,0 0),(0 0))|POLYGON((0 0,10 10,10 0,0 0))
 #1578|f|f
 #1580.1|Point[S]
-ERROR:  transform: tolerance condition error (-20)
+#1580.2|0101000020430D000093107C45F81B7341B97937FF14E8AC41
 #1580.3|Point[S]
 #1596.1|public.road_pg.roads_geom SRID:3395 TYPE:POINT DIMS:2
 ERROR:  invalid SRID: 330000 not found in spatial_ref_sys
root@f8b2ad8223aa:/tmp/pgis_reg# cat test_120_diff
--- subdivide_expected  2020-11-04 04:17:44.241731143 +0000
+++ /tmp/pgis_reg/test_120_out  2020-11-04 04:25:45.722958155 +0000
@@ -4,7 +4,7 @@
 ERROR:  lwgeom_subdivide_prec: cannot subdivide to fewer than 5 vertices per output
 #3522|POINT(1 1)
 #3744|1600000000000000
-4|29321996468.6|29321996468.6|1857|256
+4|29321996468.6|29321996468.6|1856|256
 #4211|0.00008316000
 #4217|0.00002463668
 #4301|0.00213614552
smellman commented 3 years ago

@ImreSamu I test with old master branches of proj, geos and gdal.

PostGIS current master branch can build if use old branches of proj and geos. I think pgis_reg/test_103 issue come from proj, and pgis_reg/test_120 issue come from geos. Also building gdal current master branch may fail with proj and geos old branch.

smellman commented 3 years ago

GEOS issue may fixed: https://github.com/postgis/postgis/commit/4ec4fbc15c9b867f980cbc2c4d4d430f18637cb4

smellman commented 3 years ago

I write PROJ issue: https://trac.osgeo.org/postgis/ticket/4785

ImreSamu commented 3 years ago

@smellman :

PostGIS current master branch can build if use old branches of proj and geos.

it is good news !

GEOS issue may fixed I write PROJ issue:

and thanks for searching the root cause

My question: As I understand the "original issue" was about creating a smaller docker image. Do you have an early result about the new docker image size?

smellman commented 3 years ago

@ImreSamu 

postgis13-master-test      latest              9b45e60ba355        18 hours ago        850MB
postgis/postgis            13-master           cbf3e2382b1f        7 days ago          854MB

It only 4MB smaller than current posgis-master.

But I think it is useful for debug to research root cause. Now I find gdal build fail with old proj.

smellman commented 3 years ago

GEOS issue may fixed: postgis/postgis@4ec4fbc

I checked postgis current master was fixed this. So, when it fixed proj issue, I will try build again.

smellman commented 3 years ago

@ImreSamu Proj issue was fixed https://github.com/postgis/postgis/commit/5b2e3cbd52408a9870a9886e5896123e4ff86118 I can build both 12-master and 13-master now.

ImreSamu commented 3 years ago

@smellman : Thank you! :+1: All my critical and not critical comments has been resolved! I have re-tested locally ( 12-master , 13-master ) and looks OK!

@phillipross :

phillipross commented 3 years ago

@smellman @ImreSamu Thanks for your work on this! I'll take a look.