Closed Jean-Baptiste-Lasselle closed 6 years ago
docker logs :
[jibl@srv-10 carto-proto]$ docker logs -f rendereurpoulet
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
/assign @Jean-Baptiste-Lasselle :+1:
Du nouveau :
J'ai déterminé que les couillons n'avaient pas défini la valeur de POSTGRES_USER
dans le docker-compose.yml
, si bien qu'aucun utilisateur n'était utilisable pour se connecter à la base de données qui elle était créée par la version couillon (cf. POSTGRES_DB
dans le ./docker-compose.yml
)
Désormais, ce processus osm2pgsql
peut effectivement se connecter avec le user existant, et alors j'obitens un processus batch de téléchargement, de type ETL (extraction, après le téléchargement, il y aura traitement par osm2pgsql
), qui lui, est donc bien long.
J'ai mesuré à la grosse fourchette, qu'au bout de 30 minutes, j'arrive à 20% de la tâche complétée, et avec des logs de la forme :
20% <=> 30 minutes
100 % <=> 5 x 30 min = 2h30min (papa va regarder un film)
Ps Papa:
La forme des logs de l'exécutable osm2pgsql
:
161950K .......... .......... .......... .......... .......... 26% 130K 2h32m
162000K .......... .......... .......... .......... .......... 26% 86.0K 2h32m
162050K .......... .......... .......... .......... .......... 26% 114K 2h32m
162100K .......... .......... .......... .......... .......... 26% 233K 2h32m
162150K .......... .......... .......... .......... .......... 26% 82.9K 2h32m
162200K .......... .......... .......... .......... .......... 26% 37.7K 2h32m
==>> Es-ce que le chiffre de la forme 2h32m
, en fin de chaque ligne, ne serait pas l'évaluation dela durée de la tâche par wget
? Non, c'est la durée restante de processing qui est ainsi affichée par osm2pgsql
, car lorsque l'on consulte les logs en live d'un processus batch, il est courant que l'on ait aucune idée à priori, de l'heure à laquelle a commencé le processus batch.
Oui, typiqument UNIX, comme forme. ;)
Et en effet, après 30 minutes de film commencé :+1:
You know, daddy 's gona have to re-design it from scratch :
osm2pgsql
is installed and excuted in the same container as the database server instance: that is stupid, importing new data in the run, wont be scalable though postgresql releases are definitely rock-solid, and fit for salability.
( bored daddy during adverts in middle of movie)
final logs of docker build process, including osm2pgsql
successful execution :+1:
6100K .......... .......... .......... .......... .......... 89.6M
6150K .......... .......... .......... .......... .......... 87.3M
6200K .......... .......... .......... .......... .......... 89.1M
6250K .......... .......... .......... .......... .......... 73.0M
6300K .......... .......... .......... .......... .......... 91.7M
6350K .......... .......... .......... .......... .......... 95.6M
6400K .......... .......... .......... .......... .......... 77.2M
6450K .......... .......... .......... .......... .......... 65.5M
6500K .......... .......... .......... .......... .......... 58.3M
6550K .......... .......... ....... 62.0M=1.0s
2018-10-21 20:48:02 (6.65 MB/s) - '1.075R-it.tar.gz' saved [6735736]
source-sans-pro-2.020R-ro-1.075R-it/
source-sans-pro-2.020R-ro-1.075R-it/EOT/
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Black.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-BlackIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Bold.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-BoldIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-ExtraLight.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-ExtraLightIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-It.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Light.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-LightIt.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Regular.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-Semibold.eot
source-sans-pro-2.020R-ro-1.075R-it/EOT/SourceSansPro-SemiboldIt.eot
source-sans-pro-2.020R-ro-1.075R-it/LICENSE.txt
source-sans-pro-2.020R-ro-1.075R-it/OTF/
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Black.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-BlackIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Bold.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-BoldIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-ExtraLight.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-ExtraLightIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-It.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Light.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-LightIt.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Regular.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-Semibold.otf
source-sans-pro-2.020R-ro-1.075R-it/OTF/SourceSansPro-SemiboldIt.otf
source-sans-pro-2.020R-ro-1.075R-it/README.md
source-sans-pro-2.020R-ro-1.075R-it/TTF/
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Black.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-BlackIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Bold.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-BoldIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-ExtraLight.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-ExtraLightIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-It.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Light.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-LightIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Regular.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-Semibold.ttf
source-sans-pro-2.020R-ro-1.075R-it/TTF/SourceSansPro-SemiboldIt.ttf
source-sans-pro-2.020R-ro-1.075R-it/WOFF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Black.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-BlackIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Bold.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-BoldIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-ExtraLight.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-ExtraLightIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-It.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Light.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-LightIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Regular.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-Semibold.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/OTF/SourceSansPro-SemiboldIt.otf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Black.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-BlackIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Bold.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-BoldIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-ExtraLight.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-ExtraLightIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-It.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Light.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-LightIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Regular.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-Semibold.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF/TTF/SourceSansPro-SemiboldIt.ttf.woff
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Black.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-BlackIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Bold.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-BoldIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-ExtraLight.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-ExtraLightIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-It.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Light.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-LightIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Regular.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-Semibold.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/OTF/SourceSansPro-SemiboldIt.otf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Black.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-BlackIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Bold.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-BoldIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-ExtraLight.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-ExtraLightIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-It.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Light.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-LightIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Regular.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-Semibold.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/WOFF2/TTF/SourceSansPro-SemiboldIt.ttf.woff2
source-sans-pro-2.020R-ro-1.075R-it/bower.json
source-sans-pro-2.020R-ro-1.075R-it/source-sans-pro.css
Removing intermediate container d911f04124da
---> 397fd637a568
Step 5/17 : RUN git clone https://github.com/dooman87/openstreetmap-carto.git
---> Running in d70e926736aa
Cloning into 'openstreetmap-carto'...
Removing intermediate container d70e926736aa
---> de4c0135bda1
Step 6/17 : WORKDIR /openstreetmap-carto
---> Running in 10593c68c94b
Removing intermediate container 10593c68c94b
---> a43571453916
Step 7/17 : ADD shapes/ data/
---> 79f5b61b1ccb
Step 8/17 : RUN shapeindex --shape_files data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp data/water-polygons-split-3857/water_polygons.shp data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp
---> Running in 882f8548394e
max tree depth:8
split ratio:0.55
processing data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp
9994
length=14943720
version=1000
type=5
extent:box2d(-20037508.3399999998509884,-14773407.4192944951355457,20037508.3399999998509884,20037508.3399999998509884)
number shapes=5121
number nodes=2468
processing data/water-polygons-split-3857/water_polygons.shp
9994
length=378892596
version=1000
type=5
extent:box2d(-20037508.3427892439067364,-14773407.4192944895476103,20037508.3427892439067364,20037508.3427892439067364)
number shapes=42337
number nodes=2131
processing data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp
9994
length=34678596
version=1000
type=5
extent:box2d(-20037508.3427892178297043,-20037508.3427892550826073,20037507.2295943312346935,-8479237.0339447762817144)
number shapes=14825
number nodes=952
processing data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp
9994
length=36341768
version=1000
type=3
extent:box2d(-20037508.3427892439067364,-20037508.3427892550826073,20037508.3427892439067364,-8479237.0339447762817144)
number shapes=78810
number nodes=1904
processing data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp
9994
length=27922
version=1000
type=3
extent:box2d(-140.9977800000000059,-54.8968100000000021,141.0338517600138175,70.1641900000000049)
number shapes=186
number nodes=176
done!
Removing intermediate container 882f8548394e
---> 283d0715bfa8
Step 9/17 : RUN mkdir /map_data
---> Running in 0590b4cdbcfd
Removing intermediate container 0590b4cdbcfd
---> f90f0b430c72
Step 10/17 : VOLUME /map_data
---> Running in 1b7009b680df
Removing intermediate container 1b7009b680df
---> bcb35bb1f204
Step 11/17 : RUN mkdir /scripts
---> Running in 2b43ef0143ae
Removing intermediate container 2b43ef0143ae
---> 69601d9a89d7
Step 12/17 : COPY scripts/compile_style.sh /scripts
---> 233d0e36923d
Step 13/17 : COPY scripts/run_render.sh /scripts
---> 5933e5eb0c3a
Step 14/17 : COPY entrypoint.sh /
---> 4269f8ec06cb
Step 15/17 : RUN chmod 755 /scripts/*.sh && chmod 755 /entrypoint.sh
---> Running in ac496c54249d
Removing intermediate container ac496c54249d
---> 6c99491086df
Step 16/17 : WORKDIR /
---> Running in f210c3e8fc76
Removing intermediate container f210c3e8fc76
---> 8cf0112b049d
Step 17/17 : ENTRYPOINT /entrypoint.sh
---> Running in 9c983f5c6741
Removing intermediate container 9c983f5c6741
---> 85e1ac85aebd
Successfully built 85e1ac85aebd
Successfully tagged carto-proto_renderer:latest
WARNING: Image for service renderer was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling web (nginx:1.11-alpine)...
1.11-alpine: Pulling from library/nginx
709515475419: Pull complete
4b21d71b440a: Pull complete
c92260fe6357: Pull complete
ed383a1b82df: Pull complete
Digest: sha256:5aadb68304a38a8e2719605e4e180413f390cd6647602bee9bdedd59753c3590
Status: Downloaded newer image for nginx:1.11-alpine
Creating postgis ... done
Creating carto-proto_web_1 ... done
Creating rendereurpoulet ... done
[jibl@pc-100 carto-proto]$
en tout moins de 2h30 c'est certain, pour le batch
Remarque :+1:
selon le moment de la journée, le jour, l'import australie dure moins de 30 secondes au lieu de 2h30, et c'est le téléchargement du PBF, qui prend du temps, soit exactement l'instruction wget https://download.geofabrik.de/australia-oceania-latest.osm.pbf
dans le Dockerfile postgis :
Step 4/8 : RUN git clone https://github.com/dooman87/openstreetmap-carto.git && wget https://download.geofabrik.de/australia-oceania-latest.osm.pbf
---> Running in 0557eb4cb2f1
Cloning into 'openstreetmap-carto'...
--2018-10-22 09:47:11-- https://download.geofabrik.de/australia-oceania-latest.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 88.99.24.113, 88.99.105.98
Connecting to download.geofabrik.de (download.geofabrik.de)|88.99.24.113|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 633086720 (604M) [application/octet-stream]
Saving to: ‘australia-oceania-latest.osm.pbf’
0K .......... .......... .......... .......... .......... 0% 592K 17m25s
50K .......... .......... .......... .......... .......... 0% 420K 20m58s
100K .......... .......... .......... .......... .......... 0% 654K 19m14s
Ce n'est donc pas le traitement osm2pgsql
. Ilffaudra doc optimiser cela, avec un service batch scallable, chargé de télécahrger les PBF avec promesse tenue.
J'ai toujours des anomalies (les tuiles n'aparaissen tpas). Symptômes : le conteneur postgis indique dans ses logs un arrêt incorrect + conteneur rendereurpoulet indiquant qu'il est en attente de la BDD + la connexion à bdd par psql du conteneur rendereurpoulet
vers le conteneur postgis
.
[jibl@pc-100 carto-proto]$ docker logs -f rendereurpoulet
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
Waiting while database is initializing...
^C
[jibl@pc-100 carto-proto]$ docker logs -f postgis
LOG: database system was interrupted; last known up at 2018-10-21 21:18:53 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: invalid record length at 0/2EE27D8
LOG: redo is not required
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
^C
[jibl@pc-100 carto-proto]$ docker exec -it rendereurpoulet sh -c "apt-get update -y && apt-get install -y postgresql-client"
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
Get:3 http://archive.ubuntu.com trusty Release.gpg [933 B]
Get:4 http://archive.ubuntu.com trusty Release [58.5 kB]
Get:5 http://archive.ubuntu.com trusty-updates/main Sources [524 kB]
Get:6 http://archive.ubuntu.com trusty-updates/restricted Sources [6449 B]
Get:7 http://archive.ubuntu.com trusty-updates/universe Sources [268 kB]
Get:8 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1387 kB]
Get:9 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [21.4 kB]
Get:10 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [634 kB]
Get:11 http://archive.ubuntu.com trusty-security/main Sources [212 kB]
Get:12 http://archive.ubuntu.com trusty-security/restricted Sources [5050 B]
Get:13 http://archive.ubuntu.com trusty-security/universe Sources [104 kB]
Get:14 http://archive.ubuntu.com trusty-security/main amd64 Packages [967 kB]
Get:15 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [18.1 kB]
Get:16 http://archive.ubuntu.com trusty-security/universe amd64 Packages [338 kB]
Get:17 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Get:18 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:19 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Fetched 23.3 MB in 3s (6531 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libpq-dev libpq5 postgresql-client-9.3 postgresql-client-common
Suggested packages:
postgresql-doc-9.3 postgresql-9.3
The following NEW packages will be installed:
postgresql-client postgresql-client-9.3 postgresql-client-common
The following packages will be upgraded:
libpq-dev libpq5
2 upgraded, 3 newly installed, 0 to remove and 174 not upgraded.
Need to get 1047 kB of archives.
After this operation, 3224 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpq-dev amd64 9.3.24-0ubuntu0.14.04 [140 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpq5 amd64 9.3.24-0ubuntu0.14.04 [78.5 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client-common all 154ubuntu1.1 [25.4 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client-9.3 amd64 9.3.24-0ubuntu0.14.04 [797 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/main postgresql-client all 9.3+154ubuntu1.1 [5048 B]
Fetched 1047 kB in 0s (7039 kB/s)
(Reading database ... 40968 files and directories currently installed.)
Preparing to unpack .../libpq-dev_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking libpq-dev (9.3.24-0ubuntu0.14.04) over (9.3.15-0ubuntu0.14.04) ...
Preparing to unpack .../libpq5_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking libpq5 (9.3.24-0ubuntu0.14.04) over (9.3.15-0ubuntu0.14.04) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_154ubuntu1.1_all.deb ...
Unpacking postgresql-client-common (154ubuntu1.1) ...
Selecting previously unselected package postgresql-client-9.3.
Preparing to unpack .../postgresql-client-9.3_9.3.24-0ubuntu0.14.04_amd64.deb ...
Unpacking postgresql-client-9.3 (9.3.24-0ubuntu0.14.04) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_9.3+154ubuntu1.1_all.deb ...
Unpacking postgresql-client (9.3+154ubuntu1.1) ...
Setting up libpq5 (9.3.24-0ubuntu0.14.04) ...
Setting up libpq-dev (9.3.24-0ubuntu0.14.04) ...
Setting up postgresql-client-common (154ubuntu1.1) ...
Setting up postgresql-client-9.3 (9.3.24-0ubuntu0.14.04) ...
update-alternatives: using /usr/share/postgresql/9.3/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-client (9.3+154ubuntu1.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
[jibl@pc-100 carto-proto]$ docker exec -it rendereurpoulet sh -c "psql -d gis -U renderer-user -h postgis -p 5432"
psql (9.3.24, server 9.5.14)
WARNING: psql major version 9.3, server major version 9.5.
Some psql features might not work.
Type "help" for help.
gis=#
gis-# \q
[jibl@pc-100 carto-proto]$
tests à papa
export APPEL_HTTP_TUILE_CHOPE_DS_FIREFOX=http://192.168.1.30:8080/14/14790/10054.png
docker exec -it carto-proto_web_1 sh -c "apk update -y && apk add curl && curl $APPEL_HTTP_TUILE_CHOPE_DS_FIREFOX"
Ma conclusion temporaire : La BDD est nikel Le conteneur NGINX est OK et l'appli SPA Leaflet OK aussi.
Dans le même temps, il ne reste que le conteneur du renderer
qui lui indique dans ses logs, qu'il est en attente de la base de données.
ALors, je vais vérifier que la base de données existe, et ensuite, je me demanderai comment le renderer peut bien savoir quel est le nom de la BDD, ou le username/password PostGIS qu'il doit utiliser.
Mon avis :
POSTGRES_DB
et POSTGRES_USER
Donc, mon inbtuition :
Si bien que je vais supposer qu'il suffit que je déclare des variables d'environnement POSTGRES_USER
et POSTGRES_DB
, pour le conteneur renbdereurpoulet
dans le ./docker-compose.yml
, pour que l'application renderer ait connaissance du nom de la BDD, et des credentials pour s'y connecter.
Pour plus tard :+1:
Selon cette source :
Getting ready to load
osm2pgsql uses overcommit like many scientific and large data applications, which requires adjusting > a kernel setting to allow the data import to work successfully with multiple processes. Changing overcommit is also recommended for renderd and postgresql.
sudo tee /etc/sysctl.d/60-overcommit.conf <<EOF
# Overcommit settings to allow faster osm2pgsql imports
vm.overcommit_memory=1
EOF
sudo sysctl -p /etc/sysctl.d/60-overcommit.conf
Une recommandation intéressante pour le RUN, selon la même source :
We also want to monitor the new database for size and other useful statistics
sudo munin-node-configure --sh | sudo sh
sudo service munin-node restart
résultat : eh ben ça a pas loupé nos poulerts de chez shinesolutions.com ont tellement rien branlé, qu'un import de 2h30 pour démarrer un dicker-compose, ça n,e les a pas du tout choqué, manifestement, et puis c'est comme si en cherchant 1 semaine, j'aivais trouvé de quoi réduire la même tâche à 30 secondes au lieu de 2h30, simplement en consultant la doc officielle OpenSTreetMap/PostGIS.
Oh! and there we are :
Okay, tout va bien, sauf que le renderer n'arrive pas à accéder à la BDD POstGIS. Et pour cause, nos poulets de chez shiesolutions, comme je l'avais soupçonné, ont bêtement recopié du code trouvé sur une page internet, dans des dockerfiles, et en plus, sans rien comprendre à docker.
En effet, si nos pouilets on un volume partagé entre le conteneur PostGIS, et le conteneur renderer, c'est parce que la recette originale d'OpenStreetMap, qu'ils ont trouvée [ici utilisée par dooman87], recommande d'utiliser un connexion par socket UNIX, pour les connexions du renderer vers la BDD PostGIS :+1:
Pour que nous osyons bien d'accord, je cite le commentaire sur l'issue OpenStreetMap :
but when I omit the password, it can't connect to the database
Omit the user, password, hostname, and port, giving only the dbname. This will use local unix sockets to connect. For extents use -20037508,-20037508,20037508,20037508. This is perhaps slightly sub-optimal for only poland, but will still work. Also use 900913 for the SRS, not the custom string they gave. Overall reading through https://www.mapbox.com/tilemill/docs/guides/postgis-work/ I see numerous problems with it. My guide to setting up TileMill to work with OpenStreetMap Carto is https://hackpad.com/TileMill-I6rxRVszKMv but I haven't had time to finish it off.
Voilà pourquoi notre cerveau nous sert: analyser, avant de copier bêtelent, pour s'apercevoir après que l'on s'est appuyé sur un travail .... Améliorable.
No comment.
okay, so I have to understand how I can configure openstreetmap-carto 's connection to postgis
ok, about the long download of australia's PBF file, problem is solved by idempotence see. ./download.sh
modifications, and "VAULT"
After computing Australia all night long, PostGIS ended up in a state with errors, and as I acces the Leaflet app through NGINX, I get a connection refused:
Here are the logs after a whole night computing the australia data :