Nominatim european map indexing/import

[giacomone83]

Hi all guys, I have a problem with nominatim and european map. I have installed nominatim on my local centOS 6.6 machine following the wiki.guide there http://wiki.openstreetmap.org/wiki/Nominatim/Installation I follow the instructions line to line and I manage to import a little map (monaco-latest.osm.pbf) and everything goes fine. Now I tried to import the european map (15.8 gb), i follow the same procedure, I launch the command

./utils/setup.php --osm-file european.latest.osm.pbf --all --osm2pgsql-cache 2048 2>&1 | tee setup.log

5 days ago and if I launch the command ps -aux | more I see the following output for nominatim/postgres process:

500      25306  0.0  0.0 230024   528 ?        S    Oct14   0:00 /usr/bin/php -C
q ./utils/setup.php --osm-file /srv/mappe/europe-latest.osm.pbf --all --osm2pgsq l-cache 2048
postgres 25310  0.0  0.0 509284  2792 ?        Ss   Oct14   0:00 postgres: mapserver nominatim [local] idle
500      25336  2.6 51.3 3908808 2015768 ?     R    Oct14 182:24 /srv/Nominatim-2.3.1/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 2048 -P 5432 -d nominatim /srv/mappe/europe-latest.osm.pbf
postgres 25338  0.7  3.6 352208 142580 ?       Ss   Oct14  49:17 postgres: mapserver nominatim [local] COPY
postgres 25339 39.3  3.5 325964 141056 ?       Rs   Oct14 2726:28 postgres: mapserver nominatim [local] idle in transaction
postgres 25340  0.2  3.5 326520 141116 ?       Ss   Oct14  16:17 postgres: mapserver nominatim [local] COPY
postgres 25341  0.0  0.0 325808  3156 ?        Ss   Oct14   0:00 postgres: mapserver nominatim [local] COPY

If, I launch the command:

 psql -d postgres -c "select * from pg_stat_activity where datname = 'nominatim'"

I see some log but, a thing that worry me a lot, if I write the follow code:

su - postgres
\connect nominatim
\dt

I see just 9 table...

What I have to do?? The process is still running?? Is stucked?? I'm trying to install the european map in a machine with centOS 6.7 and 250gb Disk space 4gb ram Please help me. Thanks a lot

[mtmail]

The server is underpowered for the import. The whole OSM planet (world) data file is currently 29GB so you're importing the equivalent of 50% of the OSM world data. I'd expect the output, the Postgres database, to take 250-350GB on disc.

If the last lines of your setup.log show something like Processing: Node(172640k 493.3k/s) Way(26854k 24.46k/s) Relation(400890 2045.36/s) then the import is still running and the indexing hasn't even started. That would mean the server already hit a bottleneck, either RAM (swapping to disc constantly) or slow disc and it will never finish.

If you have access to another server I'd recommend at least 4 cores, 16GB RAM and 500GB fast (SSD) disc. Depending on CPU speed and if the server is "in the cloud" (virtualized) you're still looking at days of processing. http://wiki.openstreetmap.org/wiki/Nominatim/Installation#Hardware

[giacomone83]

Ok, on the same machine (500gb Disk space, 16gb RAM) the import process run since 21/10/2015. It terminated the european map parsing in 115 hours and now I see in the log something like this:

Way stats: total(191121729), max(372765486) in 364760s
Relation stats: total(2649973), max(5543857) in 31870s
node cache: stored: 426005976(27.28%), storage efficiency: 78.24% (dense blocks: 324005, sparse nodes: 106346494), hit rate: 26.33%
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_rels
Stopping table: planet_osm_ways
Building index on table: planet_osm_rels (fastupdate=off)
Building index on table: planet_osm_ways (fastupdate=off)
Stopped table: planet_osm_rels in 1191s

Just this, no progress in the log if I launch the tail -f command, but the postgresql data folder still growing and the in the ps -aux | more I can see these process:

500  8069  0.0  0.0 229980  1196 ?    S    Oct21   0:00 /usr/bin/php -C
q ./utils/setup.php --osm-file europe-latest.osm.pbf --all --osm2pgsql-cache 18000
postgres  8073  0.0  0.0 509868  3008 ?   Ss   Oct21   0:00 postgres: mapserver nominatim [local] idle
500   8098  2.4  0.7 447376 124820 ?   Sl   Oct21 283:30 /home/nominatim/Nominatim-2.3.1/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore -C 4154 -P 5432 -d nominatim /home/nominatim/Nominatim-2.3.1/europe-latest.osm.pbf
postgres  8100  0.5  1.1 415516 186872 ?   Ss   Oct21  60:49 postgres: mapserver nominatim [local] COPY
postgres  8102  1.3  1.2 396696 208828 ?   Ds   Oct21 153:50 postgres: mapserver nominatim [local] CREATE INDEX

Please tell me that everything is fine.. Actually the amount engaged disk space is about 245gb and it still growing

Thanks a lot

[mtmail]

Looks fine. Postgres is busy creating indices. The next step will be Nominatim inspecting all imported places and creating a hierarchy (among other things). This will go rank by rank (2..30) each showing an estimated time in the log file. The last rank (30) will be the slowest.

[giacomone83]

Hi all and thanks a lot for your help.. It is really precious.. Than I think the setup has almost finished but now, after 11 days of process work, I see in the setup.log, after many create table and other insert, this advice :

....
Reanalysing database...
ANALYZE
Please download osmosis.
If it is already installed, check the path in your local settings (settings/local.php) file.
ANALYZE
....

But in the other log file, I can see the progress as below:

Done 94209 in 2674 @ 35.231487 per second - Rank 19 ETA (seconds): 23808.873047
Done 94212 in 2675 @ 35.219440 per second - Rank 19 ETA (seconds): 23816.931641
Done 94214 in 2676 @ 35.207024 per second - Rank 19 ETA (seconds): 23825.275391
Done 94218 in 2676 @ 35.208519 per second - Rank 19 ETA (seconds): 23824.148438
Done 94219 in 2677 @ 35.195740 per second - Rank 19 ETA (seconds): 23832.771484
Done 94220 in 2677 @ 35.196114 per second - Rank 19 ETA (seconds): 23832.490234
Done 94222 in 2678 @ 35.183720 per second - Rank 19 ETA (seconds): 23840.828125
Done 94226 in 2678 @ 35.185211 per second - Rank 19 ETA (seconds): 23839.70312

What does it mean? Is everything ok?? Tanks a lot

[mtmail]

Osmosis is used for regular updates, e.g. minute/hourly/daily. It's optional and only needed after the initial import finished. You can ignore the warning. http://wiki.openstreetmap.org/wiki/Nominatim/Installation#Setting_up_the_update_process

[lonvia]

The second logfile output shows that your import is progressing but is very, very slow. You probably will have to wait another 5-10 days before indexing has finished.