mediagis / nominatim-docker

100% working container for Nominatim
Creative Commons Zero v1.0 Universal
1.13k stars 445 forks source link

Problem installing Nominatim with France pbf #580

Open blaurent49 opened 2 months ago

blaurent49 commented 2 months ago

Describe the bug Hello, I tried to install Nominatim on a Debian VM (500 GB of hard disk available) via Docker with the France-latest.osm.pbf file previously downloaded from https://download.geofabrik.de/europe/.

Here is the command I used: sudo docker run -it \ -e PBF_PATH=/nominatim/data/france-latest.osm.pbf \ -e REPLICATION_URL=https://download.geofabrik.de/europe/france-updates/ \ -e POSTGRES_SHARED_BUFFERS=2GB \ -e POSTGRES_WORK_MEM=64MB \ -e POSTGRES_MAINTENANCE_WORK_MEM=1GB -p 8001:8080 \ -v $PWD:/nominatim/data \ --name nominatim \ mediagis/nominatim:4.4

The process started perfectly. It has been running for hours (at least 10h !) until encoutering the following error: psycopg2.errors.Diskfull: could not resize shared memory segment "/PostgreSQL.4111479138" to 50438144 bytes: No space left on device.

I don't understand the space problem because the VM has 500 GB of hard disk available.

I tried to check the posgresql.conf file but the shared_buffers, work_mem and maintenance_work_mem are in comments or at their default value. How can I be sure that the environnement variables are taken into account?

Am I doing something wrong?

Thank you in advance for your help.

Screenshots & Logs Capture d’écran 2024-09-10 225441

Desktop / Server (please complete the following information):

achedeuzot commented 2 months ago

Hi @blaurent49 Have you checked if your docker/postgres installation has access to the full size of your disk ? I had this issue in the past because docker couldn't use all the space available: and the error you're getting is quite explicit about it. Can you run df -h to check how much free space is available ?

For France latest, 175GB should be enough 😉 Depending on your RAM / CPU, it can take many hours. For a 8 CPU + 16GB of RAM server, it took around 18 hours.

blaurent49 commented 2 months ago

Thank you for your answer.

Here is the result of the dh -h command: image

As you can see, there should be enough space, shouldn't it?

achedeuzot commented 2 months ago

@blaurent49 I figure that the VM or mapped space can't use the whole disk you have configured. I'd look into your setup to fix this issue. Best of luck 🙂

blaurent49 commented 2 months ago

What can I look, more precisely?

achedeuzot commented 2 months ago

@blaurent49 I think the issue is not on nominatim but in your storage setup. You'll have to check why your nominatim container can't use more storage (software limit somewhere?). I would look into that first 🤔

Also, have you run the 'df -h' command from inside the nominatim container?

blaurent49 commented 2 months ago

I ran a bash in my container with docker exec -it nominatim /bin/bash and ran thedf -f command from inside it.

Here is what I got: image

It seems all right at this stage, doesn't it?

blaurent49 commented 2 months ago

Hello, I eventually solved my problem. Here is the command I used:

docker run -d \
  -e PBF_PATH=/nominatim/data/france-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/europe/france-updates/ \
  -e POSTGRES_SHARED_BUFFERS=2GB \
  -e POSTGRES_WORK_MEM=64MB \
  -e POSTGRES_MAINTENANCE_WORK_MEM=1GB \
  -p 8080:8080 \
  -v $PWD:/nominatim/data \
  --name nominatim \
  --shm-size=2gb \
  mediagis/nominatim:4.4

That wasn't a Nominatim related problem. It came from Docker: the containers by default have a shared memory (/dev/shm) size of 64MB. The important part in my command is: --shm-size=2gb

After that, I could install Nominatim. I didn't time precisely but it may have taken 24h. At least, it's a victory!

GildasLepennetier commented 2 months ago

Thanks @blaurent49, I also had an error at this step with a error message that was not helping. Using your extra options, I can reach the end of the import of the data into the database 👍.

achedeuzot commented 2 months ago

Interesting, it's in the default options recommend in the docker documentation but I didn't know it would return a storage issue. Good find!