search

wiktorn / Overpass-API

Overpass API docker image
MIT License
134 stars 48 forks source link

Broken Pipe during database initialization #64

Closed kmehta-nuro closed 3 years ago

kmehta-nuro commented 3 years ago

After running the command

sudo docker run   -e OVERPASS_META=yes   -e OVERPASS_MODE=init   -e OVERPASS_PLANET_URL=http://download.geofabrik.de/north-america/us-latest.osm.bz2   -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/   -e OVERPASS_RULES_LOAD=10   -v /big/docker/overpass_db/:/db   -p 12345:80   -i -t   --name overpass_usa wiktorn/overpass-api

I get the following output. This has happened multiple times in a row. Is it simply a transient network failure or am I doing something wrong?

No database directory. Initializing
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.8G  100 11.8G    0     0  19.2M      0  0:10:29  0:10:29 --:--:-- 18.7M
Reading XML file ... elapsed node 61788857. Flushing to database ..terminate called after throwing an instance of 'File_Error'
/app/bin/init_osm3s.sh: line 44:    28 Broken pipe             bunzip2 < $PLANET_FILE
        29 Aborted                 (core dumped) | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION
Failed to process planet file
wiktorn commented 3 years ago

Looks like you might be running out of memory. How much your host has memory available?

kmehta-nuro commented 3 years ago

I ran it on a host with 20G of RAM and 30G of disk

wiktorn commented 3 years ago

Sounds like enough of RAM, but maybe the disk is getting filed? 30G might be quite smallish for whole North America

kmehta-nuro commented 3 years ago

Ok, thanks for the help. I'll try increasing the disk

Edit: Increasing memory solved the problem! I increased to 100G of disk