Example query not working on fresh install of osm-3s_v0.7.54 or 0.7.57

[RyanDeRose-TomTom]

On Ubuntu 18.04, I have followed these instructions: https://wiki.openstreetmap.org/wiki/Overpass_API/Installation#Ubuntu_or_Debian_6.0_.28squeeze.29_or_Debian_7.0_.28wheezy.29 using option 1 (and download_clone.sh instead of the whole planet.osm).

When I try to run the example query,

I get: ``` me@my-VM:/data/osm-3s_v0.7.54$ $EXEC_DIR/bin/osm3s_query --db-dir=$DB_DIR encoding remark: Please enter your query and terminate it with CTRL+D. encoding remark: Your input starts with a tag but not the root tag. Thus, a line with the datatype declaration and a line with the root tag 'osm-script' is added. This shifts line numbering by -2 line(s). runtime error: The dispatcher (i.e. the database management system) is turned off. ``` I first tried this with 0.7.57 and had the same problem. I saw something in one of the issues here about there being a possible incompatibility between 0.7.57 and the database clone, so I decided to try again with 0.7.54, since that was the one specified here: https://dev.overpass-api.de/no_frills.html

[mmd-osm]

Somehow you forgot to start the dispatcher process. It's described in the documentation you've linked to, and also here: https://dev.overpass-api.de/full_installation.html

Not sure how this relates to https://github.com/wiktorn/Overpass-API/issues/81. If you're using a Docker container, please follow up there.

Note: this issue tracker is mainly used for bug reports and enhancement requests. I'd suggest to look for community sites mentioned on the OSM wiki for HOWTO questions.

[RyanDeRose-TomTom]

I didn't forget to start the dispatcher. Both sources describe it as optional, and the query I'm trying is from the step before "Starting the dispatcher daemon"

Regardless, I've started the dispatcher to see if it helps. With this command:

 bin/osm3s_query

And the same query:

encoding remark: Please enter your query and terminate it with CTRL+D.
<query type="node"><bbox-query n="51.0" s="50.9" w="6.9" e="7.0"/><has-kv k="amenity" v="pub"/></query><print/>

I get:

encoding remark: Your input starts with a tag but not the root tag. Thus, a line with the
datatype declaration and a line with the root tag 'osm-script' is
added. This shifts line numbering by -2 line(s).
runtime error: open64: 2 No such file or directory /home/azureuser//osm3s_v0.7.54_osm_base Unix_Socket::7

It's strange that it's using the /home/azureuser directory, since my pwd is /data/osm-3s_v0.7.54, and I specified my satabase directory to be /data/osm-3s_v0.7.54/database_dir. So I tried the query again, making sure to be explicit with the database:

bin/osm3s_query --db-dir=database_dir

and get:

encoding remark: Your input starts with a tag but not the root tag. Thus, a line with the
datatype declaration and a line with the root tag 'osm-script' is
added. This shifts line numbering by -2 line(s).
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.54.13 ff15392f">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base=""/>

</osm>
runtime error: Query failed with the exception:  �}�SV

[mmd-osm]

0.7.54.13 is almost 4 years old by now, so that's certainly not a good starting point to set up a new instance. Please try to recreate this on a more recent 0.7.57 release. Also, please add some description how you set up your database. Did you use cloning? Import a database using a planet?

[RyanDeRose-TomTom]

Ok, fair enough. I'll try 0.7.57 again. I have been using the cloning method, but I did at one point follow the steps here: https://wiki.openstreetmap.org/wiki/User:Breki/Overpass_API_Installation#Trying_It_Out_With_A_Sample_OSM_Dataset_.28Optional.29 with the Isle of Man, and got that example query to work. So I may try downloading the planet if cloning keeps giving me trouble.

[RyanDeRose-TomTom]

with planet file downloaded, I ran

bin/init_osm3s.sh planet-latest.osm.bz2 $DB_DIR $EXEC_DIR

It ran for hours, but eventually:

Reading XML file ... finished reading nodes. Flushing to database ...... done.
Reorganizing the database ...terminate called after throwing an instance of 'File_Error'
bin/init_osm3s.sh: line 44: 31948 Broken pipe             bunzip2 < $PLANET_FILE
     31949 Aborted                 (core dumped) | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION

[mmd-osm]

Most likely you ran out of main memory. How much memory do you have available?

[RyanDeRose-TomTom]

16 GB, though the documentation only suggests "at least 4 GB of RAM". I can get more if necessary.

[mmd-osm]

Yeah, documentation is slightly outdated due to the crazy OSM data growth, 16GB will probably not be sufficient, unless you provide a smaller than default --flush-size parameter (see https://wiki.openstreetmap.org/wiki/User:Mmd/Overpass_API/Raspberry for an extreme case with --flush-size=1). We run on 64GB main memory everywhere. I would recommend at least 32GB.

[RyanDeRose-TomTom]

This keeps happening:

/data/osm-3s_v0.7.57.1$ bin/init_osm3s.sh planet-latest.osm.bz2 db_dir .
Reading XML file ... elapsed node 1. Flushing to database ...... done.
Reading XML file ... elapsed node 41661696. Flushing to database ..terminate called after throwing an instance of 'File_Error'
bin/init_osm3s.sh: line 44:  3823 Broken pipe             bunzip2 < $PLANET_FILE
      3824 Aborted                 (core dumped) | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION

edit: This is on a machine with 64 GB of RAM

edit 2: the planet file and database were filling the disk (256 GB). Moved the former and deleted the latter, and retrying.

[RyanDeRose-TomTom]

How large is your database directory? If it's more than 256 GB, I should stop this and get a bigger drive.

[drolbr]

What you observe is consistent with an empty environment varible $DB_DIR. Please try to write the path explicitly after --db-dir= and tell us if the problem persists.

The other things are likely unrelated to this.

[mmd-osm]

Actual space requirements depend on a number of factors. As an example uncompressed db without meta and without attic needs 350GB (with compression roughly 250 GB) . Otoh, attic + meta + lz4 compression needs around 400-450GB.

[RyanDeRose-TomTom]

What you observe is consistent with an empty environment varible $DB_DIR. Please try to write the path explicitly after --db-dir= and tell us if the problem persists.

The other things are likely unrelated to this.

Thanks, this does confirm some of my suspicions. I tried doing echo $db_dir and got nothing, as well as printenv, which didn't show any of the env variables related to this project. I didn't understand why/how they were set any more than why they ultimately weren't. I did end up explicitly calling them like this: init_osm3s.sh planet-latest.osm.bz2 db_dir . Is this any different from what you are suggesting? Which, I think would be this: init_osm3s.sh planet-latest.osm.bz2 --db-dir=db_dir .

[mmd-osm]

According to https://github.com/drolbr/Overpass-API/blob/master/src/bin/init_osm3s.sh#L44, the second parameter needs to be the directory name rather than "--db-dir=db_dir", as this would end up in variable $2 as -db-dir=--db-dir=db_dir

Previous comment by @drolbr regarding --db-dir parameter was referring to osm3s_query really.

[RyanDeRose-TomTom]

@mmd-osm gotcha, thanks. I think I am doing that part correctly (at least) since my database is filling up.

Unrelated: If I let this run until it fills the drive, and then restart the computer with the drive resized larger, can I then run init_osm3s again and have it pick up where it left off? I sort of did this earlier (without resizing) and it seems like it didn't start from the very beginning.

[mmd-osm]

I wouldn't recommend that approach as you might be hitting unknown issues. My recommendation is to always start fresh with an empty database directory and sufficient space available to run an import.