search

drolbr / Overpass-API

A database engine to query the OpenStreetMap data.
http://overpass-api.de
GNU Affero General Public License v3.0
727 stars 91 forks source link

Cloning of areas #564

Open zabullet opened 4 years ago

zabullet commented 4 years ago

The snapshots/daily clones of the database are a nice way to bootstrap overpass, however it lacks the areas which means a delay of hours before one can use a new build (assuming you're using areas).

Any technical reason these aren't included in the snapshot?

zabullet commented 4 years ago

I am aware of https://github.com/drolbr/Overpass-API/issues/503, but want to understand whether areas can't be included for any technical reason.

mmd-osm commented 4 years ago

Cloning areas is neither supported by the clone mechanism, nor by the shell script to download a clone. It's somewhat unlikely that this will be added in the near future, given that the effort to create areas yourself is quite small compared to creating a full database.

zabullet commented 4 years ago

I'm just trying to ascertain whether there is a technical blocker or whether it is a lack of will.

The clone script is easily modified with as simple PR. The project is opensource, yet the clone mechanism and data is decidedly closed source (highly appreciated, but objectively it's closed).

I disagree in the assessment of "effort...is quite small", but this isn't the forum for that debate.

Shedding some light on the technical mechanisms of the cloning process or any constraints would be helpful.

drolbr commented 4 years ago

The cloning process is built such that the transfer of large files can be done with tools designed for that job, i.e. wget or curl. For that purpose, the command osm3s_script --clone copies the data files blockwise to a download directory. File transfer is then done as sequence of ordinary downloads. The downloaded files are immediately ready for use as a database.

Extending this to areas is technically possible. While the data files are a verbatim copy of the OSM main database, the areas are somewhat arbitrary. You can change both the rules and get slightly different results depending on the point of time of creation of the areas. Given that non-faithfulness, the areas have not been included in the cloning mechanism.

Please read also this warning.

mmd-osm commented 2 years ago

Now that area creation in 0.7.57 needs to process relations only (which takes about 40 minutes on reasonable hardware), I'd suggest to close this issue.