:wave: Welcome!
Read more about the 2022 planet import challenge:
Supporting CMake based build, in addition to autotools.
cpack: creates debian packagesHAVE_* flags to control libraries and features in useLibosmium-based output modes:
[out:pbf] and [out:opl][out:pbf(lz4)][out:osmxml] - faster output; Overpass API-specific OSM XML format extensions are not supportedRegular expressions:
[regexp:ICU][regexp:PCRE] or [regexp:PCREJIT]New/improved statements:
node[place=city|town|village]; query syntax can be used instead of node['place'~'^(city|town|village)$'];node[amenity=recycling][!~"^recycling:"]; (Related upstream issue: https://github.com/drolbr/Overpass-API/issues/589)New function all_vertex() which evaluates to true, if all vertices of a way fulfill a given expression.
Query
way({{bbox}})[building] (if:lrs_in(1,per_vertex(abs(angle()) > 170)));
can now be rewritten as:
way({{bbox}})[building] (if:!all_vertex(abs(angle()) <= 170));
Ad-hoc area creation on any closed way/relation
Dockerfile to facilitate building 0.7.59_mmd and 0.7.56 binaries (see docker/ directory)
PBF planet and diff files can be imported without any external PBF->XML conversion tools. By avoiding expensive XML parsing, and leveraging libosmium parallel file processing, imports see a significant speedup. To enable this new file importer, add command line parameter --use-osmium when calling update_from_dir or update_database. Command line parameter -f can be used to override the default input file format (PBF). See https://osmcode.org/file-formats-manual/ for permitted values.
PBF planet initial load supports LocationsOnWays extension. When using osmium tool to create the PBF file, it's essential to use the, --keep-untagged-nodes option to keep the untagged nodes in the output file.
osmium add-locations-to-ways -n extract.osm.pbf -o extract_low.osm.pbfmap_demo: alternative implementation for API 0.6 /map call for very large areas with > 100 million nodes. Returns PBF file format.
Improved dispatcher process security for use in multi-user environments (upstream issue 247)
Improved dispatcher signal handling for SIGTERM and SIGINT, same behavior as --terminate command line parameter
Environment variables:
OVERPASS_MAX_TIMEOUT: global override for maximum permitted [timeout:...] value.OVERPASS_MAX_ELEMENT_LIMIT: global override for maximum permitted [maxsize:...] value.OVERPASS_FCGI_MAX_REQUESTS: number of FastCGI requests before interpreter process is being terminated (when idle)OVERPASS_FCGI_MAX_ELAPSED_TIME: maximum time after which FastCGI process is being terminated (when idle)OVERPASS_REGEXP_ENGINE: default regexp engine to use if none is specified in Overpass QL settings. Possible values include: POSIX, ICU, PCRE and PCREJIT. PCREJIT is recommended for best performance.OVERPASS_LOG_LEVEL: define transactions.log log level. Available levels: 0 (error), 1 (warn), 2 (info), 3 (debug, default value), 4 (trace)OVERPASS_SHARED_NAME_SUFFIX: define /dev/shm/osm3s*... shared memory file suffix, allowing multiple parallel Overpass instances on one systemOVERPASS_MAX_SPACE_LIMIT: maximum size of a FastCGI process's virtual memory (address space), in bytes. Default value: 2^33 (=8 GiB); memory will be unlimited when the parameter value is set to 0.area(area))(poly))Three conversion tools are available for easy conversion from the official database file format to the custom one.
create_tagged_nodes [db_dir])create_node_changepack [db_dir])All conversion tools need to be executed without an active dispatcher instance.
A full attic database using lz4 compression for bin+map needs about 400G on 0.7.59 (mmd), based on 01/2023 data.
(List does not include some rather technical changes)
FastCGI support: avoid starting a new interpreter binary for each request, thereby enabling further caching options (upstream pull request 383)
Index and username caching: further taking advantage of FastCGI, database indices and usernames are only updated once per minute, and can be reused for many queries.
epoll based dispatcher processing for better scalability. This replaces round robin based unix domain socket polling with 10ms..100ms time intervals.
Hybrid array/bitmap container for better memory efficiency of large query statements (subset of features found in roaring bitmaps, https://arxiv.org/pdf/1603.06549.pdf)
Avoid object instantiation to access node/way/relation properties wherever possible in osm3s template db backend by leveraging CRTP.
Attic: speed up object reconstruction, avoid expensive copying of objects
Parallel processing support for update_from_dir/update_database. New parameter --parallel=n, where n denotes the max. number of parallel processes
around statement:
Area caching, use of memoization to avoid expensive calculation
Lazy way geometry store loading for "out qt" to reduce memory consumption, also in attic mode. In addition avoid recalculation of loop invariants in Way_Geometry_Store.
Ignore bounding box if it covers the whole globe.
Lazy (if: ) filter evaluation to short circuit boolean expression evaluation.
(if: ) expression evaluation now uses std::variant for intermediate results to avoid back and forth conversions from number to string types, resulting in significant speedups for angle() et al.
Changeset based (changed: ) filtering to process huge changesets in Achavi. Additional caching in changed to avoid recalculation of already available results.
Example syntax for changeset 46503970: (syntax is subject to change)
[adiff:"2017-03-01T20:28:34Z","2017-03-01T20:28:42Z"];
(node(changed!46503970);way(bn);way(changed!46503970);relation(changed!46503970););out meta geom qt;(._; .result;)->.result; statements inside foreach loops.out qt; based output mode using index based prefetching.--clone-parallel=n command line parameter.[out:custom] and [out:popup] output modes--disable-overpassxml to remove support for Overpass XML dialectTest setup:
Results:
Query runtime:
mean 0.054 s
min 0.000 s
max 205.513 squantile runtime (in s)
---------------------------
10% 0.002
50% 0.007
90% 0.042
95% 0.111
99% 0.601
99.5% 1.065
99.9% 5.829
99.95% 12.425
99.99% 30.380
99.995% 56.768
99.999% 120.544
99.9999% 148.805(-> 99.5% of all queries take less than 1 second)
Used supervisord config settings
environment=
OVERPASS_FCGI_MAX_REQUESTS=10000,
OVERPASS_FCGI_MAX_ELAPSED_TIME=900,
OVERPASS_REGEXP_ENGINE="PCREJIT",
OVERPASS_DEFAULT_TIMEOUT=60,
OVERPASS_MAX_TIMEOUT=120,
OVERPASS_MAX_SPACE_LIMIT=8589934592,
OSMIUM_POOL_THREADS=1,
OVERPASS_LOG_LEVEL=2Base image: Ubuntu 20.04
git clone https://github.com/mmd-osm/Overpass-API.git
cd Overpass-API
git checkout test7591
git submodule update --initsudo apt-get update -qq || true
sudo apt-get install -y g++ git make autoconf automake ca-certificates libtool \
libfcgi-dev libxml2-dev zlib1g-dev \
expat libexpat1-dev liblz4-dev libbz2-dev libicu-dev \
libfmt-dev libpcre2-dev libcereal-dev libgoogle-perftools-dev \
--no-install-recommendsC++17 compiler support is mandatory to build binaries.
pushd src/
chmod u+x test-bin/*.sh
autoscan
aclocal
autoheader
libtoolize
automake --add-missing
autoconf
popd
mkdir -p build
cd build
../src/configure CXXFLAGS="-Werror=implicit-function-declaration -D_FORTIFY_SOURCE=2 -fexceptions -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fno-omit-frame-pointer -flto -fwhole-program -march=native -O2 -ftree-vectorize -g3 -ggdb" LDFLAGS="-ltcmalloc -flto -fwhole-program -lpcre2-8 -lfmt" --prefix=$EXEC_DIR --enable-lz4 --enable-fastcgi --enable-tests
make V=0 -j3
make install(also see docker/ directory for examples)
/etc/supervisor/conf.d/overpass.conf
[fcgi-program:interpreter]
socket=unix:///var/run/interpreter.socket
socket_owner=www-data
socket_mode=0660
environment=
OVERPASS_FCGI_MAX_REQUESTS=10000,
OVERPASS_FCGI_MAX_ELAPSED_TIME=900,
OVERPASS_REGEXP_ENGINE="PCREJIT"
command=/home/user/osm3s/fcgi-bin/interpreter
numprocs=6
priority=999
process_name=%(program_name)s_%(process_num)02d
user=www-data
autorestart=true
autostart=true
startsecs=1
startretries=3
stopsignal=QUIT
stopwaitsecs=10
redirect_stderr=true
stdout_logfile=/var/log/interpreter.log
stdout_logfile_maxbytes=10MBForwarding calls to /api/interpreter to local socket managed by supervisord. Requires mod_proxy_fcgi.
ProxyPass /api/interpreter unix:///var/run/interpreter.socket|fcgi://localhost/api/interpreterReplacing /api/map shell script: use Apache rewrite engine to use /api/interpreter endpoint instead:
<LocationMatch "^/api/map$">
RewriteEngine On
RewriteCond %{QUERY_STRING} ^bbox=([\-0-9\.]+),([\-0-9\.]+),([\-0-9\.]+),([\-0-9\.]+)$
RewriteRule ".*" "/api/interpreter?data=[timeout:300][maxsize:2000000000][bbox:%2,%1,%4,%3];(node(%2,%1,%4,%3);way(bn);node(w););(._;(rel(bn)->.a;rel(bw)->.a;);rel(br););out meta;" [PT]
</LocationMatch>
<LocationMatch "^/api/map\.pbf$">
RewriteEngine On
RewriteCond %{QUERY_STRING} ^bbox=([\-0-9\.]+),([\-0-9\.]+),([\-0-9\.]+),([\-0-9\.]+)$
RewriteRule ".*" "/api/interpreter?data=[out:pbf][timeout:300][maxsize:2000000000][bbox:%2,%1,%4,%3];(node(%2,%1,%4,%3);way(bn);node(w););(._;(rel(bn)->.a;rel(bw)->.a;);rel(br););out meta;" [PT]
</LocationMatch>