osm-transform
Tool for reduction of OSM data and processing cost for elevation data extraction during graph building.
Removes metadata, unused elements and tags, and adds ele
tags to all retained nodes.
There are several possibilities to build and run osm-transform.
To run osm-transform with Docker, first, make sure that Docker is installed on the machine and clone this repository. From the root directory, you can run:
./docker_run.sh -p planet-latest.osm.pbf
This builds the Docker image and runs the preprocessor as a container, passing the provided OSM file as source file. The script maps the current working directory to the working directory within the container. Note that in this case all files required to run osm-transform must be located within the project folder, and the path to the OSM file needs to be relative to the project directory. All arguments are passed to osm-transform.
To build osm-transform on the command line we recommend you use cmake/ninja/g++.
Prerequisites for installing osm-transform at the command line are the following libraries:
From the project root directory, run
# on ubuntu
sudo apt install g++ cmake ninja-build libgdal-dev libproj-dev libosmium-dev libboost-all-dev
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -G Ninja -B ./cmake-build
cmake --build ./cmake-build --target osm-transform
cp ./cmake-build/osm-transform .
You can then use the tool by running
./osm-transform -p planet-latest.osm.pbf
The minimal required argument that needs to be provided is the OSM data file to process.
Generic options:
-v [ --version ] print version string
-h [ --help ] produce help message
Configuration:
-e [ --skip ] skip elevation data merge
-i [ --interpolate ] interpolate intermediate nodes
-p [ --osm-pbf ] arg Absolute file path to osm pbf file to process
-T [ --remove_tag ] arg (=(.*:)?source(:.*)?|(.*:)?note(:.*)?|url|created_by|fixme|wikipedia) regex to match removable tags
-F [ --geo_tiff_folders ] arg (=tiffs, srtmdata, gmteddata) absolute paths to Geotiff folders
-S [ --cache_limit ] arg (=1073741824) maximum memory used to store tiles in cache
-t [ --threshold ] arg (=0.5) only used in combination with interpolation, threshold for elevation
-f [ --config-file ] arg absolute file path to config file to use
-d [ --debug_mode ] debug_mode
Generic options:
-v [ --version ] print version string
-h [ --help ] produce help message
Configuration:
-p [ --osm_pbf ] arg path to osm pbf file to process
-e [ --skip_elevation ] skip elevation data merge
--srtm fetch SRTM tiles and exit
--gmted fetch GMTED tiles and exit
-i [ --interpolate ] interpolate intermediate nodes
-T [ --remove_tag ] arg (=(.*:)?source(:.*)?|(.*:)?note(:.*)?|url|created_by|fixme|wikipedia) regex to match removable tags
-F [ --geo_tiff_folders ] arg (=tiffs, srtmdata, gmteddata) paths to geotiff folders
-S [ --cache_limit ] arg (=1073741824) maximum memory used to store tiles in cache
-t [ --threshold ] arg (=0.5) only used in combination with interpolation, threshold for elevation
-a [ --area_mapping ] arg path to area mapping file to use
--area_mapping_id_col arg (=0) column number (zero-based) in area mapping file of area id
--area_mapping_geo_col arg (=1) column number (zero-based) in area mapping file of area geometry
--area_mapping_geo_type arg (=wkt) type of geometry string in area mapping file (possible values: 'wkt' (default), 'geojson')
--area_mapping_has_header arg (=1) area mapping file has header row
--area_mapping_processed_file_prefix arg (=mapping_) file prefix for processed mapping files
-f [ --config_file ] arg path to config file to use
--index_type arg (=flex_mem) index type for locations, needed for interpolate. see https://docs.osmcode.org/osmium/latest/osmium-index-types.html
-d [ --debug_mode ] debug_mode
A osm-transform.cfg
file can be used to set up the tool. The default configuration is as follows:
# number of elevation tiles to keep open in memory simultaneously
cache_limit = 10;
# regex for detecting tags that can safely be removed
remove_tag = "(.*:)?source(:.*)?|(.*:)?note(:.*)?|url|created_by|fixme|wikipedia";
# activate debug output
#debug_mode = true;
cache_limit
determines how many tile GTIFs are kept open in memory during the processing.
A too high number might result in paging overhead degrading performance.remove_tag
regex are stripped from the data.debug_mode
flag activates some verbose diagnostic output.debug_no_filter
flag deactivates filtering of elements, so that only metadata and tags are reduced.debug_no_tag_filter
flag deactivates filtering of tags, so that all tags are retained.For the elevation data merge, also download the elevations tiles for CGIAR SRTM data (see srtm.csi.cgiar.org) and GMTED2010 data (as a fallback option, see GMTED2010). The osm-transform tool provides an operation modeto download (and, in the case of CGIAR data, unpack) these resources. Be aware those require approx. 63.1 Gb and 26.3 Gb drive space respectively.
In the project directory, run
./docker_run.sh --srtm # download CGIAR SRTM tile files
./docker_run.sh --gmted # download GMTED tile files
If you have built the tool on your machine, you can run it natively using the following commands:
./osm-transform --srtm # download CGIAR SRTM tile files
./osm-transform --gmted # download GMTED tile files
This will create directories srtmdata
and gmteddata
respectively and download the geotiff tiles. Note that in those modes the tool does not perform any other operations.
The tool processes OSM data in two steps.
During the first pass, it filters way and relation elements and records the IDs and those of all referenced nodes where the elements are valid and used by ORS graphs. This process removes OSM ways representing objects unrelated to the road graph, with tags like "building", "landuse", "boundary", "natural" and a few others.
During the second pass, a new PBF file [file].ors.pbf
is written, containing only the relevant elements for the ORS
graphs.
From the retained elements, all tags matching the remove_tag
regular expression (see configuration file example above)
as well as irrelevant metadata (version, user_id, timestamp etc.) are stripped.
Before the first pass, the tool needs to allocate the required memory to store the flags for the valid data sets in
the source file. The max IDs numbers set in the config file need to be larger than the highest ID in the source file.
You can use the option -m
to let the tool determine the required memory automatically or option -c
to only do the
memory check to adjust the config values.
For each node, the preprocessor determines the elevation from the CGIAR data (if no data is available, falls back to
GMTED data).
You can skip this step by setting the -e
option. Any ele
tags already present in the OSM data are overwritten,
since this is current ORS (and GH) behavior.
You can pass the -o
option to have the preprocessor retain the ele
tag values where present in the OSM data.
The elevation value defaults to 0 where it could not be determined (outside tiles coverage or invalid data in both data
sets), since this is GH behavior.
Overall, a ca. 70% file size reduction is achieved. Skipping the elevation data merge increases this only by a few percent, and since the elevation data retrieval during graph building (which can be skipped if the preprocessor already merged the elevation data) has a significant impact on memory consumption during graph building, the default behavior is to include this operation.
Example console output while processing OSM planet file:
Max IDs from config: Node 7300000000 Way 800000000 Relation 20000000
Allocating memory: 870.23 Mb nodes, 95.37 Mb ways, 2.38 Mb relations
Processing first pass: validate ways & relations...
[======================================================================] 100% %
valid nodes: 1551699772 (5385098645), valid ways: 165335973 (596956981), valid relations: 3006399 (6985641)
Processed in 1043.549 s
Processing second pass: rebuild data...
Progress: 5989041267 / 5989041267 (100.00 %)
Processed in 10288.611 s
Original: 49796963050 b
Reduced: 14914077359 b
Reduction: 34882885691 b (= 70.05 %)
Elevation: 0.04 % failed (660459)
A log file is written containing all coordinates where no elevation value could be determined. The number of cases in the above example is normal (paths on Antarctica and other remote areas etc.).
usr/bin/g++
)Vcpkg
from bottom menu bar (or double tap shift
and search vcpkg
)libosmium
, boost-program-options
and gdal
(if not installed on system). This should install the correct
binaries for your OS.-m path-to-your.osm.pbf
absolute-path-to/osm-transform
If not:
CMake options
contains -DCMAKE_TOOLCHAIN_FILE=
with value like
/Users/your-user/.vcpkg-clion/vcpkg/scripts/buildsystems/vcpkg.cmake
This tool is under development and still experimental, though it has been successfully tested on several OSM subsets (Germany, DACH). Use at own risk.