Closed bmorlo closed 4 months ago
Hi Benedikt,
When I now run docker_run.sh -p planet-latest.osm.pbf a second time, it tells me Error response from daemon: No such container: osm-transform.
This is correct as the script tries to clean up/delete a previous osm-transform
container (which should not exists) to rebuild it fresh. We should try to suppress this confusing warning! Thank you!
To track down your problem, could you check this question:
docker --version
osm-transform.cfg
and could you share it if so?I did just run this script on the latest main branch, with docker version Docker version 25.0.3
and got following log for the second run. Could you compare your log with mine below and share it with us if possible?
git/osm-transform$ ./docker_run.sh -p stuttgart.osm.pbf
Error response from daemon: No such container: osm-transform
[+] Building 1.3s (10/10) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 728B 0.0s
=> [internal] load metadata for docker.io/library/debian:bookworm-slim 0.8s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 98B 0.0s
=> [1/5] FROM docker.io/library/debian:bookworm-slim@sha256:3d5df92588469a4c503adbead0e4129ef3f88e223954011c2169073897547cac 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 958B 0.0s
=> CACHED [2/5] RUN set -eux; apt-get update; apt-get install -y --no-install-recommends g++ cmake ninja-build libgdal-dev libproj-dev libosmium-dev libboost-r 0.0s
=> CACHED [3/5] COPY . /osm-transform/ 0.0s
=> CACHED [4/5] RUN set -eux; cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -G Ninja -S /osm-transform -B /osm-transform/cmake-build ; cmake --build /osm-transfor 0.0s
=> CACHED [5/5] WORKDIR /osm 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:f45afaf2ecadc2f815a905677210fb490cd6c4952b45c34c75b62de36974c8fe 0.0s
=> => naming to docker.io/library/osm-transform 0.0s
Processing first pass: validate ways & relations...
[======================================================================] 100%
valid nodes: 41072 (0), valid ways: 9724 (24900), valid relations: 766 (886)
Processed in 0.906 s
WARNING: Failed to read geotiffs from gmteddata. This might lead to a lesser success rate when determining location elevations.
Load geotiff index...
[======================================================================] 100%
geotiff tiles indexed: 3
Processed in 0.019 s
Processing second pass: rebuild data...
[======================================================================] 100%
Nodes with no country: 39082, single country: 0, multiple countries: 0
Processed in 0.274 s
Original: 2317885 b
Reduced: 1094343 b
Reduction: 1223542 b (= 52.79 %)
All Nodes: 41072 Nodes
Elevation found: 95.09 % (39055)
Custom Elevation: 90.41 % (37132)
SRTM Elevation: 4.68 % (1923)
GMTED Elevation: 0.00 % (0)
Failed Elevation: 0.07 % (27)
Notice: More nodes were referenced in ways & relations than were found in the data. This typically happens
with OSM extracts with nodes omitted for ways & relations extending beyond the extent of the extract.
Peak memory used: 2619 MBytes
Hi @rtroilo!
Maybe I should clarify a few more things: 1) I'm running Windows 2) I'm using the windows terminal to run commands like docker_run.sh -p planet-latest.osm.pbf
Now regarding your questions:
Did you get any log messages besides the missing container? Yes, but I cannot copy the entire log since it disappears so quickly. Please see the attached screenshot. Before I took the screenshot, I deleted the image from docker and tried to create it again. However, now it seems like I cannot create the image anymore. The bash script appears to get stuck after the bookworm line.
What docker version do you have on your system? v26.00
Are you on the latest main branch? Edit: Yes, definitely cloned the main branch.
Did you change/modify your osm-transform.cfg and could you share it if so? Nope, I did not change anything there!
I'd like to ask another thing here: If my plan is to run osm-transform to have the file size of the europe-latest.osm.pbf file reduced so I can run ors with docker compose on my 32GB RAM machine, what is the step-by-step process I should follow? 1) Run osm-transform on the dataset? 2) Use the new version of the dataset to run with docker ors compose? 3) ...?
Appreciate your help a lot!
Hi Benedikt,
I just spoke with Amandus and see your main confusion. This osm-transform project, while reducing the pbf-file-size by removing non ors relevant tag/elements, but does not reduce the memory consumption for graph building.
Luckily with the release of ors 8.0 we introduce a new ors-engine property
With this option set to MMAP instead, you are able to run even full planet ors graph on your 32GB machine but with a little bit slower performance!
Just try it with this new option and give your ors around 8GB memory, Xmx option in the docker-compose.
# ------------------ JAVA OPTS ------------------ #
# Configure the memory settings for JAVA or pass additional opts
# Fore more available ENV properties see Prepare CATALINA_OPTS and JAVA_OPTS
# in https://github.com/GIScience/openrouteservice/blob/main/docker-entrypoint.sh
XMS: 1g # start RAM assigned to java
XMX: 2g # max RAM assigned to java. Rule of Thumb: <PBF-size> * <profiles> * 2
# Example: 1.5 GB pbf size, two profiles (car and foot-walking)
# -> 1.5 * 2 * 2 = 6. Set xmx to be AT LEAST `-Xmx6g`
ADDITIONAL_JAVA_OPTS: "" # further options you want to pass to the java command
This should be enough to run it. Use be aware that the graph building process is around 20% slower with this option.
In case you get an memory allocation error during build time, please see our FAQ
Please give us feedback if you succeed or not!
Okay, cool - thanks for the support! Trying it right now. I'll keep you updated.
Edit: ors-app exited with code 137
- Unfortunately, I'm still getting the same error that lead to me considering osm-transform in the first place. :(
Edit 2.0: Tried it with a fresh compose.yml
file - it looks like I had set REBUILD_GRAPHS
to True
. The local client is up and running now. HOWEVER, when I use it in my python code like this:
client = ors.Client(base_url='http://localhost:8080/ors/v2/directions')
route_ors = client.request(
url='/driving-car/geojson',
post_json={
'coordinates': coords
})
I encounter the error openrouteservice.exceptions.ApiError: 400 ({'error': {'code': 2004, 'message': 'Request parameters exceed the server configuration limits. The approximated route distance must not be greater than 100000.0 meters.'}, 'info': {'engine': {'build_date': '2024-03-21T13:55:54Z', 'version': '8.0.0'}, 'timestamp': 1713858798429}})
and the program breaks.
Best, Benedikt
@bmorlo We are now entering ors config related waters :)
This error means that your config for openrouteservice needs adjusting with the following parameter
---
ors:
engine:
profiles:
car:
profile: driving-car
enabled: true
maximum_distance: 100000 -> put some higher value in here that fits your needs
If you use REBUILD_GRAPH
remember to set it to false after the rebuild. Else, every restart might rebuild your graphs ;).
Closing the slightly off-topic issue. Feel free to write us in our forum at ask.openrouteservice.org/.
Best Julian
Okay, cool - thanks for the support! Trying it right now. I'll keep you updated.
Thank you for your update!
Edit 2.0: Tried it with a fresh
compose.yml
file - it looks like I had setREBUILD_GRAPHS
toTrue
. The local client is up and running now.
Did this mean you were able and successfully build europe-latest.osm.pbf on your machine? How long did it take to build your graphs?
And as Julian wrote:
If you use REBUILD_GRAPH remember to set it to false after the rebuild. Else, every restart might rebuild your graphs ;).
Best Rafael
@rtroilo @MichaelsJP
Thanks for getting back to me! Allow me one more question, even if it appears to be off-topic here: When using a new osm file for the first time with docker, do I have to set REBUILD_GRAPHS: True
? If yes, what kind of output should I see in my cmd window? How long should it take?
I still get various errors when trying to retrieve distance data from the local api - even with the new MMAP in the compose.yml
. The local host health check returns ready
and also my docker RAM usage is at 15GB after a docker compose up
. But when I try to access the local api in the browser like this http://localhost:8080/ors/v2/directions/driving-car?start=8.97634,48.67927&end=4.88969,52.37403
, I get the error {"error":{"code":2099,"message":"Unable to get an appropriate route profile for RoutePreference = driving-car"},"info":{"engine":{"build_date":"2024-03-21T13:55:54Z","version":"8.0.0"},"timestamp":1714042611332}}
.
Here is my cmd window output:
And this is what Docker is doing:
@MichaelsJP It appears to me that even after changing the maximum distance to 100000000m I run into issues. I thus believe the maximum distance is not the problem.
Best, Benedikt
@rtroilo @MichaelsJP Since the issue is not yet resolved, I'd really appreciate your assistance. Best, Benedikt
@bmorlo I'm sorry it did not lead to a functioning setup for you.
Your problem might be relevant for other users as well. Please, head over to our forum and get in touch with our support and community there. Issues in our repositories in general, are meant for developer topics only.
When opening an issue in the forum, please state your problem, provide us with your config and compose and reference the url to this issue, so the support team know the history of it.
That way we can ensure that the free support effort we provide the community and their results can also be found in a central place by the community.
Best
Hi all,
I've just used
docker_run.sh -p planet-latest.osm.pbf
from within the cloned git-repo. This made the image appear in docker. But the container does not start. I feel like I'm missing something.When I now run
docker_run.sh -p planet-latest.osm.pbf
a second time, it tells meError response from daemon: No such container: osm-transform
.Additionally, I cannot run
osm-transform -p planet-latest.osm.pbf
.Best, Benedikt