Error response from daemon: No such container: osm-transform

bmorlo commented 4 months ago

Hi all,

I've just used docker_run.sh -p planet-latest.osm.pbf from within the cloned git-repo. This made the image appear in docker. But the container does not start. I feel like I'm missing something.

When I now run docker_run.sh -p planet-latest.osm.pbf a second time, it tells me Error response from daemon: No such container: osm-transform.

Additionally, I cannot run osm-transform -p planet-latest.osm.pbf.

Best, Benedikt

rtroilo commented 4 months ago

Hi Benedikt,

When I now run docker_run.sh -p planet-latest.osm.pbf a second time, it tells me Error response from daemon: No such container: osm-transform.

This is correct as the script tries to clean up/delete a previous osm-transform container (which should not exists) to rebuild it fresh. We should try to suppress this confusing warning! Thank you!

To track down your problem, could you check this question:

Did you get any log messages besides the missing container?
What docker version do you have on your system? docker --version
Are you on the latest main branch?
Did you change/modify your osm-transform.cfg and could you share it if so?

I did just run this script on the latest main branch, with docker version Docker version 25.0.3 and got following log for the second run. Could you compare your log with mine below and share it with us if possible?

git/osm-transform$ ./docker_run.sh -p stuttgart.osm.pbf
Error response from daemon: No such container: osm-transform
[+] Building 1.3s (10/10) FINISHED                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                              0.0s
 => => transferring dockerfile: 728B                                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-slim                                                                                                                           0.8s
 => [internal] load .dockerignore                                                                                                                                                                 0.0s
 => => transferring context: 98B                                                                                                                                                                  0.0s
 => [1/5] FROM docker.io/library/debian:bookworm-slim@sha256:3d5df92588469a4c503adbead0e4129ef3f88e223954011c2169073897547cac                                                                     0.0s
 => [internal] load build context                                                                                                                                                                 0.1s
 => => transferring context: 958B                                                                                                                                                                 0.0s
 => CACHED [2/5] RUN set -eux;   apt-get update;   apt-get install -y --no-install-recommends     g++     cmake     ninja-build     libgdal-dev     libproj-dev     libosmium-dev     libboost-r  0.0s
 => CACHED [3/5] COPY . /osm-transform/                                                                                                                                                           0.0s
 => CACHED [4/5] RUN set -eux;    cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -G Ninja -S /osm-transform -B /osm-transform/cmake-build ;    cmake --build /osm-transfor  0.0s
 => CACHED [5/5] WORKDIR /osm                                                                                                                                                                     0.0s
 => exporting to image                                                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                                                           0.0s
 => => writing image sha256:f45afaf2ecadc2f815a905677210fb490cd6c4952b45c34c75b62de36974c8fe                                                                                                      0.0s
 => => naming to docker.io/library/osm-transform                                                                                                                                                  0.0s
Processing first pass: validate ways & relations...
[======================================================================] 100%
valid nodes: 41072 (0), valid ways: 9724 (24900), valid relations: 766 (886)
Processed in 0.906 s

WARNING: Failed to read geotiffs from gmteddata. This might lead to a lesser success rate when determining location elevations.
Load geotiff index...
[======================================================================] 100%
geotiff tiles indexed: 3

Processed in 0.019 s

Processing second pass: rebuild data...
[======================================================================] 100%
Nodes with no country: 39082, single country: 0, multiple countries: 0
Processed in 0.274 s

Original:              2317885 b
Reduced:               1094343 b
Reduction:             1223542 b (= 52.79 %)
All Nodes:               41072 Nodes
Elevation found:         95.09 % (39055)
Custom Elevation:        90.41 % (37132)
SRTM Elevation:           4.68 % (1923)
GMTED Elevation:          0.00 % (0)
Failed Elevation:         0.07 % (27)

Notice: More nodes were referenced in ways & relations than were found in the data. This typically happens
with OSM extracts with nodes omitted for ways & relations extending beyond the extent of the extract.

Peak memory used: 2619 MBytes

bmorlo commented 4 months ago

Hi @rtroilo!

Maybe I should clarify a few more things: 1) I'm running Windows 2) I'm using the windows terminal to run commands like docker_run.sh -p planet-latest.osm.pbf

Now regarding your questions:

Did you get any log messages besides the missing container? Yes, but I cannot copy the entire log since it disappears so quickly. Please see the attached screenshot. Before I took the screenshot, I deleted the image from docker and tried to create it again. However, now it seems like I cannot create the image anymore. The bash script appears to get stuck after the bookworm line.

What docker version do you have on your system? v26.00

Are you on the latest main branch? Edit: Yes, definitely cloned the main branch.

Did you change/modify your osm-transform.cfg and could you share it if so? Nope, I did not change anything there!

I'd like to ask another thing here: If my plan is to run osm-transform to have the file size of the europe-latest.osm.pbf file reduced so I can run ors with docker compose on my 32GB RAM machine, what is the step-by-step process I should follow? 1) Run osm-transform on the dataset? 2) Use the new version of the dataset to run with docker ors compose? 3) ...?

Appreciate your help a lot!

issue

rtroilo commented 4 months ago

Hi Benedikt,

I just spoke with Amandus and see your main confusion. This osm-transform project, while reducing the pbf-file-size by removing non ors relevant tag/elements, but does not reduce the memory consumption for graph building.

Luckily with the release of ors 8.0 we introduce a new ors-engine property

https://giscience.github.io/openrouteservice/run-instance/configuration/ors/engine/
graphs_data_access (default RAM-STORE)

With this option set to MMAP instead, you are able to run even full planet ors graph on your 32GB machine but with a little bit slower performance!

Just try it with this new option and give your ors around 8GB memory, Xmx option in the docker-compose.

https://giscience.github.io/openrouteservice/run-instance/running-with-docker#running-prebuilt-images

  # ------------------ JAVA OPTS ------------------ #
  # Configure the memory settings for JAVA or pass additional opts
  # Fore more available ENV properties see Prepare CATALINA_OPTS and JAVA_OPTS
  # in https://github.com/GIScience/openrouteservice/blob/main/docker-entrypoint.sh
  XMS: 1g  # start RAM assigned to java
  XMX: 2g  # max RAM assigned to java. Rule of Thumb: <PBF-size> * <profiles> * 2
  # Example: 1.5 GB pbf size, two profiles (car and foot-walking)
  # -> 1.5 * 2 * 2 = 6. Set xmx to be AT LEAST `-Xmx6g`
  ADDITIONAL_JAVA_OPTS: ""  # further options you want to pass to the java command

This should be enough to run it. Use be aware that the graph building process is around 20% slower with this option.

In case you get an memory allocation error during build time, please see our FAQ

https://giscience.github.io/openrouteservice/frequently-asked-questions#i-get-an-error-native-memory-allocation-mmap-failed-to-map-16384-bytes-for-committing-reserved-memory

Please give us feedback if you succeed or not!

bmorlo commented 4 months ago

Okay, cool - thanks for the support! Trying it right now. I'll keep you updated.

Edit: ors-app exited with code 137 - Unfortunately, I'm still getting the same error that lead to me considering osm-transform in the first place. :(

Edit 2.0: Tried it with a fresh compose.yml file - it looks like I had set REBUILD_GRAPHS to True. The local client is up and running now. HOWEVER, when I use it in my python code like this:

client = ors.Client(base_url='http://localhost:8080/ors/v2/directions')
        route_ors = client.request(
            url='/driving-car/geojson',
            post_json={
                'coordinates': coords
            })

I encounter the error openrouteservice.exceptions.ApiError: 400 ({'error': {'code': 2004, 'message': 'Request parameters exceed the server configuration limits. The approximated route distance must not be greater than 100000.0 meters.'}, 'info': {'engine': {'build_date': '2024-03-21T13:55:54Z', 'version': '8.0.0'}, 'timestamp': 1713858798429}}) and the program breaks.

Best, Benedikt

MichaelsJP commented 4 months ago

@bmorlo We are now entering ors config related waters :)

This error means that your config for openrouteservice needs adjusting with the following parameter

---
ors:
  engine:
    profiles:
      car:
        profile: driving-car
        enabled: true
        maximum_distance: 100000 -> put some higher value in here  that fits your needs

If you use REBUILD_GRAPH remember to set it to false after the rebuild. Else, every restart might rebuild your graphs ;).

Closing the slightly off-topic issue. Feel free to write us in our forum at ask.openrouteservice.org/.

Best Julian

rtroilo commented 4 months ago

Okay, cool - thanks for the support! Trying it right now. I'll keep you updated.

Thank you for your update!

Edit 2.0: Tried it with a fresh compose.yml file - it looks like I had set REBUILD_GRAPHS to True. The local client is up and running now.

Did this mean you were able and successfully build europe-latest.osm.pbf on your machine? How long did it take to build your graphs?

And as Julian wrote:

If you use REBUILD_GRAPH remember to set it to false after the rebuild. Else, every restart might rebuild your graphs ;).

Best Rafael

bmorlo commented 4 months ago

@rtroilo @MichaelsJP Thanks for getting back to me! Allow me one more question, even if it appears to be off-topic here: When using a new osm file for the first time with docker, do I have to set REBUILD_GRAPHS: True? If yes, what kind of output should I see in my cmd window? How long should it take?

I still get various errors when trying to retrieve distance data from the local api - even with the new MMAP in the compose.yml. The local host health check returns ready and also my docker RAM usage is at 15GB after a docker compose up. But when I try to access the local api in the browser like this http://localhost:8080/ors/v2/directions/driving-car?start=8.97634,48.67927&end=4.88969,52.37403, I get the error {"error":{"code":2099,"message":"Unable to get an appropriate route profile for RoutePreference = driving-car"},"info":{"engine":{"build_date":"2024-03-21T13:55:54Z","version":"8.0.0"},"timestamp":1714042611332}}.

Here is my cmd window output:

And this is what Docker is doing:

@MichaelsJP It appears to me that even after changing the maximum distance to 100000000m I run into issues. I thus believe the maximum distance is not the problem.

Best, Benedikt

bmorlo commented 4 months ago

@rtroilo @MichaelsJP Since the issue is not yet resolved, I'd really appreciate your assistance. Best, Benedikt

MichaelsJP commented 4 months ago

@bmorlo I'm sorry it did not lead to a functioning setup for you.

Your problem might be relevant for other users as well. Please, head over to our forum and get in touch with our support and community there. Issues in our repositories in general, are meant for developer topics only.

When opening an issue in the forum, please state your problem, provide us with your config and compose and reference the url to this issue, so the support team know the history of it.

That way we can ensure that the free support effort we provide the community and their results can also be found in a central place by the community.

Best

GIScience / osm-transform

Error response from daemon: No such container: osm-transform #45