ppr: ERROR: boost::interprocess::bad_alloc when building Germany's graph data

[ch33hau]

Hi there,

We got ppr: ERROR: boost::interprocess::bad_alloc when building Germany's graph data. The same setup is working for smaller regions (e.g., Berlin, Stuttgart, etc.) though. We re-run from where it failed and got the same result. A fresh re-run also sees the same result.

Is there anything we can try in order to fix this issue?

Used Options:

  server.host: 0.0.0.0
  server.port: 8080
  server.cert_path: ::dev::
  server.priv_key_path: ::dev::
  server.dh_path: ::dev::
  server.api_key:
  server.log_path:
  server.static_path: /home/root/motis/web

  import.paths: schedule:/data/import/gtfs, osm:/data/import/input.osm.pbf
  import.data_dir: /data/data
  import.require_successful: 1

  dataset.path:
  dataset.prefix:
  dataset.graph_path: default
  dataset.write_serialized: 1
  dataset.write_graph: 0
  dataset.read_graph: 0
  dataset.read_graph_mmap: 0
  dataset.cache_graph: 0
  dataset.apply_rules: 1
  dataset.adjust_footpaths: 1
  dataset.expand_footpaths: 1
  dataset.use_platforms: 0
  dataset.begin: 20230504
  dataset.num_days: 2
  dataset.planned_transfer_delta: 30
  dataset.wzr_classes_path:
  dataset.wzr_matrix_path:
  dataset.no_local_transport: 0
  dataset.debug_broken_trips: 0

  modules: address, cc, csa, intermodal, lookup, guesser, osrm, parking, path, paxforecast, paxmon, ppr, schedule, raptor, railviz, revise, ris, routing, rt, tiles, tripbased
  exclude_modules:

  remotes:

  mode: test
  batch_input_file: queries.txt
  batch_output_file: responses.txt
  init:
  num_threads: 6
  direct: 0

  csa.bridge: 0
  csa.expand_footpaths: 0

  gbfs.update_interval: 5
  gbfs.urls:
  gbfs.db_size: 1099511627776

  intermodal.router: routing
  intermodal.revise: 0

  osrm.profiles: /home/root/motis/osrm-profiles/bike.lua, /home/root/motis/osrm-profiles/bus.lua, /home/root/motis/osrm-profiles/car.lua, /home/root/motis/osrm-profiles/foot.lua

  parking.db_max_size: 549755813888
  parking.max_walk_duration: 10
  parking.import.edge_rtree_max_size: 3221225472
  parking.import.area_rtree_max_size: 1073741824
  parking.import.lock_rtrees: 0
  parking.import_osm: 1
  parking.ppr_exact: 1
  parking.parkendd_endpoints:
  parking.parkendd_update_interval: 300

  path.use_cache:
  path.max_size: 34359738368

  paxforecast.forecast_results:
  paxforecast.behavior_stats:
  paxforecast.routing_cache:
  paxforecast.calc_load_forecast: 1
  paxforecast.publish_load_forecast: 0
  paxforecast.stats:
  paxforecast.deterministic_mode: 0
  paxforecast.min_delay_improvement: 5

  paxmon.journey_timezone:
  paxmon.journey_match_log:
  paxmon.match_tolerance: 0
  paxmon.split_groups: 0
  paxmon.split_groups_mean: 1.5
  paxmon.split_groups_stddev: 3
  paxmon.split_groups_seed: 0
  paxmon.generated_capacity_file:
  paxmon.stats:
  paxmon.capacity_match_log:
  paxmon.over_capacity_report:
  paxmon.broken_report:
  paxmon.reroute_unmatched: 0
  paxmon.reroute_file:
  paxmon.reroute_router: /tripbased
  paxmon.start_time: 1970-01-01T00:00:00Z+0000
  paxmon.end_time: 1970-01-01T00:00:00Z+0000
  paxmon.time_step: 60
  paxmon.arrival_delay_threshold: 20
  paxmon.preparation_time: 15
  paxmon.check_graph_times: 0
  paxmon.check_graph_integrity: 0
  paxmon.mcfp_scenario_dir:
  paxmon.mcfp_scenario_min_broken_groups: 500
  paxmon.mcfp_scenario_include_trip_info: 0
  paxmon.keep_group_history: 0
  paxmon.reuse_groups: 1

  ppr.import.use_dem: 0
  ppr.profile: /home/root/motis/ppr-profiles/default.json
  ppr.edge-rtree-max-size: 3221225472
  ppr.area-rtree-max-size: 1073741824
  ppr.lock-rtrees: 0
  ppr.prefetch-rtrees: 1
  ppr.verify-graph: 0

  railviz.initial_permalink:
  railviz.tiles_redirect:

  ris.db: ris.mdb
  ris.input:
  ris.db_max_size: 549755813888
  ris.init_time: 2023-05-05T21:42:32Z+0000
  ris.clear_db: 0
  ris.instant_forward: 1
  ris.gtfsrt.is_addition_skip_allowed: 0
  ris.http_proxy:
  ris.update_interval: 30
  ris.rabbitmq.host:
  ris.rabbitmq.port: 0
  ris.rabbitmq.username:
  ris.rabbitmq.password:
  ris.rabbitmq.vhost: ribasis
  ris.rabbitmq.queue:
  ris.rabbitmq.ca: cacert.pem
  ris.rabbitmq.cert: cert.pem
  ris.rabbitmq.key: key.pem
  ris.rabbitmq.log:

  rt.validate_graph: 0
  rt.validate_constant_graph: 0
  rt.print_stats: 1

  tiles.profile: /home/root/motis/tiles-profiles/background.lua
  tiles.import.use_coastline: 0
  tiles.import.flush_threshold: 10000000
  tiles.db_size: 1099511627776

  tripbased.use_data_file: 1

     address: FINISHED
         csa: FINISHED
   osrm-bike: FINISHED
    osrm-bus: FINISHED
    osrm-car: FINISHED
   osrm-foot: FINISHED
     parking: WAITING: {"PPR"}
        path: FINISHED
      paxmon: FINISHED
         ppr: ERROR: boost::interprocess::bad_alloc
    schedule: FINISHED
       tiles: FINISHED
   tripbased: FINISHED
2023-05-05T21:57:43Z [VERIFY FAIL] some imports were not successful: parking, ppr

initialization error: some imports were not successful: parking, ppr

Environment:

Release: 0.8.7
Memory: 60GB

[felixguendling]

There are several options you can try:

• first, I would turn off all modules you do not really use. It's always a good idea to explicitly list the modules you want to enable. Usually, for intermodal routing the minimum would be something like

modules=intermodal
modules=routing
modules=lookup
modules=ppr
modules=osrm

• For OSRM I would recommend turning on only the profiles you really use.
  • For example, the bus profile is only required if you use the path module. Otherwise, you probably do not have a use case for bus routing.
  • If you don't use the GBFS module, you can also turn off the foot profile for OSRM. Alternatively, if you enable the OSRM foot routing profile (which takes forever to load!), it makes sense to use this instead of the ppr profile if you do not use any special disability profile with the ppr module.
  • So probably (just guessing here), using the bike and car profile would be sufficient. This depends on which queries you send to MOTIS.

I would propose this configuration - or with foot.lua but then disable ppr completely. Note however that the MOTIS web UI uses FootPPR for requests. So this only makes sense if you use the MOTIS API directly, not via the MOTIS web UI.

[osrm]
profiles=/home/root/motis/osrm-profiles/bike.lua
profiles=/home/root/motis/osrm-profiles/car.lua

• Parking module is only needed in case you plan to send intermodal requests with CarParking as a mode enabled (or you want to use the parking API directly). If not, you can disable parking.

---

Additionally, on the master branch, @pablohoch pushed a new version of ppr that is more memory efficient. So if you're using docker, the master tag should have these changes: https://github.com/motis-project/motis/pkgs/container/motis/91040730?tag=master

If you compile your own version of MOTIS, it makes sense to enable MOTIS_MIMALLOC to save memory. The Linux release version has it already enabled.

---

Knowing more which kind of requests you are planning to send to MOTIS would enable me to tell you precisely the minimal set of modules you need.

[felixguendling]

In case you want to use the web UI:

• Since you don't use GBFS (no feeds configured in your config), you can turn off gbfs and the osrm profile foot.lua
• Tiles you can take from our server for testing and turn off the tiles module: railviz.tiles_redirect=https://t.motis-project.de/tiles/

Then, for the UI to work you could have this config.ini (use with /path/to/motis -c /path/to/config.ini or put it in the current working directory - then MOTIS will find it automatically):

modules=intermodal
modules=ppr
modules=parking
modules=osrm
modules=lookup
modules=railviz
modules=address
modules=guesser
modules=routing

ppr.profile=/home/root/motis/ppr-profiles/default.json
railviz.tiles_redirect=https://t.motis-project.de/tiles/

[import]
paths=schedule:/data/import/gtfs
paths=osm:/data/import/input.osm.pbf
data_dir=/data/data

[osrm]
profiles=/home/root/motis/osrm-profiles/bike.lua
profiles=/home/root/motis/osrm-profiles/car.lua
profiles=/home/root/motis/osrm-profiles/bus.lua

Especially not having the OSRM foot profile will save a lot of memory. Additionally, your config contains a lot of modules enabled which you don't use that each has their own copy of the timetable (csa, raptor, tripbased). You probably want to pick one, set it to be used (e.g. with intermodal.router=tripbased) and disable all the others.

[ch33hau]

Thanks for your quick reply!

Our existing code is using FootPPR so I will keep it and instead will disable the osrm-foot. I will adjust the modules and profiles accordingly, and see if they work.

Also, we do realize that the version we are using was released many months ago, actually, it will be my next question but you answered that already! I will try the master branch's version from the pre-built docker image, it seems so convenient to use.

[ch33hau]

Especially not having the OSRM foot profile will save a lot of memory. Additionally, your config contains a lot of modules enabled which you don't use that each has their own copy of the timetable (csa, raptor, tripbased). You probably want to pick one, set it to be used (e.g. with intermodal.router=tripbased) and disable all the others.

Yes, osrm-foot is taking a long time, I will disable it too. For now I don't have rt data, will keep intermodal.router=routing and disable the rest.

Thanks again!

[felixguendling]

For now I don't have rt data

In case you do not have real-time data, the tripbased routing is the fastest option. It comes with a few minutes of preprocessing and requires more memory. However, it's the fastest routing MOTIS has to offer at the moment (excluding experimental the GPU RAPTOR).

However, this is a optimization you can also do when you have the basic setup running (just add modules=tripbased and set intermodal.router=tripbased. The routing module still needs to stay enabled because it also provides trip information (converting a trip to a journey so it can be displayed when you click on a trip in the connection details).

[ch33hau]

Ah okay, I got it wrong earlier. Glad to know and I will apply it!

[felixguendling]

If you start using Docker now, it makes sense to checkout this setup: https://github.com/motis-project/motis/wiki/Installation-and-Setup#docker-setup

It uses a very small dataset that only has the Aachen city center, but you see how it works. The first part of the page (https://github.com/motis-project/motis/wiki/Installation-and-Setup) describes also the basic setup (split between input + data folder, config.ini, etc.).

Feel free to ask more questions if you get stuck.

[ch33hau]

Okay, I think the setup I have was based on the instruction here.

By the way, we have tried the GTFS data for Stuttgart before and it was working fine. Now we are trying to use the GTFS data for entire Germany (from https://gtfs.de/de/feeds/, file size is 217MB), and it always took ~1 Hour+ in the beginning step with the below logs:

    ...
     parse 0: [■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ ] 100% | Export schedule.raw
    schedule: [■                                                       ]   0% | RUNNING
    ...

The container has a memory of 60GB but it only uses around 8GB at this step.

Is this expected or anything can I do to make it faster?

[felixguendling]

I would recommend to use the official dataset: https://www.opendata-oepnv.de/ht/de/organisation/delfi/startseite?tx_vrrkit_view%5Bdataset_name%5D=deutschlandweite-sollfahrplandaten-gtfs&tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View#

The export step includes connecting nearby stops, which takes a lot of CPU (R-Tree lookups). However, I think I can add a option to turn this off. At the moment, MOTIS creates footpath and station equivalences automatically based on geometric proximity. In case of bad input data (missing footpaths and hierarchy connection), this significantly improves the routing results. However, maybe it should be possible to turn this "data improvement" off in case you don't need it.

Edit: however, I have never seen it taking 1h+ for the Germany dataset. It should be done in maximum a few minutes on up-to-date hardware.

[ch33hau]

Thanks!

Okay, good to know, I will compare with different hardware and different dataset, see if they make any differences.

[ch33hau]

Update:

After applied the changes, the build time was reducing from 11h to 5h. Also, with the latest version, ppr: ERROR: boost::interprocess::bad_alloc error is no longer appear.

I will close this issue. Thanks @felixguendling !

[felixguendling]

You're welcome :smile:

Please keep in mind that the initial import has a lot of steps that only depend on the OSM data. If you don't plan to change the OSM data as often as you change the timetable data, it makes sense to only delete the data/schedule and data/tripbased folder and keep the rest of the data directory. Then, only the import steps that depend on the timetable need to be repeated for the new timetable. This should be way faster than 5h.

For reference, this is the script we use to update the europe.motis.project.de instance:

#!/bin/sh

# DELETE PREPROCESSED SCHEDULE DATA
mkdir -p input
rm -rf /var/lib/docker/volumes/motis-europe_data-volume/_data/tripbased
rm -rf /var/lib/docker/volumes/motis-europe_data-volume/_data/schedule

# OSM DATA
if [ ! -f input/osm.pbf ]; then
  wget https://download.geofabrik.de/dach-latest.osm.pbf -O data/osm.pbf
fi

# LOAD TIMETABLES
rm -rf timetables
mkdir timetables

wget "https://www.opendata-oepnv.de/index.php?SECRET" -O timetables/delfi.zip
wget http://gtfs.ovapi.nl/nl/gtfs-nl.zip -O timetables/nl.zip
wget https://opentransportdata.swiss/de/dataset/timetable-2023-gtfs2020/permalink -O timetables/swiss.zip

# EXTRACT TIMETABLES
rm -rf input/schedule
mkdir -p input/schedule/delfi
mkdir -p input/schedule/nl
mkdir -p input/schedule/swiss

unzip timetables/delfi.zip -d input/schedule/delfi
unzip timetables/nl.zip -d input/schedule/nl
unzip timetables/swiss.zip -d input/schedule/swiss

chmod -R +r input/schedule*

docker compose down && docker compose up -d

It loads the Netherlands, Switzerland and German public transport timetable. Note that the osm.pbf file is only downloaded if it's not there yet. This prevents MOTIS from rerunning the import steps that only depend on the osm.pbf (such as ppr, tiles, address, and osrm).

Additionally, if you don't plan to offer the MOTIS web UI to users directly and you only use the routing API and the path module, you can disable even more modules to further speedup the import step.

[felixguendling]

Another flag you can try is dataset.cache_graph=true. What it does is that it writes the in-memory data structure of the timetable directly to disk after import. So the with next start, you can skip the step to create the routing graph from the schedule.raw. This reduces memory fragmentation and memory requirements in general for the routing instance. Additionally, MOTIS will start faster with the cached graph the next time.

If you really do not use any real-time updates (rt and ris module not active), you can also activate dataset.read_graph_mmap=true. This tells MOTIS to use a read-only memory map to page the routing graph into virtual memory, so it starts even faster (doesn't even load the full graph into memory - this will happen on-demand).

[ch33hau]

Thanks, we definitely need those improvements to our setup. Right now osm and schedule and store as a bundle in s3. I think your script gives us a clear idea how to re-build certain parts of data.

[felixguendling]

@ch33hau

and it always took ~1 Hour+ in the beginning step with the below logs:

    ...
     parse 0: [■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ ] 100% | Export schedule.raw
    schedule: [■                                                       ]   0% | RUNNING
    ...

The container has a memory of 60GB but it only uses around 8GB at this step.

Is this expected or anything can I do to make it faster?

The latest release has an option to choose the radius (in meters) in which stations will be automatically connected via footpaths: dataset.link_stop_distance or for the new nigiri core it's nigiri.link_stop_distance. If you set this to zero, the linking step is skipped completely and the import should be much faster. With the new nigiri core the linking step is faster, so there you might not need to turn it off. Reducing the radius also helps. The default value is 100 meters.