Documentation for creating new routing profiles is outdated

[ghost]

It seems like the documentation for creating new routing profiles is outdated (e.g. https://github.com/graphhopper/graphhopper/blob/6.x/docs/core/create-new-flagencoder.md and https://github.com/graphhopper/graphhopper/blob/6.x/docs/core/profiles.md) because the interface FlagEncoder does not exist anymore.

[easbar]

I just removed the mention of FlagEncoder from profiles.md: a3d6ca1c0ada941a86a0feace39f793a9120419c Regarding create-new-flagencoder.md, I think we should just remove the entire file. The documentation needs a complete overhaul, especially everything that describes low level stuff. But this does not make much sense before the flag encoder refactoring is finished. For now the best available documentation is the code itself and the corresponding unit tests.

[easbar]

If all you want to do is tweak some of the routing profiles, take a look at custom models. They should cover most use-cases. Otherwise you can think of VehicleTagParsers to be the new FlagEncoders (but they will eventually be removed as well). Basically there are only encoded values and parsers that write to the encoded values. Basically the only thing that makes VehicleTagParsers specially is that they write speed and access values, not just one...

[ghost]

Thanks for your reply. In my case, changing both speed and access values is quite important. So I guess I'll try to create a new implementation of VehicleTagParser until it's removed.

[easbar]

Yes, either try to create a new implementation or modify one of the existing ones.

[karussell]

Thanks for your reply. In my case, changing both speed and access values is quite important.

With custom models you can change this, i.e. speed and access (priority) values. And if you use the roads vehicle as base it is very powerful and you can model even cases where your vehicles can e.g. use motorways and footways at the same time or you can go into the opposite direction of oneways and still avoid these cases or allow them for certain areas only etc.

[HarelM]

Hi, I'm trying to migrate to version 6 and am experiencing some problems. I'm not sure if this is a documentation issue or a bug, so for now I'm writing here: I'm trying to migrate the no longer exiting car4wd to use a custom model. The following is my config.yml I use in order to create this profile:

profiles:
    - name: car4wd
      vehicle: roads
      weighting: custom
      custom_model:
        distance_influence: 1
        speed:
          - if: "track_type == GRADE4 || track_type == GRADE5"
            limit_to: 5
          - else: ""
            limit_to: "car_average_speed"
        priority:
          - if: "track_type != GRADE4 && track_type != GRADE5 && car_access == false"
            multiply_by: "0"
    - name: hike
      vehicle: hike
      weighting: short_fastest
    - name: mtb
      vehicle: mtb
      weighting: short_fastest

I'm getting the following error when running the code:

Profile: name=car4wd|vehicle=roads|weighting=custom|turnCosts=false|hints={custom_model=distanceInfluence=1.0|headingPenalty=300.0|speedStatements=[{"if": "track_type == GRADE4 || track_type == GRADE5", "limit_to": 5}, {"else": "null", "limit_to": car_average_speed}]|priorityStatements=[{"if": "track_type != GRADE4 && track_type != GRADE5 && car_access == false", "multiply_by": 0}]|areas={}, custom_model_file=empty}
Error: Cannot compile expression: priority entry invalid condition "track_type != GRADE4 && track_type != GRADE5 && car_access == false": 'track_type' not available
        at com.graphhopper.GraphHopper.checkProfilesConsistency(GraphHopper.java:1018)
        at com.graphhopper.GraphHopper.process(GraphHopper.java:792)

I've took the car4wd from the example yml in this repo. I've tried to use json inside the config.yml as well. Is it a documentation issue or a bug in the checkProfilesConsistency code?

[easbar]

You probably need to add track_type to graph.encoded_values.

[HarelM]

Thanks for the quick reply, I wish it was that simple :-)

I've basically took the latest config-example file and just changed the things I needed. The following is the config I used. It has the graph.encoded_values. The command I run uses the docker image we generate for graphhopper which has a helper shell script for better readability. Having said that, in version 5.1 I'm able to use this script and generate a graph but not in 6.2 (as car4wd was removed in 6.0 and I'm unable to make the custom model work). Any help would be appreciated. BTW the custom model in the example-config (the one commented out) is working, but it's not what I need. Also when I comment out the priority I'm getting an error about not knowing what is car_average_speed...

graphhopper:

  # OpenStreetMap input file PBF or XML, can be changed via command line -Ddw.graphhopper.datareader.file=some.pbf
  datareader.file: ""
  # Local folder used by graphhopper to store its data
  graph.location: graph-cache

  ##### Routing Profiles ####

  # Routing can be done only for profiles listed below. For more information about profiles and custom profiles have a
  # look into the documentation at docs/core/profiles.md or the examples under web/src/test/java/com/graphhopper/application/resources/
  # or the CustomWeighting class for the raw details.
  #
  # In general a profile consists of the following
  # - name (required): a unique string identifier for the profile
  # - vehicle (required): refers to the `graph.vehicles` used for this profile
  # - weighting (required): the weighting used for this profile like custom,fastest,shortest or short_fastest
  # - turn_costs (true/false, default: false): whether or not turn restrictions should be applied for this profile.
  #
  # Depending on the above fields there are other properties that can be used, e.g.
  # - distance_factor: 0.1 (can be used to fine tune the time/distance trade-off of short_fastest weighting)
  # - u_turn_costs: 60 (time-penalty for doing a u-turn in seconds (only possible when `turn_costs: true`)).
  #   Note that since the u-turn costs are given in seconds the weighting you use should also calculate the weight
  #   in seconds, so for example it does not work with shortest weighting.
  # - custom_model_file: when you specified "weighting: custom" you need to set a json file inside your custom_model_folder
  #   or working directory that defines the custom_model. If you want an empty model you can also set "custom_model_file: empty".
  #   You can also use th e`custom_model` field instead and specify your custom model in the profile directly.
  #
  # To prevent long running routing queries you should usually enable either speed or hybrid mode for all the given
  # profiles (see below). Or at least limit the number of `routing.max_visited_nodes`.
  profiles:
    - name: car4wd
      vehicle: roads
      weighting: custom
      custom_model:
        distance_influence: 1
        speed:
          - if: "track_type == GRADE4 || track_type == GRADE5"
            limit_to: 5
          - else: ""
            limit_to: "car_average_speed"
        priority:
          - if: "track_type != GRADE4 && track_type != GRADE5 && car_access == false"
            multiply_by: "0"
    - name: hike
      vehicle: hike
      weighting: short_fastest
    - name: mtb
      vehicle: mtb
      weighting: short_fastest

  # Speed mode:
  # Its possible to speed up routing by doing a special graph preparation (Contraction Hierarchies, CH). This requires
  # more RAM/disk space for holding the prepared graph but also means less memory usage per request. Using the following
  # list you can define for which of the above routing profiles such preparation shall be performed. Note that to support
  # profiles with `turn_costs: true` a more elaborate preparation is required (longer preparation time and more memory
  # usage) and the routing will also be slower than with `turn_costs: false`.
  profiles_ch:
    - profile: car4wd
    - profile: hike
    - profile: mtb

  # Hybrid mode:
  # Similar to speed mode, the hybrid mode (Landmarks, LM) also speeds up routing by doing calculating auxiliary data
  # in advance. Its not as fast as speed mode, but more flexible.
  #
  # Advanced usage: It is possible to use the same preparation for multiple profiles which saves memory and preparation
  # time. To do this use e.g. `preparation_profile: my_other_profile` where `my_other_profile` is the name of another
  # profile for which an LM profile exists. Important: This only will give correct routing results if the weights
  # calculated for the profile are equal or larger (for every edge) than those calculated for the profile that was used
  # for the preparation (`my_other_profile`)
  profiles_lm: []

  #### Vehicles ####

  # The vehicle defines the base for how the routing of a profile behaves. It can be fine tuned using the options:
  # name=mycustomvehicle,block_private=true,turn_costs=true,transportation_mode=MOTOR_VEHICLE (only for the roads vehicle)
  # Still, it is recommended to avoid changing the vehicle settings and change the custom model instead.
  graph.flag_encoders: roads|block_fords=false|block_barriers=false,hike|block_fords=false,mtb|block_fords=false|block_barriers=false
  # Other standard vehicles: foot,bike,mtb,racingbike,motorcycle,wheelchair

  #### Encoded Values ####

  # Add additional information to every edge. Used for path details (#1548) and custom models (docs/core/custom-models.md)
  # Default values are: road_class,road_class_link,road_environment,max_speed,road_access
  # More are: surface,smoothness,max_width,max_height,max_weight,hgv,max_axle_load,max_length,hazmat,hazmat_tunnel,hazmat_water,toll,track_type,
  #           mtb_rating, hike_rating,horse_rating,lanes
  graph.encoded_values: road_class,road_class_link,road_environment,max_speed,road_access,track_type,surface

  #### Speed, hybrid and flexible mode ####

  # To make CH preparation faster for multiple profiles you can increase the default threads if you have enough RAM.
  # Change this setting only if you know what you are doing and if the default worked for you.
  # prepare.ch.threads: 1

  # To tune the performance vs. memory usage for the hybrid mode use
  # prepare.lm.landmarks: 16

  # Make landmark preparation parallel if you have enough RAM. Change this only if you know what you are doing and if
  # the default worked for you.
  # prepare.lm.threads: 1

  #### Elevation ####

  # To populate your graph with elevation data use SRTM, default is noop (no elevation). Read more about it in docs/core/elevation.md
  graph.elevation.provider: srtm

  # default location for cache is /tmp/srtm
  graph.elevation.cache_dir: /usr/src/app/elevation/

  # If you have a slow disk or plenty of RAM change the default MMAP to:
  graph.elevation.dataaccess: RAM_STORE

  # To enable bilinear interpolation when sampling elevation at points (default uses nearest neighbor):
  # graph.elevation.interpolate: bilinear

  # Reduce ascend/descend per edge without changing the maximum slope:
  # graph.elevation.edge_smoothing: ramer
  # removes elevation fluctuations up to max_elevation (in meter) and replaces the elevation with a value based on the average slope
  # graph.elevation.edge_smoothing.ramer.max_elevation: 5
  # A potentially bigger reduction of ascend/descend is possible, but maximum slope will often increase (do not use when average_slope or maximum_slope shall be used in a custom_model)
  # graph.elevation.edge_smoothing: moving_average

  # To increase elevation profile resolution, use the following two parameters to tune the extra resolution you need
  # against the additional storage space used for edge geometries. You should enable bilinear interpolation when using
  # these features (see #1953 for details).
  # - first, set the distance (in meters) at which elevation samples should be taken on long edges
  # graph.elevation.long_edge_sampling_distance: 60
  # - second, set the elevation tolerance (in meters) to use when simplifying polylines since the default ignores
  #   elevation and will remove the extra points that long edge sampling added
  # graph.elevation.way_point_max_distance: 10

  #### Urban density (built-up areas) ####

  # This feature allows classifying roads into 'rural', 'residential' and 'city' areas (encoded value 'urban_density')
  # Use 1 or more threads to enable the feature
  # graph.urban_density.threads: 8
  # Use higher/lower sensitivities if too little/many roads fall into the according categories.
  # Using smaller radii will speed up the classification, but only change these values if you know what you are doing.
  # If you do not need the (rather slow) city classification set city_radius to zero.
  # graph.urban_density.residential_radius: 300
  # graph.urban_density.residential_sensitivity: 60
  # graph.urban_density.city_radius: 2000
  # graph.urban_density.city_sensitivity: 30

  #### Subnetworks ####

  # In many cases the road network consists of independent components without any routes going in between. In
  # the most simple case you can imagine an island without a bridge or ferry connection. The following parameter
  # allows setting a minimum size (number of edges) for such detached components. This can be used to reduce the number
  # of cases where a connection between locations might not be found.
  prepare.min_network_size: 200

  #### Routing ####

  # You can define the maximum visited nodes when routing. This may result in not found connections if there is no
  # connection between two points within the given visited nodes. The default is Integer.MAX_VALUE. Useful for flexibility mode
  # routing.max_visited_nodes: 1000000

  # Control how many active landmarks are picked per default, this can improve query performance
  # routing.lm.active_landmarks: 4

  # You can limit the max distance between two consecutive waypoints of flexible routing requests to be less or equal
  # the given distance in meter. Default is set to 1000km.
  routing.non_ch.max_waypoint_distance: 1000000

  #### Storage ####

  # configure the memory access, use RAM_STORE for well equipped servers (default and recommended)
  graph.dataaccess.default_type: RAM_STORE

  # will write way names in the preferred language (language code as defined in ISO 639-1 or ISO 639-2):
  # datareader.preferred_language: en

  # Sort the graph after import to make requests roughly ~10% faster. Note that this requires significantly more RAM on import.
  # graph.do_sort: true

  #### Custom Areas ####

  # GraphHopper reads GeoJSON polygon files including their properties from this directory and makes them available
  # to all tag parsers and vehicles. Country borders (see countries.geojson) are always included automatically.
  # custom_areas.directory: path/to/custom_areas

  #### Country Rules ####

  # GraphHopper applies country-specific routing rules during import (not enabled by default).
  # You need to redo the import for changes to take effect.
  # country_rules.enabled: true

# Dropwizard server configuration
server:
  application_connectors:
  - type: http
    port: 8989
    # for security reasons bind to localhost
    bind_host: localhost
  request_log:
      appenders: []
  admin_connectors:
  - type: http
    port: 8990
    bind_host: localhost
# See https://www.dropwizard.io/en/latest/manual/core.html#logging
logging:
  appenders:
  - type: file
    time_zone: UTC
    current_log_filename: logs/graphhopper.log
    log_format: "%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"
    archive: true
    archived_log_filename_pattern: ./logs/graphhopper-%d.log.gz
    archived_file_count: 30
    never_block: true
  - type: console
    time_zone: UTC
    log_format: "%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"

[easbar]

Did you try without Docker? I just edited your last comment btw, but only to show the yaml you posted with syntax highlighting.

[easbar]

Without Docker it all works as expected. Your first error goes away when you add track_type to graph.encoded_values and then you get another error about car_access which goes away when you add a car profile.

[HarelM]

Can you send me the command line you are using that works for you? I'm doing a graph import and not a "regular" run. I'll send the command later today when I'm in front of my PC. I can log into the container and run any command. I don't think it's related to docker...

[easbar]

This one: https://github.com/graphhopper/graphhopper#installation

[HarelM]

Thanks! I'll see if I can reproduce using the above command line.

[HarelM]

Ok, so the problem in the above config goes away as you said when adding the car to the flag_encoders. The problem, IMO, is with the docs as this issue is stating, as I generally thought I'm doing the right things. I added the roads vehicle to flag_encoders and used it to add a custom model, I read all the md file relate to this topic in this repo but will was unable to make it work... I'll see if I can send a PR with docs improvements. Another "related" issue is the error I'm getting when running the code, which I'm not sure is in fact a docs issues but better errors. The error I got when upgraded to 6 and saw that car4wd is no longer supported with a very good error message. This was not the case with car_access, track_type etc. I'm not sure how this can be improved though...

[easbar]

You don't need to add roads to graph.flag_encoders. When the roads vehicle appears in a profile the roads tag parser (and the corresponding encoded values) is enabled automatically. For car_access you either need to add a profile that uses the car vehicle (following the same logic) or add it to graph.flag_encoders, yes, if for some reason you want to use car_access in the (roads) custom model, but do not want to add a car profile.

Anyway, I think from the documentation it should become clear that:

• there are some encoded values, like road_class, that are always there (added in the EncodingManager builder) and that can thus be used in the custom models
• there are other encoded values, like track_type, that need to be added to graph.encoded_values if they shall be used in the custom models
• and there are yet other encoded values (_access, _average_speed and *_priority to be exact) that need to be added by either adding a profile using the corresponding vehicle or via graph.flag_encoders (and btw graph.flag_encoders was renamed to graph.vehicles in master after the 6.x release).

So if you find a good spot in the documentation where this can be made more clear a PR would be nice of course.

And honestly, if you ask me, none of this makes sense and you can consider the current state as being halfway through a big refactoring.

IMO it would be more clear if there where:

a) no 'built-in' encoded values, but just one explicit list of encoded values so there is only a single point of truth for this. b) no 'vehicles': car/bike etc. tag parsers and the corresponding _access/average_speed/priority should just be treated the exact same way as the others.

... but we are still in the process of getting there.

Another "related" issue is the error I'm getting when running the code, which I'm not sure is in fact a docs issues but better errors. The error I got when upgraded to 6 and saw that car4wd is no longer supported with a very good error message. This was not the case with car_access, track_type etc. I'm not sure how this can be improved though...

I think the error message is rather clear:

'track_type' not available

It says what it is: The encoded value isn't there. So we need to add it via graph.encoded_values (or in case of car_access via graph.flag_encoders or a profile using the car vehicle). Of course it would be very instructive to state exactly this, something like:

'track_type' not available. You need to add it to `graph.encoded_values`.

The only problem is that I think the same code runs for custom models defined in config.yml (where server admins like to be told they need to add something to graph.encoded_values), and API/client-side users, who cannot change the server-side config. For them the best information would be telling them they should query the /info endpoint to check which encoded values are available. But maybe it is even possible to do the first or latter depending on where this code is called somehow (i.e. throw an exception and act differently when catching it at the place the custom model parsing code is called).

[HarelM]

I agree, the complexity of the configuration, and the probability to get something wrong is high, IMHO. In theory, I should define profiles with the relevant information and it should just "work". i.e. something like:

  profiles:
    - name: car4wd
      vehicle: roads
      weighting: custom
      expose: road_class,road_class_link,road_environment # this is new instead of a global configuration encoded_values, but if this is not provided, the below custom model should still work, i.e. add track_type and average_speed to encoded values when the graph is being built automatically, but maybe not expose it as part of the response, again, not super critical what to expose from my point of view, but just a different take on this.
      custom_model:
        block_fords: false # this is also missing, I'm not sure if this is supported or not as I couldn't find info on it in the docs, but I would expect it to be here, and not part of the flag_encoders configuration
        distance_influence: 1
        speed:
          - if: "track_type == GRADE4 || track_type == GRADE5"
            limit_to: 5
          - else: ""
            limit_to: "car_average_speed"
        priority:
          - if: "track_type != GRADE4 && track_type != GRADE5 && car_access == false"
            multiply_by: "0"

Without the flag_encoder and encoder_values. I might not see the whole picture or how complicated this would be to implement, but from an admin who sets up graphhopper, this is what I would expect.

[easbar]

Yes.. and no :) It is not that simple but I get it right now it is more complicated than necessary (it is basically work in progress). For things like block_fords you need to use graph.flag_encoders still. So for example the roads vehicle/flag_encoder is enabled automatically if you use it in a profile, but to modify it you currently need to make it explicit using graph.flag_encoders (or graph.vehicles in master).

[HarelM]

I understand that I need to use graph.flag_encoders and I'm able to update to version 6 now and it looks like everything is running as expected. So first and foremost thank you for the super fast response and valuable input! From my point of view, specifically for the block_fords I would expect this to be either the default or if I want it differently then I need to specify it in the custom_model part. I know this is historically the location it was in the configuration, and I know that it's still the case today, and I understand that this is a work in progress, all this is totally understood and fine. I'm just saying what I would expect if this didn't have a history, or how I would imagine it in next (or next, next, next) version - i.e. near/far future. Thanks again for all the hard work you guys are putting into this, I truly appreciate it!

[easbar]

Thanks for the kind words.

The block_fords parameter belongs to the (to be created) *_access tag parser in my opinion. That is one reason something like graph.encoded_values will be needed IMO, because tag parsers may have parameters and we need a place where we can specify them. They don't really belong into a single profile, because the resulting encoded values are 'global' and can be used by different profiles (that is kind of the reason they even exist).

Whether block_fords=true or false should be the default I don't know tbh but pretty sure this was discussed elsewhere (?).

[HarelM]

The defaults are fine, there's no need to change them. Just the ability to change them should be in the profile section or custom_model not in the flag_encoders, IMO. I'm not sure I'm fully aligned with the idea of having the encoded_values as a global configuration, why not specify this per-profile? Also, I'm not sure I understand how to use the *_access stuff. I just returned to the docs now to try and see if I find an example with them, but there aren't any. Assuming block fords would be added to this *_access tag parser I'm not sure I'd know how to use it in the custom model I'll need to define, should it be:

...
"if": true,
 "fords_access": true
...

?

[easbar]

I'm not sure I'm fully aligned with the idea of having the encoded_values as a global configuration, why not specify this per-profile?

The idea is to store encoded values (think: road attributes) for each edge/road so they can be used by arbitrary profiles. Storing profile-specific data per edge for each profile (non-global encoded values) is basically what we did with the old flag encoders approach. And since encoded values are global in this sense they need a global (not profile-specific) configuration syntax.

Also, I'm not sure I understand how to use the *_access stuff. I just returned to the docs now to try and see if I find an example with them, but there aren't any.

For example car_access should work exactly like roundabout, both are boolean encoded values and you can do for example: "if": "car_access == false", "limit_to": "50". This is mentioned here: https://github.com/graphhopper/graphhopper/blob/master/docs/core/custom-models.md#edge-attributes-used-by-graphhopper-encoded-values, but not sure if this was included in 6.x already.

Assuming block fords would be added to this *_access tag parser I'm not sure I'd know how to use it in the custom model I'll need to define, should it be:

For example if you specify graph.flag_encoders: car|block_fords=true the block_fords=true parameter will be passed to the CarTagParser which writes the car_access/average_speed/priority encoded values. So the block_fords option will simply be baked into the car_access encoded value. If you wanted to write more specific custom model rules that check if a given edge has the ford attribute you would have to add a corresponding encoded value and parser.

[HarelM]

Thanks for the info! This still remains too complicated... :-( While I get it (after you explained it to me), and I understand this probably adds the required flexibility it still feels "hackish" in lack of a better term... The extra parameter that is added to the flag_encoder (i.e |block_fords) doesn't feel right to me when you can now define a custom model. As I said, I understand the historical reasoning and the current "in progress refactoring" This is just my observation. :-)

[easbar]

Ok

[karussell]

Have improved the custom model files itself (under custom_models) and did some minor clean-up.

The extra parameter that is added to the flag_encoder (i.e |block_fords) doesn't feel right to me when you can now define a custom model.

Yes, indeed. With #2705 we will be able to write road_environment==FORD in the custom model to avoid/prefer/exclude fords. Same for private roads so the block_fords and block_private will become unnecessary.

[karussell]

Regarding create-new-flagencoder.md, I think we should just remove the entire file.

For the time-being let me shortly summarize how I would create a new "vehicle" entity (@easbar feel free to edit this comment :) ):

1. create the 2 or 3 new encoded values via DefaultVehicleEncodedValuesFactory (see VehicleEncodedValues)
2. create the necessary tag parsers via DefaultVehicleTagParserFactory (see VehicleTagParsers)
3. implement the required access parser - see e.g. FootAccessParser
4. implement the required average speed parser - see e.g. FootAverageSpeedParser. (It is recommended to set speeds for roads even if the access parser rejects them)
5. (implement the optional priority parser - see e.g. FootPriorityParser)

But before you create such a new vehicle you should think twice or ask in the forum if this is really necessary. Because often times it is already sufficient to use existing parsers in combination with a custom_model. And this approach is a much simpler regarding usage and also regarding future maintenance.

We'll also not accept new vehicles and instead try hard to make them possible with existing tools (see #2710). We go even the opposite direction and try to remove existing vehicles (#2759, #2668, #2651) until they are either reduced to something "simple" and "reusable" like car,bike+foot or even better completely resolved into a "pure" custom model approach.