Figure out a way to setup functional testing for generated routes.

[aerostitch]

Issue migrated from trac ticket # 1268

component: tools | priority: minor

2014-10-25 14:06:38: @pgrandin created the issue

---

I've been thinking of building a list of small segments where we know we have had issues in the past, to ensure that we can do some functional testing using for example the gpx output. We can then be notified if a change in the routing or navigation has an impact on the generated route ( wether intended or not ).

We have added the possibility to export a generated route as GPX via DBUS, so checking for regressions / improvements is easily scriptable now, as we just need to diff two files.

Any thoughts welcome on this topic. I think we could maybe start a list somewhere as long as it is easily downloadable by a script so that it can be replayed. We could use a file in the code for that, but I'd like users to be able to participate without requiring write access to the repo, or having a dev needing to commit it for them. A wiki page maybe ? I was also thinking of a github repo with automatic commit of the gpx so we can analyze the history of these routes.

The point is that it should be easy for everyone to contribute. People reporting routing issues in trac should also be able to add an example.

[aerostitch]

2014-10-25 14:09:40: @pgrandin edited the issue description

[aerostitch]

2014-10-25 14:10:50: @pgrandin edited the issue description

[aerostitch]

2014-10-25 14:30:46: @mvglasow commented

---

How would you run these tests?

• as a test suite after each build on the server (and then compare against output of previous build to see if any routes have changed)?
• on (user's) demand?

In the first case we might want to keep the test cases to a set of representative ones, thus I'd prefer have a dev look over any new tests that come in and either reject duplicates or replace old test cases with newer ones.

In the second case we can have a somewhat more open interface. Maybe a web form into which users enter their starting point and destination, and then compare results? (Just realized that in this case we'd basically be providing a route planning service...)

We also need to consider map updates. On one hand, we need somewhat up-to-date map data (else users will come to us with test cases involving roads that aren't in the map data we use). On the other hand, map updates can also change routing, thus updating too frequently defeats the purpose of being able to compare. In the case of a post-build test suite, we would need to update the baseline data after each map update.

I really like the visualization you posted in #660 – how did you visualize the GPX file?

[aerostitch]

2014-10-25 14:52:27: @pgrandin commented

---

Thanks for you insights.

My first idea was to use a service such as cloudbees or drone.io ( which I really like, but unfortunately does not support SVN, even if we can easily figure out a way around this) to do CI. Our servers are already quite busy with the map processing, and both cloudbees and drone.io are free for FLOSS projects. Plus it would be a service that we don't need to maintain ourself.

Your point of reviewing the tests case to ensure they are representative is valid. Providing a route planning service could be interesting too, especially since it could be cheap (in terms of development) to setup.

About map updates, we could have an optional map version tag attached to a set of coordinates. Since each item in a map is usually around 2MB, your question lead me to wonder if we could put maps in a Github repo as flat files instead of zip and then we could be able to fetch a map at a given point in time. Only question then is the maximal size of a repo.

The test route was generated using gpsvisualizer.com, but I'm almost done setting up a similar service for our own needs using [https://github.com/Leaflet/Leaflet ]leaflet] with a GPX extension.

[aerostitch]

2014-10-26 14:18:42: @pgrandin commented

---

I did some tests, and here is an example of what we can achieve quite easily : geojson file stored in Github

Modifying the gpx export function to export also geojson should be trivial. Users could then submit the files to web ui, doing some basic validation ( ensuring that the file is indeed a valid geojson) and then we could push the files into a repository automatically (would be useful for bug reports).

We can also generate these files for the CI part. The benefit over GPX is that Github will render the map directly using leaflet and that the maps are browsable, with the announcement displayed when you click on a marker.

[aerostitch]

2014-10-26 14:18:42: @pgrandin

[aerostitch]

2014-11-09 13:01:52: @mvglasow commented

---

In the meantime, I have built a simple test suite consisting of a bash script and a text file. It can be found at ​https://github.com/mvglasow/navit-goodies/tree/master/scripts/routetest.

While testing announcements I was made aware that the icon displayed by the nav_next_turn OSD and the verbal instruction are generated separately. We should therefore include both in the GPX file; the icon name could go into <type>. I'm currently working on that.

[aerostitch]

2014-11-09 13:30:50: @mvglasow commented

---

<type> is implemented in 5938.

[aerostitch]

2015-04-01 09:43:12: @pgrandin commented

---

I have made some progress on this topic.

I've imported mvglasow's tests and wrote a simple python script that runs each test case and produce a junit-compatible report.

The code : https://github.com/navit-gps/routing-qa Example result : https://circleci.com/gh/navit-gps/routing-qa/36

Example test case :

# Tests for missing or unnecessary announcements
# Ammerseestr
from:
  lat: 48.1217347
  lng: 11.5026473
to:
  lat: 48.1237784
  lng: 11.484457

success:
    source: 'dbus'
    item: 'status'
    value: 33
    operator: '=='

We will need to expand the success criterion, and maybe also specify a map or bbox for each test. Currently we can test for a dbus call result, or perform an xpath query on a gpx export.

[aerostitch]

2015-12-31 05:30:35: @pgrandin

[aerostitch]

2017-12-02 04:08:44: @jkoan

[aerostitch]

2017-12-02 04:08:44: @jkoan commented

---

This ticket was pushed back in order to bring 0.5.1 out soon.