Issue when connecting to matching server

[AlanBrisset]

Hello,

I'm trying to install a matching server on my computer. I followed every step of the wiki (installed Docker, Maven, Java JDK).

When I try the command java -jar target/barefoot-0.1.1-matcher-jar-with-dependencies.jar --geojson config/server.properties config/oberbayern.properties , I'm getting :

2017-12-18 14:41:38,553 INFO [main] ServerControl: initialize server 2017-12-18 14:41:38,558 INFO [main] ServerControl: read database properties from file config/oberbayern.properties 2017-12-18 14:41:38,559 INFO [main] Loader: load map from database oberbayern 2017-12-18 14:41:38,560 INFO [main] Loader: open road reader for database oberbayern at localhost:5432 2017-12-18 14:41:38,560 INFO [main] Loader: database.host=localhost 2017-12-18 14:41:38,560 INFO [main] Loader: database.port=5432 2017-12-18 14:41:38,560 INFO [main] Loader: database.name=oberbayern 2017-12-18 14:41:38,561 INFO [main] Loader: database.table=bfmap_ways 2017-12-18 14:41:38,561 INFO [main] Loader: database.user=osmuser 2017-12-18 14:41:38,561 INFO [main] Loader: database.road-types=./map/tools/road-types.json 2017-12-18 14:41:38,596 INFO [main] PostGISReader: open reader (standard) 2017-12-18 14:41:38,599 INFO [main] PostGISReader: open reader (parameterized) 2017-12-18 14:41:38,887 INFO [main] RoadMap: inserting roads ... 2017-12-18 14:41:38,892 INFO [main] PostGISReader: execute query 2017-12-18 14:41:38,984 ERROR [main] ServerControl: Executing PostgreSQL query failed: ERROR: relation "bfmap_ways" does not exist Position : 133 2017-12-18 14:41:38,984 INFO [Thread-0] ServerControl: stopping server 2017-12-18 14:41:38,985 ERROR [Thread-0] ServerControl: stopping server failed, not yet started

I tried some solutions from this issue thread : https://github.com/bmwcarit/barefoot/issues/44

psql -h localhost -p 5432 -d oberbayern osmuser Password for user osmuser: psql (9.5.10, server 9.3.20) SSL connection (protocol: TLSv1.2, cipher: DHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help.

oberbayern=> select count() from bfmap_ways; ERROR: relation "bfmap_ways" does not exist LINE 1: select count() from bfmap_ways; ^ oberbayern=>

Can you help me understand what I'm doing wrong?

[smattheis]

Could it be possible that you skipped the import of OSM data? See step 5 here: https://github.com/bmwcarit/barefoot#matcher-server-quick-start or respectively in more detail in the wiki step 4 of the setup of the map server here: https://github.com/bmwcarit/barefoot/wiki#setup Please check what the output of the import script says and feel free to get back if it still doesn't work.

[AlanBrisset]

Nope, it wasn't that ; I just realized that the database was filling itself (the terminal didn't tell anything, but the sudo docker exec -it barefoot-oberbayern bash -c 'sudo -u postgres psql showed me that it was growing and growing. I just had to wait a bit, and then the command worked. Thanks !

I have another problem however (completly different, I don't know if I should open another thread or not) : the matching server works with your example (the Bayern map) ; I'm trying to make it work with Portugal (Porto specifically). I did the same commands, except for the OSM import :

curl http://download.geofabrik.de/europe/portugal-latest.osm.pbf -o barefoot/map/osm/oberbayern.osm.pbf

(I chose to not rename the file/the docker container/etc, to not mess with the scripts or the config files).

When I'm submitting him my JSON file, it doesn't send back anything :

python util/submit/batch.py --host localhost --port 1234 --file src/test/resources/com/bmwcarit/barefoot/matcher/databis.json SUCCESS {"coordinates":[],"type":"MultiLineString"}

Here's what's in my JSON file :

[ {"time":"2014-09-10 06:54:07+200","id":1,"point":"POINT(-8.585658 41.148486)"}, {"time":"2014-09-10 06:54:08+200","id":1,"point":"POINT(-8.585658 41.148486000000005)"}, {"time":"2014-09-10 06:54:09+200","id":1,"point":"POINT(-8.585658 41.148486000000005)"}, {"time":"2014-09-10 06:54:10+200","id":1,"point":"POINT(-8.585658 41.148513)"}, {"time":"2014-09-10 06:54:11+200","id":1,"point":"POINT(-8.585658 41.148513)"}, {"time":"2014-09-10 06:54:12+200","id":2,"point":"POINT(-8.59212 41.16073)"}, {"time":"2014-09-10 06:54:13+200","id":2,"point":"POINT(-8.591499 41.161689)"}, {"time":"2014-09-10 06:54:14+200","id":2,"point":"POINT(-8.592093 41.161581000000005)"}, {"time":"2014-09-10 06:54:15+200","id":2,"point":"POINT(-8.593029000000001 41.162517)"}, {"time":"2014-09-10 06:54:16+200","id":2,"point":"POINT(-8.593488 41.162679000000004)"} ]

Is my JSON file badly written, or am I not understanding something?

[smattheis]

Did you create a new container for the map of Portugal? (Otherwise it would collide since you use the same name of the database, i.e., 'oberbayern'.) Apart from that, please check the output of the matcher server started with java -jar ... and if it doesn't give you a hint on how to resolve the problem please post the output in this thread. I can't see a concrete problem in your input data but I'm not sure but a leading zero for the time zone like +0200 could be required by the Java parsers and also id could be required to be a string although it is usually ignored by the matcher server. BTW: The id refers to the trace and putting multiple traces in a single JSON request (which you have as you use different ids within the same request) won't work as the matcher would interpret it as a single trace because - as said - it ignores it. (Only the real-time matcher requires the id.)

[AlanBrisset]

Yep, I'm creating a new container. After I tested with Bayern, I did : 1- sudo docker stop barefoot-oberbayern 2- sudo docker rm barefoot-oberbayern 3- deleting the barefoot folder 4- cloning the git and doing it with Portugal

This is the output of the java -jar ... : java -jar target/barefoot-0.1.1-matcher-jar-with-dependencies.jar --geojson config/server.properties config/oberbayern.properties 2017-12-18 15:11:47,695 INFO [main] ServerControl: initialize server 2017-12-18 15:11:47,698 INFO [main] ServerControl: read database properties from file config/oberbayern.properties 2017-12-18 15:11:47,713 INFO [main] Loader: load map from database oberbayern 2017-12-18 15:11:47,714 INFO [main] Loader: open road reader for database oberbayern at localhost:5432 2017-12-18 15:11:47,714 INFO [main] Loader: database.host=localhost 2017-12-18 15:11:47,714 INFO [main] Loader: database.port=5432 2017-12-18 15:11:47,714 INFO [main] Loader: database.name=oberbayern 2017-12-18 15:11:47,714 INFO [main] Loader: database.table=bfmap_ways 2017-12-18 15:11:47,715 INFO [main] Loader: database.user=osmuser 2017-12-18 15:11:47,715 INFO [main] Loader: database.road-types=./map/tools/road-types.json 2017-12-18 15:11:47,737 INFO [main] PostGISReader: open reader (standard) 2017-12-18 15:11:47,739 INFO [main] PostGISReader: open reader (parameterized) 2017-12-18 15:11:47,804 INFO [main] RoadMap: inserting roads ... 2017-12-18 15:11:47,805 INFO [main] PostGISReader: execute query 2017-12-18 15:11:47,830 INFO [main] RoadMap: inserted 0 (0) roads and finished 2017-12-18 15:11:47,830 INFO [main] PostGISReader: close reader 2017-12-18 15:11:47,840 INFO [main] RoadMap: ~1 megabytes used for road data (estimate) 2017-12-18 15:11:47,874 INFO [main] RoadMap: index and topology constructing ... 2017-12-18 15:11:47,907 INFO [main] RoadMap: index and topology constructed 2017-12-18 15:11:47,913 INFO [main] RoadMap: ~0 megabytes used for spatial index (estimate) 2017-12-18 15:11:47,917 INFO [main] ServerControl: read tracker properties from file config/server.properties 2017-12-18 15:11:47,972 INFO [main] MatcherServer: matcher.radius.max=200.0 2017-12-18 15:11:47,972 INFO [main] MatcherServer: matcher.distance.max=15000.0 2017-12-18 15:11:47,972 INFO [main] MatcherServer: matcher.lambda=0.0 2017-12-18 15:11:47,973 INFO [main] MatcherServer: matcher.sigma=4.5 2017-12-18 15:11:47,973 INFO [main] MatcherServer: matcher.threads=8 2017-12-18 15:11:47,973 INFO [main] MatcherServer: matcher.interval.min=1000 2017-12-18 15:11:47,973 INFO [main] MatcherServer: matcher.distance.min=0.0 2017-12-18 15:11:47,973 INFO [main] AbstractServer: server.port=1234 2017-12-18 15:11:47,974 INFO [main] AbstractServer: server.timeout.request=15000 2017-12-18 15:11:47,974 INFO [main] AbstractServer: server.timeout.response=60000 2017-12-18 15:11:47,976 INFO [main] AbstractServer: server.connections=20 2017-12-18 15:11:47,977 INFO [main] ServerControl: starting server on port 1234 with map oberbayern

Did it download any roadmap at all? Also, when I'm using the batch.py command, to send my JSON data, the first console (the server) writes this :

2017-12-18 15:43:56,811 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,811 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,823 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,823 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,823 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,824 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,824 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,824 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,824 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,825 INFO [Thread-5] Filter: HMM break - no state emissions 2017-12-18 15:43:56,827 INFO [pool-2-thread-12] MatcherServer: response processed in 61 ms

Do you want me to modify my JSON file, to erase the time zone, the ID or anything? Sorry to take your time, I'm a student and our group is using Barefoot for our work !

[smattheis]

It's great to hear that your group is using Barefoot. 👍 The log says "inserted 0 (0) roads and finished". Please have a look to the log output of the import script and please post it here if it doesn't give you a hint on what's wrong. Thanks.

[AlanBrisset]

Ahah yes, that's nice !

The log output of the import script is : (edit: I don't know why everything's bold like that, sorry)

root@4b742bd8d514:/# bash /mnt/map/osm/import.sh Start creation and initialization of database 'oberbayern' ... CREATE EXTENSION CREATE EXTENSION Done. Start creation of user and initialization of credentials ... CREATE ROLE GRANT Done. Start population of OSM data (osmosis) ... psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:4: NOTICE: table "actions" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:5: NOTICE: table "users" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:6: NOTICE: table "nodes" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:7: NOTICE: table "ways" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:8: NOTICE: table "way_nodes" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:9: NOTICE: table "relations" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:10: NOTICE: table "relation_members" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:11: NOTICE: table "schema_info" does not exist, skipping DROP TABLE psql:/mnt/map/osm/pgsnapshot_schema_0.6.sql:14: NOTICE: function osmosisupdate() does not exist, skipping DROP FUNCTION CREATE TABLE CREATE TABLE CREATE TABLE addgeometrycolumn

public.nodes.geom SRID:4326 TYPE:POINT DIMS:2 (1 row)

CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE INSERT 0 1 ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE CREATE INDEX CREATE INDEX CREATE INDEX ALTER TABLE ALTER TABLE ALTER TABLE CREATE FUNCTION CREATE FUNCTION ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE ALTER TABLE Dec 18, 2017 3:10:08 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.43.1-SNAPSHOT Dec 18, 2017 3:10:09 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. Dec 18, 2017 3:10:09 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. Dec 18, 2017 3:10:09 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion.

It seems clean to me ; the pbf file is about 170Mo, so the file download went well..

[AlanBrisset]

Ooooh I'm stupid ; I forgot to mention that the command mvn package doesn't work (we're getting so used to it that we don't even think about that anymore). We have to use mvn package -DskipTests to have a successful build.

[smattheis]

"INFO: Pipeline executing, waiting for completion." means that Osmosis has not finished the import. I think that takes a little while. To skip the tests for the build is fine. The tests rely on the 'oberbayern' map.

[AlanBrisset]

Oh, I misunderstood the sentence (I'm not a native english speaker), I thought it meant that it's waiting for the other commands ! So I have to wait for Osmosis to tell me that it's ready to continue?

[smattheis]

Osmosis is an external tool that is used to read the *.osm.pbf file into the database. After that, the import script runs a bunch of commands to create a routable map. Once this is done the last lines of the output will show something like this:

...
548009 segments from 306200 ways inserted and finished.
Done.
Done.

Feel free to get back if there are further questions.

[AlanBrisset]

Okay, I'm gonna retry it because I don't have any space list (I'm using a Virtual Machine), I'll keep you informed. Thanks a lot !

[AlanBrisset]

Hi again !

It worked perfectly, it generates me some GeoJSON that I can display on a map (I'm using https://mygeodata.cloud to do so). I have a question though : it seems that some trajectories are missing, and most of the time it's the small ones (~5-10meters trajectories). Does it come from Barefoot or from the website?

Also, sorry to ask you so many questions, but we're planning to integrate Barefoot into Hadoop. It's written on the Wiki that it can be easily integrated, but is there any tutorial/guide available?

[smattheis]

Do you have an example for such missing trajectories? (input data + picture)

I have never integrated Barefoot into Hadoop but into Spark, see here for some hints: https://github.com/bmwcarit/barefoot/wiki#scalable-map-matching (The trick is here to use a broadcast variable that holds a matcher instance on each machine and can be used for map matching job submissions.)

[AlanBrisset]

Input data (before Barefoot) :

[ {"time":"2014-09-10 06:54:07+200","id":1,"point":"POINT(-8.585658 41.148486)"}, {"time":"2014-09-10 06:54:08+200","id":1,"point":"POINT(-8.585658 41.148486000000005)"}, {"time":"2014-09-10 06:54:09+200","id":1,"point":"POINT(-8.585658 41.148486000000005)"}, {"time":"2014-09-10 06:54:10+200","id":1,"point":"POINT(-8.585658 41.148513)"}, {"time":"2014-09-10 06:54:11+200","id":1,"point":"POINT(-8.585658 41.148513)"}, {"time":"2014-09-10 06:54:12+200","id":2,"point":"POINT(-8.59212 41.16073)"}, {"time":"2014-09-10 06:54:13+200","id":2,"point":"POINT(-8.591499 41.161689)"}, {"time":"2014-09-10 06:54:14+200","id":2,"point":"POINT(-8.592093 41.161581000000005)"}, {"time":"2014-09-10 06:54:15+200","id":2,"point":"POINT(-8.593029000000001 41.162517)"}, {"time":"2014-09-10 06:54:16+200","id":2,"point":"POINT(-8.593488 41.162679000000004)"} ]

Output data (after Barefoot) :

{"coordinates":[[[-8.58568303941301,41.14848860941041],[-8.58568303941301,41.14848860941041]],[[-8.58568303941301,41.14848860941041],[-8.58568303941301,41.14848860941041]],[[-8.58568303941301,41.14848860941041],[-8.585678206670968,41.14851510577968]],[[-8.585678206670968,41.14851510577968],[-8.585678206670968,41.14851510577968]],[[-8.592094743206205,41.16072100050247],[-8.5920429,41.1608041],[-8.5919925,41.1608869],[-8.5919314,41.1610005],[-8.5919086,41.1610312],[-8.5918206,41.1611808],[-8.5915743,41.1615364],[-8.5915235,41.1616031],[-8.591466581433634,41.16167417192939]],[[-8.591466581433634,41.16167417192939],[-8.5914117,41.1617427],[-8.5914001,41.1617892],[-8.5913912,41.1618186],[-8.5913796,41.1618386],[-8.5913681,41.1618556],[-8.5913548,41.161879],[-8.5913558,41.1619008],[-8.5913719,41.1619198],[-8.5913941,41.1619281],[-8.591418,41.1619336],[-8.59144,41.1619314],[-8.5915312,41.1618859],[-8.591755,41.1616211],[-8.5918295,41.1615213],[-8.591884,41.1615063],[-8.5919408,41.1615018],[-8.5920029,41.1615104],[-8.592065,41.1615261],[-8.59211528339095,41.1615428936449]],[[-8.59211528339095,41.1615428936449],[-8.5922198,41.1615778],[-8.5924705,41.1616762],[-8.5926872,41.1617755],[-8.5927551,41.1618174],[-8.5928142,41.1618637],[-8.5928724,41.1619176],[-8.5929218,41.1619759],[-8.5929664,41.1620485],[-8.5930036,41.1621299],[-8.5930181,41.1622975],[-8.5930133,41.162411],[-8.5930057,41.1624838],[-8.593037334419677,41.162489220429315]],[[-8.593037334419677,41.162489220429315],[-8.593109,41.1625015],[-8.5933238,41.1625567],[-8.593513128033047,41.162585877019076]]],"type":"MultiLineString"}

Map : http://puu.sh/yJCyF/6a6d83801c.jpg

The trajectory with id=1 is supposed to be towards the North-West of the red bridge, on Porto Campanhã

Oh okay, thanks ! We're gonna try to adapt it to Barefoot (we have until Friday, but still have some problems with Hadoop configuration)

[smattheis]

As said before, you must split different trajectories into separate requests, i.e., into separate JSON array files issued separately to the matcher server, because the offline matcher ignores the id and handles it as a single trace such. That means, it tries to find a patch from your points with id 1 to points with id 2. Since this results in an HMM break, the result might not be what you would expect. Also I'm not sure what to expect when assembling multiple traces in a single request.

[AlanBrisset]

Hey again !

So, an update : Barefoot is working well, we did a Java program to split the different trajectories before sending them into Barefoot.

However, the project we have changed a little bit, as we've been told to implement Spark into Hadoop, and to make Barefoot interact with Spark, to parallelize the Map Matching. We've seen your guide in the Git Wiki, but, as we're completly discovering those technologies & have never done anything similar, we're struggling quite a bit. Could you help us by telling us what files we have to change, and, basically, how to connect both of them?

[smattheis]

I'm glad that it worked out for you. With regard to your question, what do you mean with "implement Spark into Hadoop, and to make Barefoot interact with Spark, to parallelize the Map Matching"? I cannot provide a detailed description of how to scale map matching in the plethora of scalable systems and technologies since this usually depends on a variety of factors specific to the application and use case. Nevertheless, for detailed questions I'm happy to help where I can.

[smattheis]

I'm closing this issue now. If there are more questions, please open a new issue.