Openlr Performance Issue

tomtom-international / openlr

Reference implementation of OpenLR. A map agnostic, dynamic location referencing method.

Other

90 stars 30 forks source link

Openlr Performance Issue #33

Open sudhaharv77 opened 4 years ago

sudhaharv77 commented 4 years ago

Hi,

We are developing the traffic project, we are getting the traffic data from tomtom(https://traffic.tomtom.com/tsq/hdf/ARE-HDF-OPENLR/149d7921-d010-43cd-8e0d-83e5df700500/content.proto) as of now URL is not open for us. we are using the sample proto data.

Here the issue is, when we try to decode the data using openlr(openlr.decoder.OpenLRDecoder;) its consuming more time to return response( example 1 sec for 1 record).

We have almost 43000 records needs to process approx. its takes almost 40 mins which run by 200 threads.

Could you please help to sort out issue.

Java Sample:

ByteArray openLRByteArray = new ByteArray((byte[]) trafficFlow.getLocation().getOpenlr().toByteArray()); LocationReference locRef = new LocationReferenceBinaryImpl("", openLRByteArray); Location decodedLoc = new OpenLRDecoder().decode(decParam, locRef);

Thanks,

ghost commented 4 years ago

I have a few questions about the issue:

What map implementation are you using to decode the records?
Are you using the maploader-tt-sqlite module and an associated SQLite database?
If so, how large is the map? Can you perform a count of the records in the Line and Node tables?

The map implementation in the maploader-tt-sqlite was designed as a simple test map format for getting started. For your task I would suggest implementing the MapDatabase interface using a proper spatial database.

sudhaharv77 commented 4 years ago

Hi, Thanks for your valuable response, Please spare some time to read my comments

fyi info we are below dependencies

Capture

Please find my answers,

What map implementation are you using to decode the records?

[ ] We are getting the MapData list based on Zone from our DB(Data is coming from GEO db)

Capture

- [ ] openlr.map.MapDatabase ojbect is created as below Capture

- [ ] prepare openlr decode param using below Capture

- [ ] finally decoding as below.

finall results is

Capture

Are you using the maploader-tt-sqlite module and an associated SQLite database?

Yes

openlr-mapLoader tt-sqlite 1.3.0

If so, how large is the map? Can you perform a count of the records in the Line and Node tables?

Capture Capture

ghost commented 4 years ago

Thanks for posting the info! Could you try reducing the number of threads from 200 down to 10? I think the number of threads is too high and they are likely competing for resources. This might increase the throughput.

You seem to be getting throughput of about (40m 60s 1000ms / 43000 records) = 55 milliseconds per record. The OpenLR sqlite database was not really designed for high-performance but rather as a simple example implementation of a map to get started with OpenLR. I think you are just hitting on the limits of sqlite. You could try, for example, implementing MapDatabase using an in-memory copy of the map. This would perform much better.

sudhaharv77 commented 4 years ago

Hello Stephen,

Thanks for you valuable feed back. Yes as you said, we can achieve our goal if sqlite file minimal. As per our records(43000 min) its takes long time to process with 32gig machine. We need to implement In memory concept as you suggest.

i could not found any API's support for IN MEMORY with openlr.

Could you please help us to provide any snippet or steps to implement IN MEMORY.

Thanks.

ghost commented 4 years ago

You are right. We don't have a in-memory implementation of the map database. This is something we would like to do in the future. To achieve this:

the following classes would need to be implemented openlr.map.MapDatabase, openlr.map.Line and openlr.map.Node.
All lines and nodes of the map would be loaded into memory. Spatial indexes would need to be built for all lines and all nodes to enable quick searching around a point. An RTree implementation such as https://locationtech.github.io/jts/javadoc/org/locationtech/jts/index/strtree/STRtree.html might be good for this task.
Indexes would also need to be built to enable quick retrieval of line and node by ID.
Some geometrical operations would need to be implemented such as getGeoCoordinateAlongLine. A library such as JTS could also be used here to reduce the implementation work.

sudhaharv77 commented 4 years ago

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

duel-adn commented 3 years ago

Hello @sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

sudhaharv77 commented 3 years ago

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q .

duel-adn commented 3 years ago

Hi Sudhahar,

We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area).

If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches.

Can you share some information about your project?

best,

antonio

On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 @.***> wrote:

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-806516262, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q .

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

sudhaharv77 commented 3 years ago

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM ( https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx )

We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map

TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min,

I need help from you to compete the whole process with in 2 min, Or any other method to do

i have used below snippet to process the tomtom inputstream object

[image: image.png]

Regards Sudhahar

On Thu, Mar 25, 2021 at 6:36 PM antonio di noto @.***> wrote:

Hi Sudhahar,

We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area).

If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches.

Can you share some information about your project?

best,

antonio

On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 @.***> wrote:

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/tomtom-international/openlr/issues/33#issuecomment-806516262 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q

.

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-806541054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q .

sudhaharv77 commented 3 years ago

Hi Antonio,

Additional information for the decode below,

[image: image.png]

Regards Sudhahar

On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < @.***> wrote:

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM ( https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx )

We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map

TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min,

I need help from you to compete the whole process with in 2 min, Or any other method to do

i have used below snippet to process the tomtom inputstream object

[image: image.png]

Regards Sudhahar

On Thu, Mar 25, 2021 at 6:36 PM antonio di noto @.***> wrote:

Hi Sudhahar,

We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area).

If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches.

Can you share some information about your project?

best,

antonio

On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 @.***> wrote:

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/tomtom-international/openlr/issues/33#issuecomment-806516262 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q

.

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-806541054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q .

sudhaharv77 commented 3 years ago

Hi Could you please help to update on my Queries

Regards Sudhahar

On Tue, Mar 30, 2021, 4:03 PM Sudhahar Vaithilingam < @.***> wrote:

Hi Antonio,

Additional information for the decode below,

[image: image.png]

[image: image.png]

Regards Sudhahar

On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < @.***> wrote:

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM ( https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx )

We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map

TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min,

I need help from you to compete the whole process with in 2 min, Or any other method to do

i have used below snippet to process the tomtom inputstream object

[image: image.png]

Regards Sudhahar

On Thu, Mar 25, 2021 at 6:36 PM antonio di noto @.***> wrote:

Hi Sudhahar,

We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area).

If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches.

Can you share some information about your project?

best,

antonio

On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 @.***> wrote:

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/tomtom-international/openlr/issues/33#issuecomment-806516262 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q

.

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-806541054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q .

duel-adn commented 3 years ago

Hello, we better continue this discussion in private, my e-mail is: a.dinoto at dueltv dot com, can you send me yours?

On Mon, Apr 26, 2021 at 1:21 PM sudhaharv77 @.***> wrote:

Hi Could you please help to update on my Queries

Regards Sudhahar

On Tue, Mar 30, 2021, 4:03 PM Sudhahar Vaithilingam < @.***> wrote:

Hi Antonio,

Additional information for the decode below,

[image: image.png]

[image: image.png]

Regards Sudhahar

On Tue, Mar 30, 2021 at 2:55 PM Sudhahar Vaithilingam < @.***> wrote:

Hi Antonio, Let me explain our project, We are doing Advanced Traffic Management System(ATMS) for GULF country, we already got the API to get live traffic data from TOMTOM (

https://traffic.tomtom.com/tsq/hdf-detailed/ARE-HDF_DETAILED-OPENLR/xxxxxxxxx )

We are calling every 2 mins to get updated traffic speed data from the above URL to update our GIS map

TOMTOM providing 55k records as encrypted format then we are decoding each time using OPENLR ( https://github.com/tomtom-international/openlr/tree/master/decoder) We are unable to complete the whole process with in 2 min,

I need help from you to compete the whole process with in 2 min, Or any other method to do

i have used below snippet to process the tomtom inputstream object

[image: image.png]

Regards Sudhahar

On Thu, Mar 25, 2021 at 6:36 PM antonio di noto @.***> wrote:

Hi Sudhahar,

We achieve these performances because we've developed our own graph database specifically for this task and connected it to the OpenLR implementation on github. We're using it with TomTom's Multinet of a large portion of the world (~700M edges) and we decode 6k to 50k of OpenLR binary identifiers per second (it depends on the density of the graph in the area).

If you are interested, we could offer a simple REST APIs for decoding OpenLR in batches.

Can you share some information about your project?

best,

antonio

On Thu, Mar 25, 2021 at 10:58 AM sudhaharv77 @.***> wrote:

Hi Antonio

Yes I am interested, could please share more details

Regards Sudhahar

On Thu, Mar 25, 2021, 5:14 PM antonio di noto @.***> wrote:

Hello @sudhaharv77 https://github.com/sudhaharv77, we have a very fast OpenLR decoder implementation (6k-50k identifiers per second on a 4 core machine with 16GB of RAM). If you are still interested, you can contact me.

Best, antonio

Hi stephencurran,

Thanks again,

Sorry for the late reply, i understand that openlr do not have in-memory implementation is it any road map to implement in-memory in future?

Can you suggest instead Java, is there any other technologies to process the data faster with Openlr.

Regards Sudhahar

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/tomtom-international/openlr/issues/33#issuecomment-806489119

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AHZBLFHBYZELLMV3WFAVLIDTFL5HDANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <

https://github.com/tomtom-international/openlr/issues/33#issuecomment-806516262

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAVO23RT3GVXMUXKFJDLXQDTFMCKZANCNFSM4JVW3L7Q

.

-- Antonio Di Noto, CTO Duel S.p.A. via del Poggio Laurentino, 66 - 00144 Roma Tel. +39 06.5195951 Linkedin http://it.linkedin.com/in/antoniodinoto/ http://www.dueltv.com Please consider the environment before printing this e-mail

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/tomtom-international/openlr/issues/33#issuecomment-806541054 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHZBLFGWQ6U4CZUNFTMJVBTTFMG2VANCNFSM4JVW3L7Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tomtom-international/openlr/issues/33#issuecomment-826755393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVO23RDO5BTUXCYQH7VUJLTKVECDANCNFSM4JVW3L7Q .