Maps with lot of items not being plotted

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.

Other

941 stars 218 forks source link

Maps with lot of items not being plotted #1427

Closed patrickdalla closed 1 year ago

patrickdalla commented 1 year ago

Maps with lot of items is not being plotted. A case with 31K items to be plotted not working was reported by Wilson Serpa. When he filter the items to be plotted, the maps works.

Maybe some alert should be sent to the user asking to filter better the items to be plotted.

lfcnassif commented 1 year ago

Is there an intentional limit or is this a bug? Can it be increased maybe optimizing code?

patrickdalla commented 1 year ago

Is a bug. If I am not mistaken, there once was a message popup that warned about too many results, asking for the user to fiter the resultset. Reenabling, or creating, it could be a quick first option. From a forensic analysis perspective, I see no sense in plotting such a large number of items. I think some optimizations can be done, but maybe it can be limitations from leaflet or jfx webview.

lfcnassif commented 1 year ago

We can add a pop up warning user about current limitation. But I'm wondering if there is something quadratic to the number of items in our java or javascript code...

lfcnassif commented 1 year ago

I was able to reproduce this. I spent some time investigating, but I have no conclusions... There is something very strange here, since the map is plotted after very few seconds, but without any locations, instead of hanging indefinitely (it's the behavior I was expecting), so I don't think it is just a performance issue.

I guess the issue is in javascript code. @patrickdalla since you know this part of the code base much more then me, I would appreciate a lot if you could investigate this.

patrickdalla commented 1 year ago

ok.

Em seg., 21 de nov. de 2022 18:02, Luis Filipe Nassif < @.***> escreveu:

I was able to reproduce this. I spent some time investigating, but I have no conclusions... There is something very strange here, since the map is plotted after very few seconds, but without any locations, instead of hanging indefinitely (it's the behavior I was expecting), so I don't think it is just a performance issue.

I guess the issue is in javascript code. @patrickdalla https://github.com/patrickdalla since you know this part of the code base much more then me, I would appreciate a lot if you could investigate this.

— Reply to this email directly, view it on GitHub https://github.com/sepinf-inc/IPED/issues/1427#issuecomment-1322708388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG247S5L5OMMX5FUKQC33E3WJPWQPANCNFSM6AAAAAAR5VQ6GM . You are receiving this because you were mentioned.Message ID: @.***>

patrickdalla commented 1 year ago

Redesign in data exchange mechanism between Java and JFX Webkit was implemented on MapWithoutKMLexchange branch. The older approach was to send KML with all georeferenced items, leading to memory allocation problems by JFX Webkit for big KML string.

The new approach is faster, but as it involves much of the code, more tests must be done to detect any possible regression error. I asked @FelipeFcosta to help me on tests.

patrickdalla commented 1 year ago

I have implemented the same approach for google maps tile server. But the performance on this implementation still isn't good.

patrickdalla commented 1 year ago

For the leaflet implementation, I had very good response for resultset up to 65k georeferenced items on my workstation. Above this it begins to present some lags in plotting and mouse event response. But it was still reasonably acceptable for my case when plotting all 188K georeferenced items. For google maps, results set above 3K start lagging. I am reviewing the code to find point where it can be optimized.

patrickdalla commented 1 year ago

Google maps implementation improved. But it is still slower than leaflets implementation with other maps tiles source.

patrickdalla commented 1 year ago

After some research, I could found a leaflet plugin that uses maps from GoogleMaps as tile source. So only leaflet api integration is used. Old googlemaps api code can be removed (as it is used transparently by the leaflet plugin). This keeps code maintanance simpler. And the performace is way better than the old KML exchange implementation. On my workstation, on a case with 190K georeferenced items, all of them were plotted in less than 10 seconds. The navigation is also very good.

lfcnassif commented 1 year ago

Excellent!! Simpler maintenance is great! Thank you very much @patrickdalla!

PS: new performance is also great!

patrickdalla commented 1 year ago

I've proposed a PR. Future ideas for possible issues:

I think some performance improvement can be achieved by moving the clusterization logic from leaflet plugin, to java code.
Clusterization could follow politic/administrative boundaries rules. For example, in a zoom level that show all brazil states, if two items have near georeferences but one is RJ state and the other on ES, both near the border between such states, they would be clustered in the corresponding state cluster, in spite of their proximity. This feature depends on data of political entities boundaries.

As these features demand considerable effort, I just mentioned for further consideration.

lfcnassif commented 1 year ago

Agreed!