kartoza / geosafe

InaSAFE package for Geonode

GNU General Public License v3.0

7 stars 16 forks source link

Estimate job size and communicate to user #495

Open gubuntu opened 5 years ago

gubuntu commented 5 years ago

problem

Despite performance improvement introduced by InaSAFE 4 and aggregation and a well-resourced server, some analyses still take many hours, even on single or sets of small aggregation polygons.

from @cgiovando:

so we need to figure out some better warning system or extent limiting mechanism, otherwise INGC is going to run into serious usability issues.

solution

from @cgiovando:

Can you think of some formula that takes into consideration all factors determining analysis duration? (e.g number of features for vector layers, resolution for rasters, etc) We should then wire that into the UI either as feedback to the user or to automatically limit large analysis.

lucernae commented 5 years ago

I don't have any formula in mind now. But this kind of estimate should be built into InaSAFE. From GeoSAFE side, we can only estime the size of the extent (like what we did previously). Or feature counts (for vectors).

There is also that problem where QGIS crash for subsequent analysis because of SIGSEGV (and for this I'm not sure how to fix this), which affects job time as well.

cgiovando commented 5 years ago

I'm wondering if a combination of gdalinfo and ogrinfo (with -so and -spat switches) queries can be run dynamically with each AOI? Not sure if this needs to got back to InaSAFE, or just activated independently with a call from GeoSAFE? The feedback doesn't have to be immediate, even 5-10 seconds after it's launched, would be much better than waiting several hours without any estimate.

I also noticed that both analysis (168 and 170) finished exactly at the same time, even if they had a different combination of input layers. So maybe that was an issue with QGIS crashing as you mentioned?

timlinux commented 5 years ago

Just note that:

compute times will also depend on how many concurrent jobs are running
compute times will depend on the complexity of geometries. For example, if we have a boundary or flood polygon layer with many vertices doing the zonal stats, intersection/union etc. operations takes longer.

I'm thinking what other approaches we can take to give a reasonable indication of compute time.