Closed j3r3m1 closed 9 months ago
Tested on my laptop (CORE I7), I'm not able to reproduce this issue. The data is formated in less than 4 minutes. Perhaps a ghost process that has remained active on the H2GIS db. This happens when an error has been encountered in a previous process. Can you run a test with a fresh instance of GeoClimate and a new H2GIS db?
The H2GIS db is not new as long as I start a new command line with default db informations ? Anyway I have rerun the following config file and get to the same problem: the algo is still on the same SQL query 20 minutes after:
15:04:44.816 [main] INFO class org.orbisgis.geoclimate.osmtools.OSMTools -- The input OSM file has been loaded in the database. 15:04:44.816 [main] DEBUG class org.orbisgis.geoclimate.osm.OSM -- Create the building layer 15:04:44.827 [main] DEBUG class org.orbisgis.geoclimate.osmtools.OSMTools -- Start polygons transformation 15:04:44.827 [main] DEBUG class org.orbisgis.geoclimate.osmtools.OSMTools -- Indexing osm tables... 15:05:03.989 [main] DEBUG class org.orbisgis.geoclimate.osmtools.OSMTools -- Build way polygons
Here is the config file
{
"description": "Processing OSM data",
"input": {
"locations": ["Nice"]
},
"output": {
"folder": "/home/bernardj/Software/GeoClimate/"
},
"parameters": {
"rsu_indicators": {
"indicatorUse": [
"LCZ",
"TEB",
"UTRF"
],
"svfSimplified": true,
"estimateHeight": true
},
"grid_indicators": {
"x_size": 100,
"y_size": 100,
"rowCol": false,
"output" : "geojson",
"indicators" :[
"BUILDING_FRACTION",
"BUILDING_HEIGHT",
"WATER_FRACTION",
"VEGETATION_FRACTION",
"ROAD_FRACTION",
"IMPERVIOUS_FRACTION",
"LCZ_FRACTION"
]
}
}
}
It seeems that there is an issue with the execution plan.
The endless SQL query is :
SELECT 'w'||a.id_way AS id, a.the_geom , MAX(CASE WHEN b.tag_key = 'aeroway' THEN b.tag_value END) AS \"aeroway\",MAX(CASE WHEN b.tag_key = 'agricultural' THEN b.tag_value END) AS \"agricultural\",
" FROM WAYS_POLYGONS AS a, OSM b WHERE a.id_way=b.id_way and st_isempty(a.the_geom)=false AND b.TAG_KEY IN ('height','roof:height','building:levels','roof:levels','building','amenity','layer','aeroway','historic','leisure','monument','wall')
"and a.the_geom && st_setsrid('POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'::geometry, 32632)
and st_intersects(a.the_geom,st_setsrid('POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'::geometry, 32632)) GROUP BY a.id_way;
the explain plan returns
SELECT
'w' || "A"."ID_WAY" AS "ID",
CASE WHEN "PUBLIC"."ST_ISVALID"("A"."THE_GEOM") THEN "A"."THE_GEOM" ELSE "PUBLIC"."ST_MAKEVALID"("A"."THE_GEOM") END AS "THE_GEOM",
MAX(CASE WHEN "B"."TAG_KEY" = 'aeroway' THEN "B"."TAG_VALUE" END) AS "aeroway",
MAX(CASE WHEN "B"."TAG_KEY" = 'agricultural' THEN "B"."TAG_VALUE" END) AS "agricultural"
FROM "PUBLIC"."OSM" "B"
/* PUBLIC.INDEX_132: TAG_KEY IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall') */
/* WHERE B.TAG_KEY IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall')
*/
INNER JOIN "PUBLIC"."WAYS_POLYGONS" "A"
/* PUBLIC.INDEX_4F: THE_GEOM && PUBLIC.ST_SETSRID(GEOMETRY 'POLYGON ((471333.21004455804 5538649.866669843, 471333.21004455804 5560944.562704275, 485726.139629124 5560944.562704275, 485726.139629124 5538649.866669843, 471333.21004455804 5538649.866669843))', 32632) */
ON 1=1
WHERE ("B"."TAG_KEY" IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall'))
AND ("A"."ID_WAY" = "B"."ID_WAY")
AND ("A"."THE_GEOM" && "PUBLIC"."ST_SETSRID"(GEOMETRY 'POLYGON ((471333.21004455804 5538649.866669843, 471333.21004455804 5560944.562704275, 485726.139629124 5560944.562704275, 485726.139629124 5538649.866669843, 471333.21004455804 5538649.866669843))', 32632))
AND ("PUBLIC"."ST_ISEMPTY"("A"."THE_GEOM") = FALSE)
AND "PUBLIC"."ST_INTERSECTS"("A"."THE_GEOM", "PUBLIC"."ST_SETSRID"(GEOMETRY 'POLYGON ((471333.21004455804 5538649.866669843, 471333.21004455804 5560944.562704275, 485726.139629124 5560944.562704275, 485726.139629124 5538649.866669843, 471333.21004455804 5538649.866669843))', 32632))
GROUP BY "A"."ID_WAY"
If the geometry filter is removed the explain plan returns
SELECT
'w' || "A"."ID_WAY" AS "ID",
CASE WHEN "PUBLIC"."ST_ISVALID"("A"."THE_GEOM") THEN "A"."THE_GEOM" ELSE "PUBLIC"."ST_MAKEVALID"("A"."THE_GEOM") END AS "THE_GEOM",
MAX(CASE WHEN "B"."TAG_KEY" = 'aeroway' THEN "B"."TAG_VALUE" END) AS "aeroway",
MAX(CASE WHEN "B"."TAG_KEY" = 'agricultural' THEN "B"."TAG_VALUE" END) AS "agricultural"
FROM "PUBLIC"."OSM" "B"
/* PUBLIC.INDEX_132: TAG_KEY IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall') */
/* WHERE B.TAG_KEY IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall')
*/
INNER JOIN "PUBLIC"."WAYS_POLYGONS" "A"
/* PUBLIC.INDEX_4: ID_WAY = B.ID_WAY */
ON 1=1
WHERE ("B"."TAG_KEY" IN('height', 'roof:height', 'building:levels', 'roof:levels', 'building', 'amenity', 'layer', 'aeroway', 'historic', 'leisure', 'monument', 'wall'))
AND ("A"."ID_WAY" = "B"."ID_WAY")
AND ("PUBLIC"."ST_ISEMPTY"("A"."THE_GEOM") = FALSE)
GROUP BY "A"."ID_WAY"
This second query runs in few seconds.
@katzyn Could you please take a look and give us your point of view.
The complete test
Statement stat = connection.createStatement();
//Prepare data
stat.execute("DROP TABLE IF EXISTS WAYS_POLYGONS, OSM;" +
"CREATE TABLE WAYS_POLYGONS AS SELECT 1 as ID_WAY, 'POLYGON ((160 280, 240 280, 240 140, 160 140, 160 280))'::GEOMETRY as the_geom;" +
"CREATE TABLE OSM AS SELECT 1 as ID_WAY, 'aeroway' AS TAG_KEY, 'aeroway' AS TAG_VALUE; " +
"CREATE INDEX ON WAYS_POLYGONS(ID_WAY);" +
"CREATE SPATIAL INDEX ON WAYS_POLYGONS(THE_GEOM);" +
"CREATE INDEX ON OSM(ID_WAY);" +
"CREATE INDEX ON OSM(TAG_VALUE);" +
"CREATE INDEX ON OSM(TAG_KEY)");
ResultSet res = stat.executeQuery("EXPLAIN SELECT 'w'||a.id_way AS id, a.the_geom , MAX(CASE WHEN b.tag_key = 'aeroway' THEN b.tag_value END) AS \"aeroway\",MAX(CASE WHEN b.tag_key = 'agricultural' THEN b.tag_value END) AS \"agricultural\"," +
" FROM WAYS_POLYGONS AS a, OSM b WHERE a.id_way=b.id_way and st_isempty(a.the_geom)=false AND b.TAG_KEY IN ('height','roof:height','building:levels','roof:levels','building','amenity','layer','aeroway','historic','leisure','monument','wall') " +
"and a.the_geom && st_setsrid('POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'::geometry, 32632) " +
"and st_intersects(a.the_geom,st_setsrid('POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'::geometry, 32632)) GROUP BY a.id_way;" +
" ");
res.next();
System.out.println("\n With spatial query \n"+ res.getString(1));
res = stat.executeQuery("EXPLAIN SELECT 'w'||a.id_way AS id, a.the_geom , MAX(CASE WHEN b.tag_key = 'aeroway' THEN b.tag_value END) AS \"aeroway\",MAX(CASE WHEN b.tag_key = 'agricultural' THEN b.tag_value END) AS \"agricultural\"," +
" FROM WAYS_POLYGONS AS a, OSM b WHERE a.id_way=b.id_way and st_isempty(a.the_geom)=false AND b.TAG_KEY IN ('height','roof:height','building:levels','roof:levels','building','amenity','layer','aeroway','historic','leisure','monument','wall') " +
" GROUP BY a.id_way;");
res.next();
System.out.println("\n Without spatial query \n"+ res.getString(1));
@katzyn any idea about ? If not, @ebocher should we try to segment the query into severals ?
It looks like a wrong estimation of a query execution cost.
Index hints can help, but please post a complete standalone (Java / JDBC / SQL only) test case in a new issue here:
https://github.com/h2database/h2database/issues H2GIS may not be used in a test case too.
st_setsrid('POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'::geometry, 32632)
can be replaced with GEOMETRY 'SRID=32632;POLYGON ((198.5 186.5, 269.5 186.5, 269.5 115, 198.5 115, 198.5 186.5))'
.
st_intersects
can probably be replaced with &&
operator.
Done @katzyn Many thanks in advance
I will propose a temporary workaround.
GeoClimate does not seem to be able to run the following SQL query: https://github.com/orbisgis/geoclimate/blob/185f6b419a8a39e5a655d85b6dd4dc067539d917/osmtools/src/main/groovy/org/orbisgis/geoclimate/osmtools/Transform.groovy#L352
It never stops working on it. The SQL query is:
I have investigated the data, it looks OK, I have not found obvious reason for this anormally long calculation. @ebocher any idea what to investigate ? If not I can try to reduce the query size (keeping only some of the tags) to identify where does the problem come from.