oom osmosis_polygon_intersects

[Marc-marc-marc]

https://buildbot.osmose.openstreetmap.fr/#/builders/554/builds/978/steps/3/logs/stdio 2023-06-09 23:24:04 norway_vest_agder : osmosis_polygon_intersects 2023-06-09 23:24:05 run osmosis all analyser Analyser_Osmosis_Polygon_Intersects 2023-06-09 23:24:05 requires table multipolygons 2023-06-10 00:19:34 error: error on analyse osmosis_polygon_intersects... 2023-06-10 00:19:34 Traceback (most recent call last): 2023-06-10 00:19:34 File "/data/project/osmose/backend/./osmose_run.py", line 275, in execc 2023-06-10 00:19:34 analyser_obj.analyser() 2023-06-10 00:19:34 File "/data/project/osmose/backend/analysers/Analyser_Osmosis.py", line 285, in analyser 2023-06-10 00:19:34 self.requires_tables_build(self.requires_tables_common) 2023-06-10 00:19:34 File "/data/project/osmose/backend/analysers/Analyser_Osmosis.py", line 370, in requires_tables_build 2023-06-10 00:19:34 self.giscurs.execute(self.sql_create_multipolygons.format(self.config.db_schema.split(',')[0], self.config.options.get("proj"))) 2023-06-10 00:19:34 File "/data/project/osmose/.local/lib/python3.9/site-packages/psycopg2/extras.py", line 146, in execute 2023-06-10 00:19:34 return super().execute(query, vars) 2023-06-10 00:19:34 psycopg2.OperationalError: server closed the connection unexpectedly 2023-06-10 00:19:34 This probably means the server terminated abnormally 2023-06-10 00:19:34 before or while processing the request.

some other extract having the same issue : https://buildbot.osmose.openstreetmap.fr/#/builders/83/builds/974/steps/3/logs/stdio https://buildbot.osmose.openstreetmap.fr/#/builders/639/builds/1002/steps/3/logs/stdio

[Famlam]

The fix for #1891 is already live here?

Does the OOM error occur in the building of table multipolygons or in the analyser itself? (Analyser polygon_small seems to work fine in the same run, and that also uses multipolygons)...

[Marc-marc-marc]

yes this fix is on the server a new run with overallocation disabled show the query that fail https://buildbot.osmose.openstreetmap.fr/#/builders/660/builds/1012/steps/3/logs/stdio

2023-06-10 14:56:08   run osmosis base analyser Analyser_Osmosis_Polygon_Intersects
2023-06-10 14:56:08   requires table highways
2023-06-10 14:56:41   requires table multipolygons
2023-06-10 18:53:52   error: error on analyse osmosis_polygon_intersects...
2023-06-10 18:53:52     Traceback (most recent call last):
2023-06-10 18:53:52       File "/data/project/osmose/backend/./osmose_run.py", line 265, in execc
2023-06-10 18:53:52         analyser_obj.analyser_resume(remote_timestamp, already_issued_objects)
2023-06-10 18:53:52       File "/data/project/osmose/backend/analysers/Analyser_Osmosis.py", line 343, in analyser_resume
2023-06-10 18:53:52         self.analyser_change()
2023-06-10 18:53:52       File "/data/project/osmose/backend/analysers/Analyser_Osmosis.py", line 310, in analyser_change
2023-06-10 18:53:52         self.requires_tables_build(self.requires_tables_common)
2023-06-10 18:53:52       File "/data/project/osmose/backend/analysers/Analyser_Osmosis.py", line 370, in requires_tables_build
2023-06-10 18:53:52         self.giscurs.execute(self.sql_create_multipolygons.format(self.config.db_schema.split(',')[0], self.config.options.get("proj")))
2023-06-10 18:53:52       File "/data/project/osmose/.local/lib/python3.9/site-packages/psycopg2/extras.py", line 146, in execute
2023-06-10 18:53:52         return super().execute(query, vars)
2023-06-10 18:53:52     psycopg2.errors.OutOfMemory: out of memory
2023-06-10 18:53:52     DETAIL:  while in LLVM
2023-06-10 18:53:52     CONTEXT:  SQL statement "WITH
2023-06-10 18:53:52                     unary AS (
2023-06-10 18:53:52                         SELECT
2023-06-10 18:53:52                             id, tags,
2023-06-10 18:53:52                             (ST_Dump(poly)).geom AS poly
2023-06-10 18:53:52                         FROM (VALUES (
2023-06-10 18:53:52                             mp.id,
2023-06-10 18:53:52                             mp.tags,
2023-06-10 18:53:52                             ST_BuildArea(mp.linestrings)
2023-06-10 18:53:52                         )) AS t(id, tags, poly)
2023-06-10 18:53:52                     ),
2023-06-10 18:53:52                     simplified AS (
2023-06-10 18:53:52                         SELECT
2023-06-10 18:53:52                             id, tags,
2023-06-10 18:53:52                             ST_BuildArea(ST_Collect(
2023-06-10 18:53:52                                 ST_ExteriorRing(poly),
2023-06-10 18:53:52                                 (SELECT ST_Union(ST_InteriorRingN(poly, n)) FROM generate_series(1, ST_NumInteriorRings(poly)) AS t(n))
2023-06-10 18:53:52                             )) AS poly
2023-06-10 18:53:52                         FROM
2023-06-10 18:53:52                             unary
2023-06-10 18:53:52                     ),
2023-06-10 18:53:52                     multi AS (
2023-06-10 18:53:52                         SELECT
2023-06-10 18:53:52                             id, tags,
2023-06-10 18:53:52                             ST_CollectionHomogenize(ST_Collect(poly)) AS poly
2023-06-10 18:53:52                         FROM
2023-06-10 18:53:52                             simplified
2023-06-10 18:53:52                         GROUP BY
2023-06-10 18:53:52                             id, tags
2023-06-10 18:53:52                     )
2023-06-10 18:53:52                     INSERT INTO
2023-06-10 18:53:52                         hungary.multipolygons
2023-06-10 18:53:52                     SELECT
2023-06-10 18:53:52                         *,
2023-06-10 18:53:52                         ST_Transform(poly, 32633) AS poly_proj,
2023-06-10 18:53:52                         ST_IsValid(poly) AS is_valid
2023-06-10 18:53:52                     FROM
2023-06-10 18:53:52                         multi
2023-06-10 18:53:52                     WHERE
2023-06-10 18:53:52                         -- Avoid dealing with very large multi-polygons
2023-06-10 18:53:52                         ST_NPoints(poly) < 100000"
2023-06-10 18:53:52     PL/pgSQL function inline_code_block line 38 at SQL statement
2023-06-10 18:53:52     server closed the connection unexpectedly
2023-06-10 18:53:52         This probably means the server terminated abnormally
2023-06-10 18:53:52         before or while processing the request.

it seems needed to reduce the max points

[frodrigo]

I try it on an other server (osm110) and it runs fine in few minutes.

[frodrigo]

I try it on an other server (osm110) and it runs fine in few minutes.

Same thing once osm110 migrated to pg13.