pgr_createtopology get out of memory after 8M edges

[ohadmanor]

When I try to run pgr_createtopology for large area (North America) for example or any other with more the 8.5 milion edges the prosses fail with "Unexpected error out of memory" or "Unexpected error parallel worker failed to initialize"

I'm running the following command (from bug #1342 https://github.com/pgRouting/pgrouting/issues/1342)

with starts as (select generate_series as start_objectid from generate_series(0, (select count(*) from osm_routing), 1000000) ) select pgr_createtopology('osm_routing', 0.00001, 'way', 'gid', rows_where:= 'gid >= ' || starts.start_objectid || ' and gid < ' || starts.start_objectid + 1000000) from starts;

SELECT version(); PostgreSQL 12.4, compiled by Visual C++ build 1914, 64-bit

SELECT postgis_full_version(); POSTGIS="3.0.2 3.0.2" [EXTENSION] PGSQL="120" GEOS="3.8.1-CAPI-1.13.3" PROJ="Rel. 5.2.0, September 15th, 2018" LIBXML="2.9.9" LIBJSON="0.12" LIBPROTOBUF="1.2.1" WAGYU="0.4.3 (Internal)"

SELECT pgr_version(); 3.1.0

Thanks

[dkastl]

Have you tried to run the topology function on a subset of your data?

As the last example in the documentation shows, you don't need to recreate the topology for the whole table: https://docs.pgrouting.org/latest/en/pgr_createTopology.html#additional-examples

I can't guarantee this works, but with osm2pgrouting for example I break the network into parts and it allows to import large networks.

[ohadmanor]

Daniel Hi, thanks it seems to work, my quastion are:

1. why? or more accurate how?
2. rows_where is it metter if it 6 or 2 or 100000?
3. next command I'm useing nodeNetwork - to do the same or there is some other trick?

and again, thanks!

Ohad

[dkastl]

I don't understand all your questions, but basically you can create the topolgy on a subset of your network data. And if you increase your network the topology function is able to add missing topology information.

Be careful with creating nodes on line intersections. These intersections might be bridges. For questions like this I recommend to write to the pgRouting mailing list or ask on StackExchange: http://pgrouting.org/support.html