dbahrdt
commented
3 years ago Hi, I'll take a look into the crash as soon as possible (however I'm on vacation for the next 14 days and I don't know how much time I'll find to address this). Maybe you can try to take a look at https://github.com/dbahrdt/oscar-docker? This will give you a whole installation of oscar and oscar-web with which you can build the necessary files. The docker container also has support for routing. I'll try to come up with a small sample configuration so that you have something to work with.
Depending on what you're planing to do I do want to stress the fact that oscar-create needs a lot of memory if you intend to process larger files. In order to process the planet data set you'll need about 256 GiB of RAM. However you may find pre-processed files at http://data.oscar-web.de. The latest version is from February since I started porting all size related stuff to uint64_t due to OpenStreetMap containing too many items for oscar to process.
orblivion
I'm trying to get oscar-create to work for a sort of proof-of-concept project. I'm looking for any working configuration, even if it's very basic, and then experiment from there if I need any adjustments. I tried working with
sampleConfig.json, but I had issues with it, detailed below.So, what I'm wondering is, do you have a working set of config files handy, that you would not mind sharing? Or if you know what I did wrong with my config, that would work for me as well. Or, maybe the segfault is just an unrelated bug.
Thank you!
As for what I tried:
I started with
sampleConfig.json. It lists options, but I didn't see documentation for it, so I sort of had to guess as to how to get the simplest version to work. I have it successfully parsing now, but from there it ends in a segfault.My config.json
``` { "grid": { "enabled": false, "latcount": 100, "loncount": 100 }, "index": { "check": true, "deduplicate": true, "type": "simple" }, "rtree": { "config": { "latcount": 100, "loncount": 100 }, "enabled": false, "type": "gridbased" }, "stats": { "print-memory-usage": false }, "store": { "addParentInfo": false, "addRegionsToCells": false, "blobFetchCount": 1, "cellRefining": { "type": "none", "value": "cell-diag" }, "enabled": true, "fullRegionIndex": false, "geoclean": "none", "hashConfig": "auto", "itemFilter": "all", "nodeTableSize": 100000, "regionFilter": { "keyValues": "empty-obj.json", "keys": "empty.json" }, "sorting": { "order": "score" }, "splitValues": "empty.json", "tagFilter": "all", "threadCount": 0, "triangleRefining": { "simplify": "true", "type": "none", "value": "conforming" } }, "tempdir": { "fast": "/dev/shm/osmc", "slow": "./tempdir/osmc" }, "textsearch": { "config": { "_comment": "oomgeocell config", "cellLocalIds": false, "foreignObjects": false, "maxMemoryUsage": 16, "threadCount": 0, "tmpFileType": "fast file" }, "id": "unique-id-string", "type": "items" } } ```empty.json(used in myconfig.json):empty-obj.json(used in myconfig.json):My input file (~11MB):
https://download.geofabrik.de/north-america/canada/prince-edward-island-latest.osm.pbf
The output:
``` Selected Options: Stats config: print memusage: no IndexStoreConfig: input store: type: simple check: yes deduplicate: yes KVStoreConfig: Number of threads: 0 Number of blobs to fetch at once: 1 Max node table entries: 100000 Keys whose values are infalted: /home/user/oscar/build/empty.json Node HashTable size: auto ItemSaveDirector rule: everything Score config: Item sort order: score Read boundaries: no Keys defining regions=/home/user/oscar/build/empty.json Key:Values defining regions=/home/user/oscar/build/empty-obj.json FullRegionIndex: no Add parent info: no Add regions to cells they enclose: no Geometry cleaning: none min=0, max=0 Fetching residential areas without matching tags but with a place-node inside Residential area extraction: Ways' node-refs: 8830474|11678897=75.61% (1|0|1) Residential area extraction: Ways' node-refs: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Found 184 ways Residential area extraction: Need to fetch 3912 nodes Residential area extraction: Ways' nodes: 5381362|11678897=46.08% (1|1|2) Residential area extraction: Ways' nodes: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Assembling ways: 7566196|11678897=64.79% (1|0|1) Residential area extraction: Assembling ways: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Relation's ways: 10021346|11678897=85.81% (1|0|1) Residential area extraction: Relation's ways: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Found 12 relations with 59 ways Residential area extraction: Relation-ways' node-refs: 0 seconds for 11M 678K 897 Residential area extraction: Need to fetch 2315 nodes Residential area extraction: Relation-ways' nodes: 1354698|11678897=11.6% (1|7|8) Residential area extraction: Relation-ways' nodes: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Assembling relations: 556341|11678897=4.764% (1|19|20) Residential area extraction: Assembling relations: 1 seconds for 11M 678K 897 =11678897 1/s Residential area extraction: Assembled 184/184 ways and 12/12 relations GridRegionTree::create: 0 seconds for 100 OsmGridRegionTree::printStats--BEGIN GridRegionTree::printstats--BEGIN Nodes: 134 Grids: 34 ChildPtrs: 232 of which 42.67241379% are NULL LeafInfo: 288 of which 0% are enclosed Regions: 196 Real Storage usage: 6KiB 896NiB Storage usage: 7040 Containment tests: 0 of which -nan% were intersection tests GridRegionTree::printstats--END #points: 6632=155KiB 448NiB #GeoMultiPolygons: 55=3KiB 8NiB OsmGridRegionTree::printStats--END Collected 0 residential regions Region extraction: Ways' node-refs: 1615414|11678897=13.83% (1|6|7) Region extraction: Ways' node-refs: 9772077|11678897=83.67% (2|0|2) Region extraction: Ways' node-refs: 2 seconds for 11M 678K 897 =5839448 1/s Region extraction: Found 0 ways Region extraction: Need to fetch 0 nodes Region extraction: Ways' nodes: 6971169|11678897=59.69% (1|0|1) Region extraction: Ways' nodes: 1 seconds for 11M 678K 897 =11678897 1/s Region extraction: Assembling ways: 10880188|11678897=93.16% (1|0|1) Region extraction: Assembling ways: 1 seconds for 11M 678K 897 =11678897 1/s Region extraction: Relation's ways: 0 seconds for 11M 678K 897 Region extraction: Found 0 relations with 0 ways Region extraction: Relation-ways' node-refs: 1657109|11678897=14.19% (1|6|7) Region extraction: Relation-ways' node-refs: 1 seconds for 11M 678K 897 =11678897 1/s Region extraction: Need to fetch 0 nodes Region extraction: Relation-ways' nodes: 4578098|11678897=39.2% (1|1|2) Region extraction: Relation-ways' nodes: 1 seconds for 11M 678K 897 =11678897 1/s Region extraction: Assembling relations: 7465845|11678897=63.93% (1|0|1) Region extraction: Assembling relations: 1 seconds for 11M 678K 897 =11678897 1/s Region extraction: Assembled 0/0 ways and 0/0 relations GridRegionTree::create: 0 seconds for 100 OsmGridRegionTree::printStats--BEGIN GridRegionTree::printstats--BEGIN Nodes: 1 Grids: 1 ChildPtrs: 100 of which 100% are NULL LeafInfo: 0 of which -nan% are enclosed Regions: 0 Real Storage usage: 80NiB Storage usage: 80 Containment tests: 0 of which -nan% were intersection tests GridRegionTree::printstats--END #points: 0=0 NiB #GeoMultiPolygons: 0=0 NiB OsmGridRegionTree::printStats--END OsmTriangulationRegionStore: extracting points...done OsmTriangulationRegionStore: extracting segments...done Found 0 different points creating 0 different segments Converting points to CGAL points...done OsmTriangulationRegionStore: creating triangulation...took 2us Setting cellids: 0 seconds for 0 Refining cells...done Found 1 cells Finding relevant regions: 139774|11678897=1.197% (1|82|83) Finding relevant regions: 4441979|11678897=38.03% (2|3|5) Finding relevant regions: 11519249|11678897=98.63% (3|0|3) Finding relevant regions: 3 seconds for 11M 678K 897 =3892965 1/s Found 1 cells containing items GridRegionTree::create: 0 seconds for 100 OsmGridRegionTree::printStats--BEGIN GridRegionTree::printstats--BEGIN Nodes: 1 Grids: 1 ChildPtrs: 100 of which 100% are NULL LeafInfo: 0 of which -nan% are enclosed Regions: 0 Real Storage usage: 80NiB Storage usage: 80 Containment tests: 0 of which -nan% were intersection tests GridRegionTree::printstats--END #points: 0=0 NiB #GeoMultiPolygons: 0=0 NiB OsmGridRegionTree::printStats--END Creating final TriangulationRegionStore OsmTriangulationRegionStore: extracting points...done OsmTriangulationRegionStore: extracting segments...done Found 0 different points creating 0 different segments Converting points to CGAL points...done OsmTriangulationRegionStore: creating triangulation...took 3us Could not re-add 0 edges Setting cellids: 0 seconds for 0 Total time to create the region store with 0 regions: 16s 600ms 107us Fetching strings: 964429|11678897=8.258% (1|11|12) Fetching strings: 4441979|11678897=38.03% (2|3|5) Fetching strings: 10320048|11678897=88.36% (3|0|3) Fetching strings: 3 seconds for 11M 678K 897 =3892965 1/s Finalizing string table...143 msecs Inserting region store items Fetching multipolygon items: Relation's ways: 6078117|11678897=52.04% (1|0|1) Fetching multipolygon items: Relation's ways: 1 seconds for 11M 678K 897 =11678897 1/s Fetching multipolygon items: Found 571 relations with 15024 ways Fetching multipolygon items: Relation-ways' node-refs: 9122435|11678897=78.11% (1|0|1) Fetching multipolygon items: Relation-ways' node-refs: 10880188|11678897=93.16% (2|0|2) Fetching multipolygon items: Relation-ways' node-refs: 3 seconds for 11M 678K 897 =3892965 1/s Fetching multipolygon items: Need to fetch 715381 nodes Fetching multipolygon items: Relation-ways' nodes: 5550469|11678897=47.53% (1|1|2) Fetching multipolygon items: Relation-ways' nodes: 1 seconds for 11M 678K 897 =11678897 1/s Fetching multipolygon items: Assembling relations: 1992160|11678897=17.06% (1|4|5) Fetching multipolygon items: Assembling relations: 3 seconds for 11M 678K 897 =3892965 1/s Fetching multipolygon items: Assembled 0/0 ways and 571/571 relations Processing file: 139774|11678897=1.197% (3|247|250) Processing file: 964429|11678897=8.258% (4|44|48) Processing file: 2956852|11678897=25.32% (5|14|19) Processing file: 4900599|11678897=41.96% (6|8|14) Processing file: 9122435|11678897=78.11% (7|1|8) Processing file: 9529932|11678897=81.6% (9|2|11) Processing file: 10320048|11678897=88.36% (12|1|13) Processing file: 10691905|11678897=91.55% (13|1|14) Processing file: 11125186|11678897=95.26% (15|0|15) Processing file: 11228466|11678897=96.14% (16|0|16) Processing file: 11519249|11678897=98.63% (18|0|18) Processing file: 11542394|11678897=98.83% (19|0|19) Processing file: 19 seconds for 11M 678K 897 =614678 1/s Took 5 rounds to process the file Sorting items... Sorting items took 114ms 303us Pruned 0 empty cells CellCreator: creating GeoHierarchy for 0 regions out of 1 cells Creating GeoRegionGraph: 0 seconds for 0 Found 1 different cells Maximum cell-split: 0 Maximum children: 0 Constructing Hierarchy sserialize::spatial::GeoHierarchy--stats-BEGIN Hierarchy is empty sserialize::spatial::GeoHierarchy--stats-END GeoHierarchy passed the consistency check. OsmKeyValueObjectStore::stats::begin Items: 177228 Keys: 814 Values: 22274 Boundary: GeoRect[(45.58351225, -66.37246048); (52.22285217, -55.27018571)] OsmKeyValueObjectStore::stats::end Time to create KeyValueStore: 50s 923ms 870us Serializing KeyValueStore Serializing OsmKeyValueObjectStore payload...took 0 seconds DynamicKeyValueObjectStore::serialize: Serializing string tables...took 0 seconds KeyValueObjectStore::serialize: items: 0 seconds for 177K 228 terminate called after throwing an instance of 'sserialize::VersionMissMatchException' what(): VersionMissMatchException (want=4, have=90): Static::Array Aborted ```