search

Esri / arcgis-osm-editor

ArcGIS Editor for OpenStreetMap is a toolset for GIS users to access and contribute to OpenStreetMap through their Desktop or Server environment.
Apache License 2.0
395 stars 129 forks source link

"Compact" step needed in OSM File Loader for file geodatabase output #154

Closed mboeringa closed 7 years ago

mboeringa commented 8 years ago

Hi @ThomasEmge ,

This is only a minor issue, and easily fixable: I noticed the file geodatabases created by the OSM File Loader are considerably bloated after all the steps in the loading process and the Repair Geometry. My Netherlands extract resulted in a 47GB file geodatabase. Compacting reduced this to a mere 17GB. Since this is a very fast process as well, it is well worth integrating it. Maybe there is even the possibility to not only compact at the very end, but in intermediate steps where the Compact step would be appropriate and have any effect. This could then potentially reduce (intermediate) disk space requirements for very large extracts like e.g. loading a continent.

Even if compacting is only effective or sensible for the loading process at the very end, the reduction in final output size is welcome.

ThomasEmge commented 8 years ago

I looked at this issue over the past couple of weeks and found it to be a non-event. Starting from a clean geodatabase and loading the data I checked the size, then compacted and it was the same size. Since I am currently running tests with the Netherlands dataset I'll keep an eye on it but I would vote against it.

mboeringa commented 8 years ago

Starting from a clean geodatabase and loading the data I checked the size, then compacted and it was the same size.

That's weird, I created a fully new file geodatabase as well for this load session, and the size differences before and after Compact were as I reported them, although I must admit the difference was rather surprisingly big.

But this contrasts starkly with your experience... why?... is it possibly related to the issues with the loading of the multipolygon...?

Yes, please do keep an eye on it, and let me know your final size for the Netherlands just after the load (without Compact), and whether it is 47 or just under 17 GB.

Could you also report the timings for the loading of the relations and super-relations? I was also a bit surprised how fast these loaded (especially from the view of the "old" Load OSM File tool).

Mine were (using 3 cores): Relations: Done loading relations (Elapsed Time: 0 hours 4 minutes 29 seconds). Super-relations: Done loading super relations (Elapsed Time: 0 hours 1 minutes 10 seconds).

ThomasEmge commented 7 years ago

After loading the Netherlands osm file with the option to delete the supporting nodes (points), the file geodatabase is 10.1 GB. After running the compact geodatabase tool the file geodatabase is now 10.0 GB.

On my machine the relations loaded in 6 minutes and 15 seconds and the super-relations were completed in 1 minute and 30 seconds. Which in itself is not the complete time required as you should count the time it takes to append everything into a single feature class as well.

ThomasEmge commented 7 years ago

On a second country load (Poland) the file geodatabase is 7.84 GB after running the load tool and 7.8 GB after running the compact tool.