OSM File Loader (Load only) tool fails on Africa extract

[mboeringa]

I just attempted to import a recently downloaded Geofabrik Africa extract a second time with the OSM File Loader (Load only) tool. Both attempts now failed at the Append step just after the loading of the nodes, about half way through the Append step.

I have successfully imported Africa last year, but one year on, the extract is without doubt again considerably bigger. With over 280M nodes, is Africa still importable into a single File Geodatabase?

Note that I have plenty of free disk space (>1TB on a 2TB SSD), so there is no limit there.

Has ESRI done some recent power testing importing the different continents, and can possibly confirm whether or not Africa can still be imported with the tool?

I will attempt a second download of the Africa extract from Geofabrik, to be sure it's not related to the extract itself, and report back if I still get a failure.

[mboeringa]

Thomas,

I realized that during this import, I used my C: Windows system drive as scratch workspace. Even though my C drive has plenty of spare space (>200 GB), I have seen other large extracts fail if my C drive was involved as either scratch workspace, or when it was used as the final location of the file geodatabase created as output.

I have no idea why this happens, as said, it is not a disk space issue. The drive is a reliable Samsung SSD. It just fails on large extracts if the Windows system drive is involved in the import. Smaller extracts do work. What resource runs out on the system drive is a mystery to me.

Anyway, I restarted the Africa import against an external 2TB SSD drive, and will report back if the import succeeds (which it likely will, based on previous experience). The size of Africa can not be the problem, it is only marginally bigger than the France extract, which I have previously imported without issues.

[mboeringa]

Hmm... I have now attempted to re-load the Africa extract twice, ensuring no involvement of my C: Windows system drive, neither as Default, Scratch or final destination file geodatabase. All were set to an external 2TB SSD with plenty of space.

Still, both re-loads failed about half way through the Append step after loading the nodes.

@ThomasEmge, can you confirm ESRI is capable to load the current Africa extract of Geofabrik?

[mboeringa]

More food for thought. I have now attempted to Append two of the remaining node file geodatabases that were left over after the failed import. Each of these contains just over 95M records.

I can view and open them in ArcCatalog without issues.

However, trying to append the two datasets using the Append geoprocessing tool from within ArcCatalog, so without any involvement the Editor, also results in failure somewhere at the end of the processing. See the attached image.

Maybe some bug in the Append geoprocessing tool, or still some corruption in the created datasets? Maybe I also need to try this in Pro... could be a memory limitation as well.

[ThomasEmge]

Marco, I finished loading Africa without any issues. I have a single 1TB disc in my machine and the download was from geofabrik from 6/2/2017. The file gdb is roughly 170 GB in size.

[mboeringa]

Marco, I finished loading Africa without any issues. I have a single 1TB disc in my machine and the download was from geofabrik from 6/2/2017. The file gdb is roughly 170 GB in size.

OK, thanks @ThomasEmge ,

I will try it yet another time then. I will also re-download the extract from Geofabrik. With all the multipolygon fixes gone in due to Jochen's multipolygon fixing challenge, OpenStreetMap rendering is considerably improving.

[mboeringa]

@ThomasEmge ,

My last attempt of loading Africa again failed at the Append step for the nodes :-(

I have no idea what is going wrong: plenty of space, plenty of RAM free. And you are able to load it...

What version of ArcMap are you on? I am running 10.5 now. Did you use the official 10.5 release of the toolbox, or some development version you are working on? Did you have only a single session of ArcMap open during the whole time of the loading process? I routinely have multiple ArcMap sessions open... I have tried to make sure no locking issue should occur by avoiding accessing the drive altogether after starting the loading process.

[ThomasEmge]

I am running the 10.5 version of desktop and the released version of the editor. I had the tool loading over the weekend using ArcCatalog with background gp enabled and the background process running as 32bit. No other classic desktop application was running at the time.

[mboeringa]

I am running the 10.5 version of desktop and the released version of the editor. I had the tool loading over the weekend using ArcCatalog with background gp enabled and the background process running as 32bit. No other classic desktop application was running at the time.

@ThomasEmge . My last two attempts also failed. One difference is that I am running this as foreground process. I do customarily have multiple sessions of ArcMap open, as I currently do not have the luxury of a spare machine to do this.

I will now do a last attempt using background processing enabled and see if that fails as well.

Do note I have been importing other (but smaller) country extracts recently with success, so this is not a generic all fail situation.

[ThomasEmge]

@mboeringa Just finished loading Africa as a foreground process. The loading was done in ArcMap and occasionally I had multiple instances of ArcMap and/or ArcCatalog open (and Pro as well). All instances were working on their own data sources and I didn't encounter any issues.

[mboeringa]

@mboeringa Just finished loading Africa as a foreground process. The loading was done in ArcMap and occasionally I had multiple instances of ArcMap and/or ArcCatalog open (and Pro as well). All instances were working on their own data sources and I didn't encounter any issues.

@ThomasEmge : what OS are you using on that machine? I am currently over to Windows 10. I have now attempted to load France on another machine with also W10, and it also fails at the Append step.

NOTE: small datasets do work! It is just the very large extracts that I see failing.

[ThomasEmge]

The machines I am using for testing are using Windows 10 Enterprise Version 1703 build 15063.540

[mboeringa]

@ThomasEmge ,

A few new observations:

• If I take the separate left-over File Geodatabases after failure of loading with the OSM File Loader (Load Only) tool and that contain the nodes (I used parallel processing of course, so there are multiple), and try to Append in ArcMap, it fails. The Append of the node file geodatabases errors out somewhere half way.
• However, after reading on GeoNet, that some people actually had better experience with running some tools from ArcCatalog, I have now started loading France again using ArcCatalog and not ArcMap. The tool currently has passed the stage of Append and is indexing the node feature class! So that is good news. It is likely it will finish with this session started from ArcCatalog. I'll let you know when it does.
• Another observation is that if I try to Append the left-over file geodatabases from a failed load session in ArcGIS Pro, it also succeeds!

So it definitely seems some resource is running out when attempting to load very large OSM extracts from ArcMap...

Do you have any idea what may be causing the difference between ArcCatalog and ArcMap? Of course, ArcMap may be consuming more RAM already from the start, but I don't think I actually have seen it going that high with loading.

[ThomasEmge]

There isn't really much of a difference between ArcMap and ArcCatalog when geoprocessing is involved. I can't recall any code attempting to distinguish between the application. There might be more of a difference if you are doing the loading procedure as a foreground or as a background process. But you would definitely see resources being 'drained', such as ArcMap running out of memory when the loading runs a gp foreground process (if there is an issue).

[mboeringa]

There isn't really much of a difference between ArcMap and ArcCatalog when geoprocessing is involved. I can't recall any code attempting to distinguish between the application. There might be more of a difference if you are doing the loading procedure as a foreground or as a background process. But you would definitely see resources being 'drained', such as ArcMap running out of memory when the loading runs a gp foreground process (if there is an issue).

@ThomasEmge,

Hmmm... that leaves a big question mark as to why the Append (Data Management) fails in ArcMap versus ArcCatalog. Anyway, this is likely not an issue with the Editor, but a generic issue in ArcMap if anything, based on these latest observations.

I can now also confirm that the Geofabrik France extract loaded successfully! It finished in just over 28 hours using 8 parallel processes on a hyper threaded Core i7 laptop.