search

ngageoint / hootenanny

Hootenanny conflates multiple maps into a single seamless map.
GNU General Public License v3.0
353 stars 74 forks source link

.GDB takes significantly longer than .shp to ingest. #13

Closed mikejeffe closed 8 years ago

mikejeffe commented 9 years ago

Question:.gdb upload times are very long compared to shapefile upload times. It was actually faster to load the .gdb up into ArcMap, export the data to shapefiles and load it into hootenanny then it was to try and load the .gdb directly into it.

mikejeffe commented 9 years ago

@tturk we will look into this issue and investigate to see if there is any unusual latency when ingesting .gdb datasets. Note that some of longer ingests could be attributed to the loading of superflous data contained within the fgdb itself. With shapefiles you are generally loading one feature type per layer whereas .gdb you get > 1.

tturk commented 9 years ago

Ok thank you.

jasonsurratt commented 8 years ago

@mjeffe98 Could you either create or point @mingjiesu to some data sets that produce this issue in the UI?

@mingjiesu Would you recreate this issue at the command line and post some benchmarks using the command line. You can find the relevant commands in scripts/makeetl. I know @mattjdnv has experienced slower reading of FileGDB vs. Shapefile and it may just be due to the drivers.

Thanks!

mikejeffe commented 8 years ago

@jasonsurratt I will try to reproduce on some local data and send something to @mingjiesu

mikejeffe commented 8 years ago

@tturk I have not been able to reproduce any additional latency associated with fgdb uploads. I ran a test of several files comparing shapefiles and the equivalent in fgdb (basically taking the same shapefile and importing it into a fgdb than comparing the ingest time). I compared files that ranged in size from 1MB to 100MB and observed no significant difference (With 100MB range files FGDB's took about 5-10 seconds longer). Let me know if there is a specific dataset that you have been working with and we can test that to be certain.