search

tilemill-project / tilemill

TileMill is a modern map design studio
https://tilemill-project.github.io/tilemill/
BSD 3-Clause "New" or "Revised" License
3.12k stars 527 forks source link

Missing features when reading FileGDB at high zooms #2275

Open springmeyer opened 10 years ago

springmeyer commented 10 years ago

A bug exists whereby sometimes when you zoom into a FileGDB in TileMill geometries that previously rendered at lower zoom levels disappear randomly.

I've replicated this issue on Windows and OSX, but I need time to dig deeper. I think this is a problem with how TileMill/Mapnik is interacting with the in memory quad tree that the OpenFileGDB driver creates.

This bug is worked around in current TileMill after I disabled in memory indexing via OPENFILEGDB_IN_MEMORY_SPI in: https://github.com/mapbox/tilemill/commit/0f90e13d769a9f044cdb12a56cd23849f8daf2ca

This ticket stands to track a larger fix so that indexing in memory can be eventually be re-enabled.

rcoup commented 10 years ago

Where did the FGDB come from? https://trac.osgeo.org/gdal/ticket/4455 caused us all sorts of subtle features-disappearing issues when moving data between platforms

springmeyer commented 10 years ago

I've been able to replicate with a couple different files. I'm concerned that its actually a bug in Mapnik's ogr plugin whereby we make assumptions about thread safety that do not hold true in terms of accessing the OGRLayer* pointer. It may also be related to https://github.com/mapnik/mapnik/issues/2048.

If you want to try to replicate grab the data from https://www.mapbox.com/tilemill/docs/guides/filegdb-work/ then revert https://github.com/mapbox/tilemill/commit/0f90e13d769a9f044cdb12a56cd23849f8daf2ca and zoom in and shapes will randomly disappear.

springmeyer commented 10 years ago

Here's what I see:

filegdb-win

rcoup commented 10 years ago

In our experience the ESRI FileGDB library isn't threadsafe -- it's supposed to be safe if you open new FGDB sources in different threads (for reading), but I've seen some weird behaviour that suggests it's not. Sharing the same source between different threads is definitely a bad idea.

Can you open/zoom around the layer in ArcMap? Is the data geographic or projected? I'm still a bit suspicious about the internal spatial index being messed up.

rcoup commented 10 years ago

Ah, OpenFileGDB? #ignoreItAll :)

springmeyer commented 10 years ago

Right. Yeah, I originally tried to add support through the FileGDB API, but no end of locking, reference counting, or avoidance of shared state would prevent crashes. I scoured all possible forums I could fine before concluding that the FileGDB API is not thread safe.

supposed to be safe if you open new FGDB sources in different threads (for reading), but I've seen some weird behaviour that suggests it's not.

Exactly my experience too. There must be some global state they don't document that cause thread unsafe access. I wonder if its the file locking. I gave up on this and now that OpenFileGDB is here this works perfectly with the small exception of the in-memory spatial indexes (which is what this ticket is about).

springmeyer commented 10 years ago

https://github.com/mapbox/tilemill/commit/0f90e13d769a9f044cdb12a56cd23849f8daf2ca https://github.com/OSGeo/gdal/commit/3e1f8e02afde186851644ce4a3e898f2b9c06abf