search

Toblerity / Fiona

Fiona reads and writes geographic data files
https://fiona.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.15k stars 202 forks source link

Copying a sources schema, crs and river with **ds.meta does not work #53

Closed prologic closed 11 years ago

prologic commented 11 years ago

I've found the following code does not work:

#!/usr/bin/env python

"""Tool to simplify geometries in a vector file"""

import sys

import fiona

from shapely import speedups
from shapely.geometry import mapping, shape

speedups.available and speedups.enable()

def main():
    if len(sys.argv) < 3:
        print "Usage: fixgeometries <infile>"
        raise SystemExit(1)

    with fiona.open(sys.argv[1], "r") as ds:
        items = ((feature, shape(feature["geometry"])) for feature in ds)

        with fiona.open(sys.argv[2], "w", **ds.meta) as sink:
            for feature, geometry in items:
                id = feature.get("properties", {}).get("OBJECTID", None) or feature.get("id", None)

                if not geometry.is_valid:
                    print "Geometry of Feature id {0:s} is not valid. Fixing...".format(str(id))
                    geometry = geometry.buffer(0.0)

                geometry = geometry.simplify(0.01)
                if not geometry.is_valid:
                    geometry = geometry.buffer(0.0)

                feature["geometry"] = mapping(geometry)

                sink.write(feature)

if __name__ == "__main__":
    main()

The resulting output ESRI Shapefile does not contain any of the original properties nor does the output schema match the original.

The only way I've found that does work is to do something like:

with fiona.open(..., "w", crs=ds.crs, driver=ds.driver, schema=ds.schema.copy()) as sink:

Is this something that can be fixed in Fiona?

Documentation tends to lead me to believe that one could and should use **ds.meta as a nice convenient way to use the input source's schema when you're doing things like cleaning up geometries in a Shapefile, etc...

prologic commented 11 years ago

Turns out doing schema=ds.schema.copy() doesn't work on some Shapefiles either. I processes two Shapefiles. One preserved the properties and schema while the other did not.

prologic commented 11 years ago

Here are links to the two troublesome Shapefile(s) in question:

sgillies commented 11 years ago

What Fiona version are you using? I'm unable to reproduce the problem (using the head of the Fiona repo, 1.2.18 to-be):

>>> with fiona.open('/', vfs='zip:///Users/seang/Downloads/STE11aAust.zip', layer=u'STE11aAust') as c:
...     with fiona.open('/tmp/foo.shp', 'w', **c.meta) as d:
...         d.write(next(c))
...
>>>

The output:

krusty-2:~ seang$ ogrinfo -so /tmp/foo.shp foo
INFO: Open of `/tmp/foo.shp'
      using driver `ESRI Shapefile' successful.

Layer name: foo
Geometry: 3D Polygon
Feature Count: 1
Extent: (140.999475, -37.505060) - (159.109219, -28.157020)
Layer SRS WKT:
GEOGCS["GRS 1980(IUGG, 1980)",
    DATUM["unknown",
        SPHEROID["GRS80",6378137,298.257222101]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
STATE_CODE: String (1.0)
STATE_NAME: String (50.0)

Would appreciate if you could provide the smallest possible script that produces the problem.

prologic commented 11 years ago

So doing the same thing with the other Shapefile IBRA7.zip:

>>> import fiona
>>> with fiona.open("/", vfs="zip:///Users/s2092651/ccaih/ccav/data/.sources/regions/IBRA7.zip", layer=u"IBRA7_regions") as c:
...     with fiona.open("/tmp/foo.shp", "w", **c.meta) as d:
...             d.write(next(c))
... 
>>>

And the output of:

$ ogrinfo -so /tmp/foo.shp foo
INFO: Open of `/tmp/foo.shp'
      using driver `ESRI Shapefile' successful.

Layer name: foo
Geometry: Polygon
Feature Count: 1
Extent: (132.612732, -14.365490) - (136.984990, -10.903420)
Layer SRS WKT:
GEOGCS["GRS 1980(IUGG, 1980)",
    DATUM["unknown",
        SPHEROID["GRS80",6378137,298.257222101]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
FID: Real (11.0)
sgillies commented 11 years ago

Sorry, once again I can't reproduce the problem (GDAL 1.9.2 here). What's your GDAL version?

>>> with fiona.open('/', vfs='zip:///Users/seang/Downloads/IBRA7.zip', layer=u'IBRA7_regions') as c:
...     with fiona.open('/tmp/foo.shp', 'w', **c.meta) as d:
...         d.write(next(c))
...
>>>

Results:

(toblerity27)krusty-2:shapely seang$ dumpgj -d /tmp/foo.shp
{"crs": {"ellps": "GRS80", "no_defs": true, "proj": "longlat"}, "driver": "ESRI Shapefile", "name": "/tmp/foo.shp", "schema": {"geometry": "Polygon", "properties": {"AREA": "float:19.8", "FEAT_ID": "str:50", "HECTARES": "float:19.8", "OBJECTID": "int:10", "REC_ID": "int:10", "REG_CODE_6": "str:9", "REG_CODE_7": "str:6", "REG_NAME_6": "str:50", "REG_NAME_7": "str:100", "REG_NO_61": "float:19.8", "SHAPE_AREA": "float:19.11", "SHAPE_LEN": "float:19.11", "SQ_KM": "float:19.8"}}}

(toblerity27)krusty-2:shapely seang$ ogrinfo -so /tmp/foo.shp foo
INFO: Open of `/tmp/foo.shp'
      using driver `ESRI Shapefile' successful.

Layer name: foo
Geometry: 3D Polygon
Feature Count: 1
Extent: (132.612732, -14.365490) - (136.984990, -10.903420)
Layer SRS WKT:
GEOGCS["GRS 1980(IUGG, 1980)",
    DATUM["unknown",
        SPHEROID["GRS80",6378137,298.257222101]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
SHAPE_AREA: Real (19.11)
REG_NAME_7: String (100.0)
REG_NAME_6: String (50.0)
OBJECTID: Integer (10.0)
AREA: Real (19.8)
FEAT_ID: String (50.0)
REC_ID: Integer (10.0)
REG_NO_61: Real (19.8)
SHAPE_LEN: Real (19.11)
HECTARES: Real (19.8)
SQ_KM: Real (19.8)
REG_CODE_7: String (6.0)
REG_CODE_6: String (9.0)
prologic commented 11 years ago

That's really odd. I have GDAL 1.10.0 here. Let me do some more digging on my end... In the meantime I've used the ogr2ogr tool with the -simplify option.

cheers James

James Mills / prologic

E: prologic@shortcircuit.net.au W: prologic.shortcircuit.net.au

On Tue, Jul 23, 2013 at 12:38 PM, Sean Gillies notifications@github.comwrote:

Sorry, once again I can't reproduce the problem (GDAL 1.9.2 here). What's your GDAL version?

with fiona.open('/', vfs='zip:///Users/seang/Downloads/IBRA7.zip', layer=u'IBRA7_regions') as c: ... with fiona.open('/tmp/foo.shp', 'w', **c.meta) as d: ... d.write(next(c)) ...

Results:

(toblerity27)krusty-2:shapely seang$ dumpgj -d /tmp/foo.shp {"crs": {"ellps": "GRS80", "no_defs": true, "proj": "longlat"}, "driver": "ESRI Shapefile", "name": "/tmp/foo.shp", "schema": {"geometry": "Polygon", "properties": {"AREA": "float:19.8", "FEAT_ID": "str:50", "HECTARES": "float:19.8", "OBJECTID": "int:10", "REC_ID": "int:10", "REG_CODE_6": "str:9", "REG_CODE_7": "str:6", "REG_NAME_6": "str:50", "REG_NAME_7": "str:100", "REG_NO_61": "float:19.8", "SHAPE_AREA": "float:19.11", "SHAPE_LEN": "float:19.11", "SQ_KM": "float:19.8"}}}

(toblerity27)krusty-2:shapely seang$ ogrinfo -so /tmp/foo.shp foo INFO: Open of /tmp/foo.shp' using driverESRI Shapefile' successful.

Layer name: foo Geometry: 3D Polygon Feature Count: 1 Extent: (132.612732, -14.365490) - (136.984990, -10.903420) Layer SRS WKT: GEOGCS["GRS 1980(IUGG, 1980)", DATUM["unknown", SPHEROID["GRS80",6378137,298.257222101]], PRIMEM["Greenwich",0], UNIT["Degree",0.017453292519943295]] SHAPE_AREA: Real (19.11) REG_NAME_7: String (100.0) REG_NAME_6: String (50.0) OBJECTID: Integer (10.0) AREA: Real (19.8) FEAT_ID: String (50.0) REC_ID: Integer (10.0) REG_NO_61: Real (19.8) SHAPE_LEN: Real (19.11) HECTARES: Real (19.8) SQ_KM: Real (19.8) REG_CODE_7: String (6.0) REG_CODE_6: String (9.0)

— Reply to this email directly or view it on GitHubhttps://github.com/Toblerity/Fiona/issues/53#issuecomment-21389881 .

sgillies commented 11 years ago

The thought of GDAL 1.10 breaking Fiona is a bit terrifying. I'll have to look into this ASAP.

prologic commented 11 years ago

If you like I can find some time today or tomorrow morning to downgrade and re-test. I actually did not realize I was using GDAL 1.10.

cheers James

James Mills / prologic

E: prologic@shortcircuit.net.au W: prologic.shortcircuit.net.au

On Wed, Jul 24, 2013 at 1:40 PM, Sean Gillies notifications@github.comwrote:

The thought of GDAL 1.10 breaking Fiona is a bit terrifying. I'll have to look into this ASAP.

— Reply to this email directly or view it on GitHubhttps://github.com/Toblerity/Fiona/issues/53#issuecomment-21461783 .

sgillies commented 11 years ago

I can't reproduce your problem with GDAL 1.10 either. I'm sorry, but will close this.

prologic commented 11 years ago

No problems. Thanks for trying!

James Mills / prologic

E: prologic@shortcircuit.net.au W: prologic.shortcircuit.net.au

On Tue, Jul 30, 2013 at 2:51 PM, Sean Gillies notifications@github.comwrote:

I can't reproduce your problem with GDAL 1.10 either. I'm sorry, but will close this.

— Reply to this email directly or view it on GitHubhttps://github.com/Toblerity/Fiona/issues/53#issuecomment-21769571 .