geojson_intersects does not work for linestrings

[pnorman]

When testing geojson_intersects_feature I encountered a segmentation violation

Error

/path/to/imposm/imposm import -connection 'postgres: host=<host> port=5432 dbname=mapdata user=importer password=<pass>' \
-mapping /path/to/mapping.yaml -cachedir /mnt/ebs/imposm/cache -appendcache -diffdir /mnt/ebs/imposm/diff -diff -read /mnt/ebs/imposm/input/washington-191208.osm.pbf

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5eda8a]
goroutine 1 [running]:
github.com/omniscale/imposm3/geom/geojson.constructPolygonFeatures(0x0, 0x7, 0xa39140, 0x7, 0xffffffffffffffff, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/geom/geojson/geojson.go:136 +0x3a
github.com/omniscale/imposm3/geom/geojson.constructPolygonFeatures(0xc000110a10, 0xc0005691e0, 0x1, 0x1, 0xc0005691c0, 0x1)
    /home/travis/gopath/src/github.com/omniscale/imposm3/geom/geojson/geojson.go:148 +0x29c
github.com/omniscale/imposm3/geom/geojson.constructPolygonFeatures(0xc00024e9b0, 0x949b80, 0xc00024e9b0, 0x0, 0x0, 0x4346ca)
    /home/travis/gopath/src/github.com/omniscale/imposm3/geom/geojson/geojson.go:161 +0x5a9
github.com/omniscale/imposm3/geom/geojson.ParseGeoJSON(0xaff620, 0xc000162fe0, 0x0, 0x0, 0xc000162fe0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/geom/geojson/geojson.go:126 +0xaf
github.com/omniscale/imposm3/mapping.loadFeatures(0xc0001657f0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0001e98a0, 0x1a, 0xc00015dd40, ...)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/columns_intersection.go:47 +0x21e
github.com/omniscale/imposm3/mapping.MakeIntersectsFeatureField(0xc0001657f0, 0xc, 0xa46e12, 0x1a, 0xa38b6c, 0x6, 0x0, 0xa648c0, 0x0, 0x0, ...)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/columns_intersection.go:71 +0x60
github.com/omniscale/imposm3/mapping.MakeColumnType(0xc000250f00, 0xc00029e9c0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:230 +0x243
github.com/omniscale/imposm3/mapping.makeRowBuilder(0xc0002425a0, 0xa3c079, 0xa, 0x1)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:213 +0xd4
github.com/omniscale/imposm3/mapping.(*Mapping).tables(0xc00016e680, 0xa3c079, 0xa, 0xc00028f020, 0x0, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:196 +0xf8
github.com/omniscale/imposm3/mapping.(*Mapping).lineStringMatcher(0xc00016e680, 0xaff000, 0xc00028e600, 0x0, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/matcher.go:29 +0xc8
github.com/omniscale/imposm3/mapping.(*Mapping).createMatcher(0xc00016e680, 0x0, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:150 +0x75
github.com/omniscale/imposm3/mapping.New(0xc000190000, 0x6755, 0x6955, 0x6755, 0x6955, 0x0)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:113 +0xb2
github.com/omniscale/imposm3/mapping.FromFile(0x7fff24746b62, 0x20, 0x8, 0xc000096900, 0xc000111350)
    /home/travis/gopath/src/github.com/omniscale/imposm3/mapping/mapping.go:98 +0x87
github.com/omniscale/imposm3/import_.Import(0x7fff24746ab0, 0xa8, 0x7fff24746b8d, 0x15, 0x7fff24746bb9, 0x14, 0x7fff24746b62, 0x20, 0xf11, 0x0, ...)
    /home/travis/gopath/src/github.com/omniscale/imposm3/import_/import.go:50 +0xf8
main.Main(0xa64b30)
    /home/travis/gopath/src/github.com/omniscale/imposm3/cmd/imposm/main.go:48 +0x4db
main.main()
    /home/travis/gopath/src/github.com/omniscale/imposm3/cmd/imposm/main.go:77 +0x2d

The extract from the config is

tables:
  highway_linestring:
    fields:
    - {name: osm_id, type: id}
    - {name: geometry, type: geometry}
    - {name: country_code, type: geojson_intersects_feature, args: {geojson: /path/to/mapping/borders.json, property: iso1A2}}
    mapping:
      highway: [motorway, motorway_link, trunk, trunk_link, primary, primary_link,
        secondary, secondary_link, tertiary, tertiary_link, unclassified, residential,
        road, living_street, raceway, track, service, path, cycleway, bridleway, footway,
        corridor, pedestrian, steps]
      man_made: [pier]
      public_transport: [platform]
    type: linestring

/path/to/mapping/borders.json is this file from country-coder

I tried trimming borders.json to just the Canada and USA polygons but still got the error. I also tried forcing the right-hand rule with http://mapster.me/right-hand-rule-geojson-fixer/ and still got the error.

[pnorman]

I've reduced this to a smaller testcase

imposm import -connection postgis://paunorma@localhost/imposm -mapping config.yml -overwritecache -read w5255262.osm.pbf -write

Where config.yml is

tables:
  highway_linestring:
    fields:
    - {name: osm_id, type: id}
    - {name: geometry, type: geometry}
    - {name: country_code, type: geojson_intersects_feature, args: {geojson: borders.json, property: iso1A2}}
    mapping:
      highway: [residential]
    type: linestring

and borders.json is

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {"iso1A2": "US"},
      "geometry": {
        "type": "Polygon",
        "coordinates": [[[-122.7,48.8],[-122.6,48.8],[-122.6,48.9],[-122.7,48.9],[-122.7,48.8]]]
      }
    }
  ]
}

and w5255262.osm.pbf is obtained by osmium cat w5255262.osm -o w5255262.osm.pbf and w5255262.osm is

<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="osmium/1.11.1">
  <bounds minlat="45.54326" minlon="-126.7423" maxlat="49.00708" maxlon="-116.9145"/>
  <node id="37064015" version="3" timestamp="2013-12-11T18:41:15Z" lat="48.8462652" lon="-122.6477865"/>
  <node id="37064016" version="3" timestamp="2013-12-11T18:41:15Z" lat="48.8463666" lon="-122.6478154"/>
  <node id="37064017" version="3" timestamp="2013-12-11T18:41:15Z" lat="48.8474395" lon="-122.6483791"/>
  <node id="2576091539" version="1" timestamp="2013-12-11T18:41:14Z" lat="48.8473304" lon="-122.6479495"/>
  <node id="2576091541" version="1" timestamp="2013-12-11T18:41:14Z" lat="48.8482759" lon="-122.6483664"/>
  <way id="5255262" version="4" timestamp="2017-06-05T19:55:38Z">
    <nd ref="37064015"/>
    <nd ref="37064016"/>
    <nd ref="2576091539"/>
    <nd ref="37064017"/>
    <nd ref="2576091541"/>
    <tag k="highway" v="residential"/>
    <tag k="name" v="Barr Road"/>
    <tag k="tiger:cfcc" v="A41"/>
    <tag k="tiger:county" v="Whatcom, WA"/>
    <tag k="tiger:name_base" v="Barr"/>
    <tag k="tiger:name_type" v="Rd"/>
    <tag k="tiger:reviewed" v="no"/>
  </way>
</osm>

This testcase gives a different segmentation violation

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x419a5f0]

goroutine 68 [running]:
github.com/omniscale/imposm3/geom/geos.(*Geos).IndexQuery.func1(0xc000142840, 0xc00013c990, 0x0, 0xc00030e0f0, 0x203000)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/geom/geos/index.go:65 +0x40
github.com/omniscale/imposm3/geom/geos.(*Geos).IndexQuery(0xc000142840, 0xc00013c990, 0x0, 0x0, 0x0, 0x0)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/geom/geos/index.go:65 +0xee
github.com/omniscale/imposm3/mapping.MakeIntersectsFeatureField.func1(0x0, 0x0, 0xc00023e020, 0xc00023e040, 0x463705f, 0x7, 0x463a24a, 0xb, 0xc00014c100, 0x12, ...)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/mapping/columns_intersection.go:88 +0x77
github.com/omniscale/imposm3/mapping.(*valueBuilder).Value(...)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/mapping/matcher.go:242
github.com/omniscale/imposm3/mapping.(*rowBuilder).MakeRow(0xc000142400, 0xc00023e020, 0xc00023e040, 0x463705f, 0x7, 0x463a24a, 0xb, 0xc00014c100, 0x12, 0x0, ...)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/mapping/matcher.go:270 +0x2e2
github.com/omniscale/imposm3/mapping.(*Match).Row(...)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/mapping/matcher.go:118
github.com/omniscale/imposm3/database/postgis.(*PostGIS).InsertLineString(0xc000198140, 0x50305e, 0xc000146540, 0x0, 0x0, 0xc00031c000, 0xba, 0xba, 0xc000240050, 0x1, ...)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/database/postgis/postgis.go:474 +0x475
github.com/omniscale/imposm3/writer.(*WayWriter).buildAndInsert(0xc0000ab7c0, 0xc0001bcf70, 0xc00029a050, 0xc000240050, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/writer/ways.go:152 +0x760
github.com/omniscale/imposm3/writer.(*WayWriter).loop(0xc0000ab7c0)
    /Users/paunorma/go/src/github.com/omniscale/imposm3/writer/ways.go:101 +0x57a
created by github.com/omniscale/imposm3/writer.(*OsmElemWriter).Start
    /Users/paunorma/go/src/github.com/omniscale/imposm3/writer/writer.go:52 +0x81

Debugging with delve and stepping to https://github.com/omniscale/imposm3/blob/3b6e6b38f4529ac468580040dff25a0b47a037c9/mapping/columns_intersection.go#L88 tells me that geom is

(dlv) print geom
*github.com/omniscale/imposm3/geom.Geometry {
    Geom: *github.com/omniscale/imposm3/geom/geos.Geom nil,
    Wkb: []uint8 len: 186, cap: 186, [48,49,48,50,48,48,48,48,50,48,49,49,48,102,48,48,48,48,48,53,48,48,48,48,48,48,102,102,98,54,56,51,50,52,56,99,48,97,54,97,99,49,54,50,97,48,100,102,102,99,53,98,100,54,53,55,52,49,53,52,55,100,56,54,...+122 more],}

Trying with and without the geojson_intersects_feature column, the Geom is nil in both cases.

@olt, do you have a working example of geojson_intersects_feature?

[olt]

Thanks for digging into this. Apparently this feature does not work with linestrings. It expects a GEOS geometry but simple linestrings are directly converted to WKB.

[pnorman]

Are GEOS geometries not constructed for linestrings for performance reasons? If so, do you think building them would be an unacceptable performance loss?

[pnorman]

ping @olt

[olt]

Yes, it speeds up the import. Unacceptable? No, but it's a nice performance gain and I actually wanted to extend this in the future for polygons, but the Area column types are also require a GEOS geometry.

Disabling this optimization if there is a mapping that requires GEOS geometries might be a good compromise.