Mapping __any__: [__any__] leads to empty columns

[ghtyrant]

Context

We are importing OSM data using imposm and have some "general-purpose" tables that simply import all polygons. Our mapping looks like this:

tags:
  load_all: true
  exclude: ["note", "note:*", "source", "source_ref", "source:*", "attribution", "comment", "fixme", "created_by", "odbl", "odbl:note", "SK53_bulk:load", "tiger:*", "NHD:*", "nhd:*", "gnis:*", "geobase:*", "accuracy:meters", "sub_sea:type", "waterway:type", "KSJ2:*", "yh:*", "osak:*", "kms:*", "ngbe:*", "naptan:*", "CLC:*", "3dshapes:ggmodelk", "AND_nosr_r", "import", "it:fvg:*"]
areas:
  area_tags: [building, landuse, leisure, natural, aeroway]
tables:
  planet_osm_point:
    type: point
    mapping:
      __any__: [__any__]
    columns:
      - {name: osm_id, type: id}
      - {name: way, type: geometry}
      - {name: tags, type: hstore_tags}
      - {name: "access", type: string}
      - {name: "admin_level", type: string}
      - {name: "aerialway", type: string}
      - {name: "aeroway", type: string}
      - {name: "amenity", type: string}
      - {name: "area", type: string}
      - {name: "barrier", type: string}
      - {name: "bicycle", type: string}
      - {name: "boundary", type: string}
      - {name: "brand", type: string}
      - {name: "bridge", type: string}
      - {name: "building", type: string}
      - {name: "capital", type: string}
      - {name: "construction", type: string}
      - {name: "covered", type: string}
      - {name: "culvert", type: string}
      - {name: "cutting", type: string}
      - {name: "denomination", type: string}
      - {name: "disused", type: string}
      - {name: "ele", type: string}
      - {name: "embankment", type: string}
      - {name: "foot", type: string}
      - {name: "harbour", type: string}
      - {name: "highway", type: string}
      - {name: "historic", type: string}
      - {name: "intermittent", type: string}
      - {name: "junction", type: string}
      - {name: "landuse", type: string}
      - {name: "layer", type: string}
      - {name: "leisure", type: string}
      - {name: "lock", type: string}
      - {name: "man_made", type: string}
      - {name: "military", type: string}
      - {name: "motorcar", type: string}
      - {name: "name", type: string}
      - {name: "natural", type: string}
      - {name: "office", type: string}
      - {name: "oneway", type: string}
      - {name: "place", type: string}
      - {name: "population", type: string}
      - {name: "power", type: string}
      - {name: "power_source", type: string}
      - {name: "public_transport", type: string}
      - {name: "railway", type: string}
      - {name: "ref", type: string}
      - {name: "route", type: string}
      - {name: "service", type: string}
      - {name: "shop", type: string}
      - {name: "sport", type: string}
      - {name: "surface", type: string}
      - {name: "toll", type: string}
      - {name: "tourism", type: string}
      - {name: "tower:type", type: string}
      - {name: "tunnel", type: string}
      - {name: "water", type: string}
      - {name: "waterway", type: string}
      - {name: "wetland", type: string}
      - {name: "width", type: string}
      - {name: "wood", type: string}

  planet_osm_line:
    type: linestring
    mapping:
      __any__: [__any__]
    columns:
      - {name: osm_id, type: id}
      - {name: way, type: geometry}
      - {name: tags, type: hstore_tags}
      - {name: "access", type: string}
      - {name: "admin_level", type: string}
      - {name: "aerialway", type: string}
      - {name: "aeroway", type: string}
      - {name: "amenity", type: string}
      - {name: "area", type: string}
      - {name: "barrier", type: string}
      - {name: "bicycle", type: string}
      - {name: "boundary", type: string}
      - {name: "brand", type: string}
      - {name: "bridge", type: string}
      - {name: "building", type: string}
      - {name: "building:height", type: string}
      - {name: "building:levels", type: string}
      - {name: "construction", type: string}
      - {name: "covered", type: string}
      - {name: "culvert", type: string}
      - {name: "cutting", type: string}
      - {name: "denomination", type: string}
      - {name: "disused", type: string}
      - {name: "embankment", type: string}
      - {name: "foot", type: string}
      - {name: "harbour", type: string}
      - {name: "highway", type: string}
      - {name: "historic", type: string}
      - {name: "intermittent", type: string}
      - {name: "junction", type: string}
      - {name: "landuse", type: string}
      - {name: "layer", type: string}
      - {name: "leisure", type: string}
      - {name: "lock", type: string}
      - {name: "man_made", type: string}
      - {name: "military", type: string}
      - {name: "motorcar", type: string}
      - {name: "name", type: string}
      - {name: "natural", type: string}
      - {name: "office", type: string}
      - {name: "oneway", type: string}
      - {name: "place", type: string}
      - {name: "population", type: string}
      - {name: "power", type: string}
      - {name: "power_source", type: string}
      - {name: "public_transport", type: string}
      - {name: "railway", type: string}
      - {name: "ref", type: string}
      - {name: "route", type: string}
      - {name: "service", type: string}
      - {name: "shop", type: string}
      - {name: "sport", type: string}
      - {name: "surface", type: string}
      - {name: "toll", type: string}
      - {name: "tourism", type: string}
      - {name: "tower:type", type: string}
      - {name: "tracktype", type: string}
      - {name: "tunnel", type: string}
      - {name: "water", type: string}
      - {name: "waterway", type: string}
      - {name: "wetland", type: string}
      - {name: "width", type: string}
      - {name: "wood", type: string}

  planet_osm_polygon:
    type: polygon
    mapping:
      __any__: [__any__]
    columns:
      - {name: osm_id, type: id}
      - {name: way, type: geometry}
      - {name: tags, type: hstore_tags}
      - {name: "access", type: string}
      - {name: "admin_level", type: string}
      - {name: "aerialway", type: string}
      - {name: "aeroway", type: string}
      - {name: "amenity", type: string}
      - {name: "area", type: string}
      - {name: "barrier", type: string}
      - {name: "bicycle", type: string}
      - {name: "boundary", type: string}
      - {name: "brand", type: string}
      - {name: "bridge", type: string}
      - {name: "building", type: string}
      - {name: "building:height", type: string}
      - {name: "building:levels", type: string}
      - {name: "construction", type: string}
      - {name: "covered", type: string}
      - {name: "culvert", type: string}
      - {name: "cutting", type: string}
      - {name: "denomination", type: string}
      - {name: "disused", type: string}
      - {name: "embankment", type: string}
      - {name: "foot", type: string}
      - {name: "harbour", type: string}
      - {name: "highway", type: string}
      - {name: "historic", type: string}
      - {name: "intermittent", type: string}
      - {name: "junction", type: string}
      - {name: "landuse", type: string}
      - {name: "layer", type: string}
      - {name: "leisure", type: string}
      - {name: "lock", type: string}
      - {name: "man_made", type: string}
      - {name: "military", type: string}
      - {name: "motorcar", type: string}
      - {name: "name", type: string}
      - {name: "natural", type: string}
      - {name: "office", type: string}
      - {name: "oneway", type: string}
      - {name: "place", type: string}
      - {name: "population", type: string}
      - {name: "power", type: string}
      - {name: "power_source", type: string}
      - {name: "public_transport", type: string}
      - {name: "railway", type: string}
      - {name: "ref", type: string}
      - {name: "route", type: string}
      - {name: "service", type: string}
      - {name: "shop", type: string}
      - {name: "sport", type: string}
      - {name: "surface", type: string}
      - {name: "toll", type: string}
      - {name: "tourism", type: string}
      - {name: "tower:type", type: string}
      - {name: "tracktype", type: string}
      - {name: "tunnel", type: string}
      - {name: "water", type: string}
      - {name: "waterway", type: string}
      - {name: "wetland", type: string}
      - {name: "width", type: string}
      - {name: "wood", type: string}

  osm_building:
    type: polygon               
    mapping:                            
      building: [__any__]                
    columns:                             
      - {name: osm_id, type: id}     
      - {name: footprint, type: geometry}     
      - {name: levels, key: "building:levels", type: string}
      - {name: building_height, key: "building:height", type: string}
      - {name: height, type: string}
      - {name: building_type, type: mapping_value}
      - {name: roof_type, key: "roof:shape", type: string}

Expected Behavior

I expect our custom columns (e.g. "water") of the tables above to contain values even if the mapping is set to __any__: [__any__]

Actual Behavior

All columns except osm_id, way and tags are empty. The tags contain valid data, they also contain the data which I would expect to also be stored in the columns.

Tables that have an explicit mapping work as expected.

Steps to Reproduce

1. Use the above mapping to import data from a PBF (I used australia-oceania-latest.osm.pbf)
2. Read & Write the data using imposm
3. Run: SELECT * FROM planet_osm_polygon WHERE levels != '';. For comparison, run SELECT * FROM osm_building WHERE levels != '';

Your Environment

• Version used: Both latest release and HEAD of git
• Environment name and version (e.g. PostgreSQL x.y.z with PostGIS a.b.c): PostgreSQL 10 with PostGIS 2.4
• Operating System and version: Ubuntu 18.04

[olt]

Your columns are missing a key https://imposm.org/docs/imposm3/latest/mapping.html#key You should also check out the hstore option, which might be more efficient.

[ghtyrant]

Thank you, that fixed the issue. Should've read more thoroughly, as I was under the impression that key is only necessary if the name deviates from the actual tag's name.