doit_buildingfootprints script update

[td928]

Hey DE! @AmandaDoyle @damonmcc @fvankrieken Happy Friday! Hope you are doing well.

I think I raised question aboutdoit_buildingfootprints having issues with being updated

After investigating for a little bit, I found that doit_buildingfootprints is a little unique in that data library is not directly treating it as because it doesn't support a geojson instead doit_buildingfootprints.py needs to point it explicitly to the API endpoint then export it as a shapefile. I was able to narrow the issue down to the API endpoint this line below where the endpoints are fetched from the metadata.

  @property
  def uid(self):
      return self.metadata["childViews"][0]

But there are two endpoints, the scirpt currently is pointing stopped working and simply switched the other worked for me. See below for the change.

  @property
  def uid(self):
      return self.metadata["childViews"][1]

I also modified the yml template a bit for it to be more align with the data description and geometry type from the open data portal

dataset:
  name: &name doitt_buildingfootprints
  version: "{{ version }}"
  acl: public-read
  source:
    script: *name
    options:
      - "AUTODETECT_TYPE=NO"
      - "EMPTY_STRING_AS_NULL=YES"
    geometry:
      SRS: EPSG:4269
      type: POLYGON

  destination:
    name: *name
    geometry:
      SRS: EPSG:4326
      type: POLYGON
    options:
      - "OVERWRITE=YES"
      - "PRECISION=NO"
    fields: []
    sql: null

Probably gonna run some more tests about this on DevDB after I finished pulling in the latest version. But I just want to flag what I find just in case you are interested.

Best, Te

[fvankrieken]

Hmm. So the uid for buildingfootprints on opendata is nqwf-w8eh, which is what we have pinned in the script in data library. This is the view with both multipolygon footprints and points that seem to be centroids. Querying that link in the script for metadata manually, we see the two childviews

"childViews" : [ "7w4b-tj9d", "qb5r-6dgf" ]

Looks like the first one is points data and the second is multipolygon.

So it seems like the order flipped? Regardless, grabbing the "first" childview doesn't seem like the safest way to refer to the actual source data. Do we know if the uid for the underlying multipolygon data (qb5r-6dgf) is stable? It should be, right? If so, we could just use socrata as a source and aim straight at that, no? With information on how to verify these uids in the info section of the template yml

[td928]

Good catch on spotting the difference between the views are points and multipolygon. I guess that makes sense then for the transformation to fail for this nqwf-w8eh. I didn't actually try whether pointing to this qb5r-6dgf and export as geojson would work. But I think that would definitely be a more cleaner solution and do without the script!