Model: Gazetteer - user/project specific geospatial shapes

[mjy]

As a curator I want to be able to add and name my own shapes outside the Georeference model.

Likely keep this simple, just multi-polygon.

[proceps]

I would also add an option to combine few poligons in a single shape.

[debpaul]

Interesting. How does that look (what is the use case for multi-polygons?

• More than one polygon, say for known ranges that don't seem to overlap?
• More than one polygon b/c of multiple georeferencing attempts?

FYI GEOLocate (as part of two upcoming Thematic Collection Networks in the ADBC program), will be adding some functions to GEOLocate, for example, polygons with a "hole" in the middle. (Use case, I want to study terrestrial life whose habitat is fresh-water shorelines. So I need essentially, to be able to make "donut" shapes around lakes. I do not know anything about how Nelson and DJ (developer) will do that, from either the UI or the data format point-of-view.

[debpaul]

@proceps what do you mean by multiple polygons in a single shape?

[proceps]

For example, we have shapes of individual US states, and I would like to combine them into Midwest.

[mjy]

@debpaul "multi_polygon" is a data type in postgres, its allows for lines, single polygons, etc., i.e. we're not necessarily saying we want multiple shapes.

This is a spatial library for the user. They can use their shapes for queries, clone them to georeferences, etc. It's not just a georeference function.

[mjy]

Meh- sorry, "multi_polygon" should be "geometry_collection".

[mjy]

Model

Base (required)

• geographic_item_id The ID of a GeographicItem
• name - string

Utility

• parent_id the ID of a Gazetter which encloses(?) this shape. Defining the semantics of this is pretty important, is it some set, subdivisions, or ?! I.e. we need to know well what geo-spatial inferences we can make with this field, if any
• iso2 as optional
• iso3 as opional
• adm_level ?
• adm_level_0|1|2_parent_id polymorphic for GA or G record?!

Relations to GeographicArea

• Should "parent" be polymorphic and populable by GeographicArea? This would let users extend GA to some extent. Similar questions via ADM levels

RElations to GeographicItem

We tried to keep all spatial representation in a single table. This means we should add a geography table to GeographicItem (with the downstream plan of merging all the remaining geo columns into it).

Extensions

• Citations
• Notes
• DataAttributes
• AlternateValue (on name)

[bpescador]

I would like the option to add protected areas, ecoregions, and bioregions: why? so that I can filter these areas to generate species lists, COs, etc.

[kleintom]

Some first thoughts - I'm sure I'm missing/confused about some things; corrections or things to think about that I'm leaving out welcome :)

Creation

• Which shapes for Gazs should be allowed? Only multi-polygons was suggested as an option in the first comment, but maybe now more (I'm leaning that way)? A geometry shape column does support storing a shape of any other type (I haven't tested with has_m which doesn't seem to be supported with the existing factory).

• Create using the same options as for Georeferences

  • Creating by drawing by hand/leaflet
  • Entering WKT by hand/copying / entering lat/lng by hand
  • Enter GEOLocate output by hand (maybe also as in http://localhost:3000/georeferences/geo_locates/new?)

• Importing (some version of) geojson/shapefiles, maybe ecoregions (here's the latest link, for 2017, afaict: https://www.geographyrealm.com/terrestrial-ecoregions-gis-data/ 150Mb zipped) At first blush this to me seems reasonable at first for "flat" data - where fixed parent would be earth (or one chosen GA/G/Gaz shape) - where the name field could maybe be specified or assumed, and with some kind of limits (upload size and/or number of records - probably batching/timeout/transaction issues here I'm not yet familiar with). Hierarchical data maybe requires more custom imports?

• Unions (other operations?) of existing GAs/Gs/Gazs, as multi-(point/line/polygon/geometry) (not merged along boundaries, though maybe that's a reasonable option/optimization).

  • Special case: clone a single GA into a Gaz (maybe with a specified parent)
  • Consideration: GAs have a has-many relation with GeographicItems, but iirc there aren't currently any GAs that associate to multiple GeoItems (in the opposite direction there are). Not sure how that's handled in code now (does existing code check for/handle multiple GeoItems for a GA?). (Also implications if GA/Gaz are STIed: Gazs only associate to one GeoItem.)

• Clone parts of the GA tree as Gazs, which could then be edited (for example to support new shapes for some region)?

Updating

• GAs can't be updated or deleted, their backing GeoItems can't be updated or deleted. I would think we'd want to be able to do all of those updates for Gazs. This has implications for the tree structure (validations).

Uses

• Mentioned above or elsewhere:

  • Cloned into a georeference, i.e. (I guess?) there would be an option on Georeference creation/selection to search for a Gaz to use as the backing GeoItem of a Georeference (a little bit strange since there's currently an entirely separate input for GA in New CE, but that's okay).
  • A Gaz could be specified instead of a GA for the GA input on New CE (might require the Gaz to have certain data).
  • Queries
  • For example the current CE or CO Filter has a GA search to filter in various spatial ways against a shape, similar could be done for Gazs. The issue of parenting will play a role in regards to how GA filtering can be combined with Gaz filtering (the simplest way would be to have completely separate GA/Gaz filters, but more is possible).
  • Dropins for updated or new GA shapes that haven't made it into GA yet (for which ideally Gaz and GA would interact in some mostly seamless way). As discussed elsewhere, to me this is seeming like it would more take the shape of recreating the relevant part of the GA tree as Gazs (by cloning or some other means) rather than inserting a new Gaz into the GA tree in some way.

• Autocomplete on GA names may or may not return Gaz names as well (probably with some indication in the label), depending on context.

Additions welcome!

Parenting

• parent_id the ID of a Gazetter which encloses(?) this shape. Defining the semantics of this is pretty important, is it some set, subdivisions, or ?! I.e. we need to know well what geo-spatial inferences we can make with this field, if any

My first thought is that if you can draw/input a Gaz by hand then anything other than containment would seem difficult to specify for a parent. Maybe for two things that have a level/iso/adm code specified more should/would be required. (Probably not validated on GA because that's all determined at build time by the build scripts?) I don't have a real feel for other possible answers to this yet though (and am not yet very clued into the ways in which current parentage is used), thoughts welcome.

• Right now for GAs parentage is static, determined at GA build time(?). Once you can create new shapes on the fly you'll probably want to be able to change parents (similar to operations permitted on taxon names (I guess) or leads). For example, inserting a new shape below a given one, where you may only want to insert between some of the existing children (based on containment or something else). Maybe do a basic case to start.

Relations to GeographicArea

• Should "parent" be polymorphic and populable by GeographicArea? This would let users extend GA to some extent. Similar questions via ADM levels

First thoughts - I'm not yet clued in to all of the implications/technical details of these decisions:

• closure_tree supports STI polymorphic types, so we could make GA and GAZ subclasses of some STI base (spelling it out to make sure I'm on the right track). Then you would be able to make a GA a parent of a Gaz and vice versa. We probably (almost certainly?) only want to support the former.

  • Without the combined tree it would seem like anywhere you're currently using a GA in the above use cases you'd have to have a completely separate path to use a GAZ (in the UI and/or combined in separate queries under the hood, for example).
  • As mentioned above, with the combined tree you'd also then be able to "easily" associate the closest parent (GA) adm_level
  • If you do combine the tree there might still be cases where you want to only use GAs, or only Gazs (I"m not sure).

• Things that currently refer to a GA - ADs, CEs - could have a polymorphic relation to either a GA or a Gaz (make sure Gaz holds the needed data for these).

Relations to GeographicItem

• We tried to keep all spatial representation in a single table. This means we should add a geography table to GeographicItem (with the downstream plan of merging all the remaining geo columns into it).

Purely in terms of storing an arbitrary shape in a column of type geometry, this seems to work fine. GeographicItem has many methods, and I haven't gotten through them all yet in accounting for the new separate data type, but that should work out.

In terms of testing, if GeographicItem is eventually going to put all shapes in a single geography column, does it make sense to just duplicate all of the existing GeographicItem specs and adjust them for a geography-column shape?

[mjy]

Which shapes for Gazs should be allowed?

All. Anything that we can spatially compute with.

Hierarchical data maybe requires more custom imports?

Target individual (== flat) records initially. If we can determine semantics between other records (see parent_id musings) then we can add those after the fact. Our primary goal is to 1) add shapes and 2) reference those in place of GAs.

This reminds me that we need to think about how to optionally share across projects. We could project_id == nil assert that it is sharable, however managing updates in that case is difficult downstream. Another approach is to make use of the is_public flag or create is_shared. Finally (and this is where I would lean to start) we could simply clone Gaz entries, while pointing to the identical shape. This would permit discovery and forking of use.

but iirc there aren't currently any GAs that associate to multiple GeoItems

Correct. The multiple shapes relationship is 1:1 representation of the same GA, from different origins.

does existing code check for/handle multiple GeoItems for a GA?

We have default priority, this logic is encoded in GI.

GA tree as Gazs, which could then be edited

Don't sweat editing now. However, the first (and perhaps only) edit I'd like to include is to clip to a bounding box. This use case is to setup for rendering base-maps as png/svg.

we'd want to be able to do all of those updates for Gazs

Maybe. A write once read/replace pattern would also work. I.e. if a user needs to edit the shape then they add a new one or replace the shape. Other attributes that we add should be editable.

issue of parenting will play a role in regards to how GA filtering can be combined with Gaz filtering

Yes, we have multiple combining paths for GA already (see descendants options). These can be parameterized out in the filter. A major theme throughout is to use spatial queries whenever possible, and to guide/direct users to start thinking spatially, not with labels.

would be to have completely separate GA/Gaz filters

Yes.

Autocomplete on GA names

Should only hit GAs in the smart selector. Merged spatial facets are also possible, but these are different code. I want to keep the two isolated at first, with convenience second.

clued into the ways in which current parentage

Parentage needs much cleaner semantics in Gaz, if indeed we use it at all (is it geopolitical, spatial, set defining). Let's not try to sync it with GA for now, because there it means "as used in TW to build the Gaz, roughly geopolitical sets", which could be better defined.

Then you would be able to make a GA a parent of a Gaz and vice versa

No editing GA by users. This will remain an Admin-only defined set of data. Down the road we could envision a user suggesting some GZ be included in the GA, but that's for future consideration. No swapping parenthood for GAs. Asserting a GA is some semantic relation to GZ is possible, but downstream priority re implementation. In general we want to allow "pick a name, use the shape as a spatial search, .pick a set, use that set in a combined spatial search".

does it make sense to just duplicate all of the existing GeographicItem specs and adjust them for a geography-column shape?

Yes, definitely, certainly on a dev branch.

[kleintom]

Are we able to assume postgis >= 3.0 yet? It's not necessary but it would simplify some of the existing and new code around things like ST_Contains(geometry_collection) which isn't supported in earlier versions.

[mjy]

@LocoDelAssembly can confirm what we are using in production, but you should assume for dev evolution 3.3.6 and PG 15. I'd be happy to push that as far foward as we can vs. 1) Ubuntu support and 2) the postgis adapter (https://github.com/rgeo/activerecord-postgis-adapter) and related rgeo library. I'm happy to push all fronts forward as part of this (including feedback to the adapter folk).

[LocoDelAssembly]

Postgres	Postgis
PostgreSQL 15.6 (Ubuntu 15.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit	POSTGIS="3.4.1 ca035b9" [EXTENSION] PGSQL="150" GEOS="3.10.2-CAPI-1.16.0" PROJ="8.2.1 NETWORK_ENABLED=OFF URL_ENDPOINT=https://cdn.proj.org USER_WRITABLE_DIRECTORY=/tmp/proj DATABASE_PATH=/usr/share/proj/proj.db" GDAL="GDAL 3.4.1, released 2021/12/27" LIBXML="2.9.13" LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)" RASTER

[kleintom]

@bpescador, I'm copying your comment from my Ecoregions gist link to here where the TW team can comment:

I assume that with the new Ecoregion shapefile gazetteer, one could filter COs and CEs on a particular Ecoregion (Biome or "biogeographic realm"). However, would there be a way to create a predicate field "Ecoregion," "Biome," and "biogeographic realms" and have the data filled in based on the gazetter shapefiles so the names could be exported and used in other analyses?