Add Lat/Lon and URI fields to Subjects (documentation needed)

[benwbrum]

Subjects are the core of indexing and semantic mark-up in FromThePage, representing indexable people, places, themes, events, species, animals, and other project-defined entities within FromThePage. (For historical reasons, they are stored in the Article model.)

Adding latitude/longitude attributes for GIS support has been funded through the Pelagios RDG:

FTP2 Enhancing subject data in FromThePage: ​Currently subjects within FromThePage are represented with title and article text attributes as well as index entries into the places they are mentioned within the text. We will add new fields for latitude and longitude for any subjects that are places. In addition, we will add URIs attributes for all subjects regardless of category, allowing them to be cross referenced to authorities like Pleiades or VIAF. All exports will include these new fields.

To implement this, we should allow subjects in the Places category (or child categories of Places) to accept an additional two fields from editors to represent latitude and longitude. In addition, all subjects should accept a new URI field for pointers to external authority records like Pleiades.

Open questions:

• What datatype is standard for latitude/longitude in MySQL? What's compatible with Postgres, in the event that we upgrade databases in the future to take better adantage of plugins like PostGIS?
• Should we accept coodinates in decimal only, or in degrees, minutes, and seconds?
• How do we add the lat/lon fields to Place subjects but not to other subjects? Is this a case for Single Table Inheritance? Should a poor-man's version be implemented by adding the fields and just conditionally hiding them? One can imagine a future enhancement that adds something like alphebetized name to Person subjects. (Note also the many-to-many relationship between subjects and categories.)

[jackweinbender]

On Storing Lat/Lon

There's no question that Postgres/postgis is the preferred DB for GIS applications; the consensus seems to be that the MySQL Spacial extensions (MySQL's attempt, introduced in MySQL 5.7) is well-behind Postgres/postgis in just about every way.

That said, if we're just storing lat/lon pairs and not doing spacial analysis on sets of coordinates, it seems like MySQL would be just fine---even without the spacial extensions---for storing decimal numbers. The major benefits of the geospacial data as its own datatype is to offload the computation to the database for finding "nearby" things or "things within X miles of", etc. Even there, assuming we aren't dealing with millions of records, surely this could be done effectively with traditional DB queries.

Re: Decimal Degrees (DD) v. Degrees/Minutes/Seconds (DMS), everything I've read seems to describe storing Lat/Lon as pairs of decimal/float (or, as the special POINT datatype), so it seems like DD is the way to go on that level (as much as I love a good hexagesimal system...) As far as display, I like the look of DMS. The math for converting between DD/DMS is pretty straightforward, so I imagine we'll want to accept either as user input. Could it be helpful for people to be able to click on a map, too?

Gems:

I did some poking around on Github for gems that could be helpful. These were the two that stood out to me. Both are, likely, more robust than our application requires. Both seem to be currently maintained.

• RGeo: a suite of ruby/rails gems for geospacial stuff (Requires ruby 2.2.2). This seems to be the most widely used, esp. with a postgres/postgis setup, but has adapters for MySQL (for both mysql and mysql2 gems), Postgres/postgis and Spacialite for use with ActiveRecord.

  • rgeo
  • rgeo-activerecord
  • activerecord-mysql2spatial-adapter
  • Other adapters: activerecord-postgis-adapter / activerecord-spatialite-adapter / activerecord-mysqlspatial-adapter

• GeoKit also has Rails-specific gem GeoKit-Rails

  • Focused on Geocoding; Uses 3rd party API's for getting lat/lon from City/Address data, but also supports reverse geocoding (lookup by lat/lon)
  • provides lat/lon database abstractions for calculating within distances, etc.
  • Uses float datatype for Database columns

[jbhoward-dublin]

Agreed regarding PostgreSQL/postgis, where you can express the lat/lng as a spatial attribute/geometry. That said, it probably makes sense, even with PostgreSQL/postgis available, to also include separate lat and lng fields. It also would make sense, I think, to specify you only accept GPS coordinates.

It might also be worth thinking about IIIF manifests as a carrier of spatial information. We currently use the GeoJSON service to express one or more spatial attributes, typically a Point. But the Point is often just the centrepoint of a polygon or bounding box, and we also specify bounding boxes in IIIF with values, whereas we specify polygons by references to a geospatial API that returns a GeoJSON data. Might it be possible to then use spatial data expressed in manifests for import or export of geo information?

Enabling the specification of the source of spatial information is a good idea as well; would you wish to provide a dropdown list or just free text entry?

[benwbrum]

On datatypes, it sounds like individual decimal (or float?) fields for latitude and longitude would be ideal, with ActiveRecord validation to enforce consistency. Moving to GIS-specific fields can be done in the future if we move to PostGIS -- I just don't want us to discover that we've painted ourselves into a corner when that happens.

On user interface, at the moment we're just talking about simple text entry fields. (There is an early-stage grant application to plug in georeferencing tools like Peripleo or Geo-locate--or even GeoKit--that would address better UI for lat/lon and external authority URIs. Again, we'd like to leave our options open for that--if it's approved--but not build that in this story.)

RGeo looks like it would provide the most logical path for full GIS support like the GeoJSON @jbhoward-dublin describes. (Speaking of which, @jbhoward-dublin, how is the GeoJSON data exposed and reused in your cases? We'd thought about exposing GeoJSON as annotations on canvases, since we're talking about marking up lots of placenames within a text. For a document like Alcedo's Diccionario Historico-Geographico, which has tens of thousands of places, how would you expose those?)

[saracarl]

We are at ruby-2.2.0 in prod, but upgrading to 2.2.2 shouldn't be that risky if we choose to use RGeo.

I think we're going to have to talk through this from a strategy point-of-view; this initial project doesn't do any lookups, but further projects have very specific look-up requirements (some are specific to project domain).

We can see the need to move to postgres in the future -- it supports a lot of great NLP functionality that we'd like to leverage as well. I don't think we'll want to move there for this initial geo project.

@jbhoward-dublin -- agreed on the IIIF export ideas. Likely not hard to do. We'll have to see what our collaborator at the WHG will want (probably not IIIF). Do we know of any apps that will consume a IIIF collection with GIS annotations and display the associated images on a map? Now that would be cool.

We have no thought about this at all, but will: Enabling the specification of the source of spatial information is a good idea as well; would you wish to provide a dropdown list or just free text entry?

[benwbrum]

One of the major deliverables for the Pelagios RDG is an export of place data in FromThePage as Pelagios Gazetteer Intetconnection Format, which turns out to be a subset of GeoJSON. We'll need to do this as a bulk export for the LatAm project, but should also be able to expose PGIF as annotations on canvases or manifests.

cc @kgeographer and @jbhoward-dublin

[jbhoward-dublin]

Re Do we know of any apps that will consume a IIIF collection with GIS annotations and display the associated images on a map?

This is not exactly what you describe, but UCD's Mirador maps plugin (on my github repo) parses the GeoJSON service metadata in a manifest, captures either coordinate values or references to a GeoJSON API, and uses an integrated visualisation app using MapBox/Leaflet to display a map. An example:

Manifest: https://data.ucd.ie/api/img/manifests/ucdlib:40789 Mirador view (click marker icon to display map): https://digital.ucd.ie/view-media/ucdlib:40863/canvas/ucdlib:42232 Map view in standalone mapping app: https://digital.ucd.ie/maps-geojson/?src=https%3A%2F%2Fdata.ucd.ie%2Fapi%2Fgeo%2Fv1%2Fdl%2FgeoFilter%3Dbbox(-6.269730311044%2C53.334592993114%2C-6.255084003975%2C53.340369960211)%26format%3Dgeojson&title_field=title&fields=title,id,pid,place,date,category,theme,centroid,geonamesreference,url,license&cluster=true&logo=https://digital.ucd.ie/images/logos/ucd_logo_sm.png&attribution=UCD%20Library,%20University%20College%20Dublin&iframe=true

So the plugin is exploiting the call to a geospatial data API to request geocoded items that lie within the bounding box of a georeferenced digitised paper map: https://data.ucd.ie/api/geo/v1/dl/geoFilter=bbox(-6.2843601643,53.340373126268,-6.269710164495,53.346121175475)

The GIS is returning the polygon is, by the way, a PostgreSQL/postgis resource.

[jackweinbender]

After looking more closely at the database schema, my instinct is to just add a place/GIS table (or maybe even "place metadata" if you're going for max-flexibility?) that stores the Lat/Lon separate from the article/subject itself (at least avoiding all the null values of adding the columns to all articles/subjects). If the need for more GIS-type data arises, then we can add more columns to the GIS table.

Assuming that additional type-specific data could be useful to add to other article/subject types, such an approach might be fruitfully implemented in conjunction with Single Table Inheritance which could allow for articles/subjects of differing types to access type-specific data through a unified API. STI, from my understanding, mostly is good for sets of data that share properties, but differ in behavior. Supposing we've pulled the GIS data into another type-specific table, the unique "behavior" would then become fetching the additional type-specific data.

Hopefully, that makes sense.

[benwbrum]

Subject categories get a new menu item for owners, "Enable GIS" or "Disable GIS" as a toggle, based on the value of a new categories.gis_enabled field in the model.

In the UI level, when Enable GIS is set on a parent category, that is percolated down to all of the children. (Ditto Disable GIS). When a new child is added to a category, that will inherit the parent's category.

When determining whether or not a subject has GIS enabled, we should so something like:

class Article
  def gis_enabled?
    self.categories.where(:gis_enabled => true).present?
  def

[benwbrum]

Please include some documentation with screenshots. (in wpengine site)

[jackweinbender]

Documentation completed: https://content.fromthepage.com/project-owner-documentation/enabling-gis-latitude-longitude-features/