Dataset Collections and "AggPlaces" (new approach)

[kgeographer]

Designing a major modification to the existing feature, 'dataset collection' (DC), began walking through its implementation and bogged down at the required changes in mapAndTable.js. The implied changes to backend and frontend are closely intertwined, and I am unable to determine what the front end plausibly needs, nor am I able to refactor the frontend javascript.

The scenario being supported:

• a researcher or group of researchers want to assemble a gazetteer of a particular region or region/period from multiple datasets.
• in WHG, datasets (Dataset model) have places (Place model), which have a 'dataset' FK to a Dataset
• s/he or they have multiple datasets compiled from various sources (or are compiling), which they have uploaded to WHG and which are now fully accessioned; i.e. they have been reconciled against WHG's growing union index ('whg'), and any matches found between a dataset's place records are recorded in parent/child relations within 'whg' and will be also in a new database table, close_matches (CloseMatch model). close_matches is therefore in graph terms an undirected edge table; its meaning corresponds to the skos:closeMatch relation.
• DCs are created and managed in a multi-tabbed 'ds_collection_build' (dc_build) screen
• DCs are viewed in a 'ds_collection_browse' screen (dc_browse) with a full-screen map, and table, detail and intro/info overlay panels; this browse screen is private to the owner(s) until it is flagged public/published

Use case steps

• create a new empty DC (dc1), with a name and minimal metadata

• add an indexed dataset (ds1) to dc1: a record is created in collection_colldataset (id, collection_id, dataset_id, date_added)

• at this stage, ds_collection_browse (dc_browse) displays only the place records from ds1

• add a 2nd dataset (ds2)

  • a networkx connected_components() function finds any groups of records in ds1 and ds2 that have been matched during their prior indexing step (i.e. they appear as an edge in close_matches);
  • the nodes are all the Place ids (pids) of dc1 so far, from both datasets
  • the function returns a list of sets of >=1 place ids, e.g.
  • [{1234, 5678}, {9876}, {2468, 3579}, ...] NB place {9876} has yet to be matched

• now the table in dc1's browse screen must either:

  • Option 1: have one row for each set, not each Place instance
  • Option 2: have a row for each Place instance, but group those that have been linked with each other, with visual cue like alternating background

• as it stands, dc_browse receives a 'ds_list' context object containing only metadata for each dataset in the collection: ds_list.keys() = ['id', 'label', 'bounds', 'title', 'dl_est', 'numrows', 'modified']

• code in mapAndTable.js constructs a 'data' object for each dataset in the collection; these are aggregated into allFeatures[], which is used to a) render the table rows, and b) interact with the collection's map tileset features

challenges The refactor bogs down there -- allFeatures[] is generated from lists of individual features (Place records) in datasets. Now, each item in allFeatures[] needs to comprise a summary of data from a set of >=1 Place instances, not from a single Place instance

On the backend, the addition of a dataset to a DC must generate a new list of that collection's place sets. With the addition of a dataset, what was {1234, 5678} may now be {1234, 5678, 7654}, and the summary of that "AggPlace" needs to be reflected in the collection's dc_browse screen map and table.

If there is to be an AggPlace model/table, it would have at minimum fields (id, collection_id, pid_set), and before regenerating records upon a dataset_add() function, previous records for that collection would be deleted.

So...the dc_browse template would have to fetch a collection.agg_places() context object (not ds_list), and deliver that to mapAndTable.js, which constructs allFeatures[] from it.

Or...the generation of new AggPlace rows could include building the composite/summary info needed for dc_browse

Simple, eh? (not) Figures below for options 1 and 2 (#1 was favored by the team so far)

[docuracy]

I think a more efficient answer will be to:

1. ~Add a close_matches @property to the Place model.~

2. ~Extend the fetch_mapdata_ds function in datasets.utils so that if passed a close_matches=True parameter each feature it returns (i.e. each place) will include the pids of close matches.~

3. ~Adapt the mapAndTable.js code to perform the grouping, headword computation, and aggregation currently envisaged for the various functions in collection.views.py.~

4. ~Write a separate fetch_mapdata function for Dataset Collections, which aggregates the features into a single FeatureCollection, each feature having a matchset UUID and its dataset id.~

5. ~Adapt the mapAndTable.js code to group matching matchsets and to perform the headword computation and aggregation currently envisaged for the various functions in collection.views.py.~

This will much simplify dealing with geometry visualisation and events associated with the Datasets selector.

EDIT: no, make a single feature for each matchset, including a GeometryCollection (including a centroid representative point), headword, and aggregated types.