Major data retrieval refactor

[abought]

Ticket: #253

💣 This is a work in progress PR. It contains significant breaking changes that will affect every LZ usage, particularly involving the mechanisms of data adapters (retrieval) and how layouts deal with data. The syntax will evolve over time. 💣

Purpose

Revise and extend the way in which LZ retrieves data.

Decouples the act of retrieval from the act of combining datasets; for example, would allow any payload-compliant source to feed LD information, without needing to write a custom LD subclass in order to get the join logic.

More functional style to make request processing much easier to follow, debug, and extend.

Support multi-value subset caches (for smoother region switching + smarter LD)

Revise the current mechanism of namespaces- eliminate the distinction between "abstract" and "concrete" layouts by emphasizing locally (not globally) distinct field names

TODO

• [x] Move data-spec parsing from requester to layer init, and revise/finalize the join + dependency management syntax.
• [x] Heavily revise unit test suite for breakages.
• [x] Update all demos to working state.
• [x] Implement field contracts + extraction for more adapters
• [x] Rewrite or remove applyNamespaces function since local-view means no global-namespaces
• [x] Revise subscribeToData syntax to support new namespace behavior
  • [x] Allow subscribeToData to give a string (name of data layer we want to mirror) OR a ns+DO spec.
• [ ] Revise all docs to describe new format and write a migration guide
• [x] Rename JoinFunctions to DataOperations and support arbitrary number of data/option args (align with more generic spec)
• [x] Allow getElementID to parse a template string instead of using a field value as given. This will remove the need for synthetic IDs across many adapters, and improve object persistence across redraws.
• [x] Remove references to fields array across all tests, demos, etc.
• [ ] Explore mixins for dedup: data layers should be able to emit their own events, not just delegate to panel (useful for sourceID to be correct)
• [x] - Unset LDRefvar on region change if variant no longer in view
• [x] Don't autogenerate panel IDs if not present
• [x] Should data operations be given a copy of plot.state, so they can act with the same capabilities as annotateRecords?

[abought]

Merging into feature branch so that it can be tested in conjunction with TRAITS and PARSERS (each separate changesets, which are intended to work together towards several new features).

Rough release notes so far:

This release makes many internal changes to how LocusZoom retrieves data. It provides new functionality that should make it easier to create custom plots from various types of file.

• Namespaces have been redesigned as the central mechanism for plotting two datasets with one reusable layout.
  • The difference between "abstract" and "concrete" layouts has been eliminated. Syntax like {unnamespaced: true} or {{{{namespace[assoc}}afield}} is no longer necessary. All layers will now refer to fields by an internal name that is easier to read, write, and remember: assoc:afield.
• New data_operations make it possible to mix-and-match different datasets, by moving the join logic from adapter to the point where the data is used (the data layer).
  • This replaces the old combineChainBody method of adapters. Closely coupling the act of retrieval (API) to the join logic (display) made it hard to use data from somewhere else. For example, it is now easier to source custom LD from a local tabix file, without having to write custom code or use one specific base class whose assumptions do not match your data source.
  • This also makes it much easier for a single API call to be displayed in several different ways throughout the plot.
• Plots can now switch between different view regions more quickly, thanks to a new multi-value LRU cache that can eliminate unnecessary network requests.
• subscribeToData call signature has changed. It now accepts either an explicit specification of what data is requested, or the name of a data layer ("give me exactly what is being used to draw something that is already specified elsewhere").

Internal improvements

• TabixUrlSource now accepts a pre-made reader instance . This is useful for sites that want to validate file contents before adding the file to the plot.
• Data layers now accept a template expression for id_field. This means that data elements can be uniquely identified as a combination of several fields. This eliminates the need for "fake ID fields" in plots like phewas or coaccessibility, where no single ID field would have been globally unique. This also means that certain tooltip and mouse behaviors will be more stable as the user pans and zooms.

Deprecations and major breaking changes

• Major change to layout syntax
  • Namespaces have changed (useful migration regex: replace \{\{namespace\[(\w+)\]\}\} with $1:)
  • The fields array has been removed completely. Many people wanted to customize their data with just one extra field, and manually updating the contract of expected data proved cumbersome.
  • Adapters will usually return all the fields from the API.
  • All pre-defined data layer layouts now contain an _auto_fields variable that defines what information a layer expects. At the time of render, a user-friendly error message will warn if the server fails to provide expected fields.
• Many private variables were renamed. This should make the console debugging experience nicer, by hiding things that were not useful to most developers.
• Adapters have been completely rewritten in a more functional style, making them easier to understand and debug. Many methods have been renamed slightly to reflect that they have a different call signature; custom subclasses will need to update code.
  • Adapter options are now specified as top-level keys, instead of inside a nested key calledparams: {}. Existing DataSources.add calls will need to be widely updated.
• Require that panel layouts provide an explicit id . Most layouts already did this.

Bugfixes:

• If the user selects an LD refvar, ignore the choice when switching to another region far from that variant.
• Cached adapter data now does a better job of isolating each individual layer that uses the data. Previously, adapters had to return a string or else risk layers mutating a global shared reference.