The current mechanisms of data retrieval have several important limitations. We plan to refactor this in large, quite breaking ways, with a long term plan of simplifying syntax and reusability of layouts across LocusZoom.
This plan will affect long term LZ users, and so a preview of the details is below. This plan may evolve during implementation.
Problem:
The adapter class handles two responsibilities: data retrieval AND joins. The lack of separation makes certain tasks harder.
Eg, plots cannot power LD from a StaticSource. A custom subclass is not intuitive for new developers, nor always sufficient (eg when desired features are across two classes, like Static source for retrieval/caching and join logic in LDServer).
This also makes it hard to use the same kind of data on two tracks, because the "chain" assumes that every source has one specific, guaranteed precursor. it is hard to show data annotated on the plot and by itself in a table, without resorting to inscrutable hacks like "connectors".
Solution
Introduce joins as a first class entity at the point of usage (eg, a new configuration section in data layers, with a plugin registry for custom join functions)
Add a DAG-based dependency resolver to auto-parallelize requests, and allow formation of more flexible request chains.
Add an LRU cache so that the same adapter can feed multiple plots while still using cached data where appropriate in each. (eg multi-LD; single valued cache is insufficient)
Problem:
Layouts using data are sometimes difficult to read and customize.
Even when documented, no one enjoys the namespace syntax. It tries to provide globally unique names for a locally unique view of data, and achieves the worst of all worlds.
The namespace parser is regex based. It does not report malformed syntax because regexes either match or they don't. In absence of a namespace, it tries to fall back to a widely deprecated "base namespace" syntax
Solution:
Remove the difference between abstract and concrete layouts. Add new namespace spec with dependency syntax, and use those namespaces directly in layouts as assoc.variant, ld.correlation, etc. This will heavily rename many layout keys and affect a wide range of customizations.
Remove any concept of a base namespace and validate all requested entities.
Problem:
LocusZoom tries to support many kinds of APIs that have different views of data. In doing so, join tasks become very difficult to write in a reusable way (find a field with 5 known names, match to a field in the other source, hope data looks the same...)
Solution
Rewrite the BaseAdapter class to support a more functional style and simplify logic flow, so it will be easier to write new subclasses that control differences in format vs field contents, etc.
Introduce a new "fields contract" syntax where a layer can ask for the fields it uses, and an adapter can guarantee that it will be able to satisfy the request. Reject API responses that do not meet that format (or cannot be normalized/annotated to said format client side after parsing)
The current mechanisms of data retrieval have several important limitations. We plan to refactor this in large, quite breaking ways, with a long term plan of simplifying syntax and reusability of layouts across LocusZoom.
This plan will affect long term LZ users, and so a preview of the details is below. This plan may evolve during implementation.
Problem:
Solution
Problem:
Solution:
assoc.variant,ld.correlation, etc. This will heavily rename many layout keys and affect a wide range of customizations.Problem:
Solution