improve HiGlass packaging / types

[manzt]

Motivation

HiGlass is well implemented and a core dependency of Gosling. However, it is untyped and packaged as a pre-bundled UMD JavaScript file. (pre-bundled meaning all dependencies except react, react-dom and pixi.js are included in hglib.js. This has lead to workarounds and inconsistencies in this repo, which could likely be solved all together by spending some time on improving how HiGlass is packaged.

Examples

• Maintenance of HiGlass "types" in this repo. Requires us to keep this repo in sync with HiGlass repo. If HiGlass published types from source code, then there is one place to make changes.

  • https://github.com/gosling-lang/gosling.js/blob/c00f4f8ae2fee485df7d153e56fb970815f649cd/src/core/gosling.schema.ts

• Code interfacing with higlass or higlass-modules (e.g., higlass-register) are almost entirely untyped. It works for now, but we could have much more confidence in this code which is so fundamental. Basically all data-fetchers are typed as any, and so much of the source code is just copied-pasted from elsewhere.

  • https://github.com/gosling-lang/gosling.js/blob/c00f4f8ae2fee485df7d153e56fb970815f649cd/src/core/higlass-component-wrapper.tsx#L7-L8
  • https://github.com/gosling-lang/gosling.js/blob/8bcbe7ef3f0b0b3a7b01d406131a6ae31a1d50f3/src/core/init.ts#L1-L4

• Shared modules in higlass and gosling.js aren't consistently accessed (imported vs used via HGC.libraries.*. For example, the d3-scale ends up twice in any gosling bundle (once from higlass and once from our import).

  • https://github.com/gosling-lang/gosling.js/blob/dfc7c0d19320d93d5d1c4e7f05d1626bce275428/src/gosling-track/gosling-track.ts#L111
  • https://github.com/gosling-lang/gosling.js/blob/1881e7df4607e64c2458689ee7f3bc40600a8dd5/src/core/mark/grid.ts#L1 Improving how HiGlass is packaged would make it so that much less code is duplicated, and decrease our large bundle sizes.

Conclusion

I'm not sure if any of this is on the Gosling roadmap, but as we continue to add features I think it's important to consider how we could improve its foundation for greater long-term stability. We have come up with a lot of ad-hoc solutions in this repo for using higlass more like a module, but many of those solutions would be eliminated and improved if we spent some time cleaning up higlass for our use case.

[sehilyi]

💯 agreed.

Some additional thoughts:

1. Another related example is that we use higlass.schema.ts which was semi-automatically generated at some point based on higlass.schema.json of the HiGlass core repo. We are using this ts file for constructing compiled HiGlass viewconfigs.

2. Regarding the duplicated codes of data fetchers between Gosling and HiGlass, I wonder if we can find a unified method, e.g., if a new data fetcher is implemented in one library (say Gosling), the other library (i.e., HiGlass) can also use the data fetcher, and vice versa. The current challenge is due to different use cases between Gosling and HiGlass. (1) Gosling primarily uses a tabular data format to use the concept of the visualization grammar while HiGlass data fetchers use varying data formats (e.g., data generated by a vcf data-fetcher is only interpretable in an sv track). (2) HiGlass data fetchers sometimes manage the render parts as well (e.g., BAM and VCF fetchers) while Gosling data fetchers do not care about rendering.