Mini RFC - Consistent API for data loaders

[Pessimistress]

Background

deck.gl core includes two loaders: json and image

There are 4 official layers that currently require specialized loaders:	layer	loader	package dependency	can customize?	notes
MVTLayer	MVTLoader	@loaders.gl/mvt	no
TerrainLayer	TerrainLoader	@loaders.gl/terrain	no
ScenegraphLayer	GLTFLoader	none	yes, with registerLoaders	requires explicit import in app
Tile3DLayer	Tiles3DLoader	@loaders.gl/3d-tiles, @loaders.gl/gltf, @loaders.gl/draco	yes, with loader prop

There are multiple design inconsistencies here:

• ScenegraphLayer is the only layer that does not bundle the required loader nor specify a loaders.gl package as dependency. The user is required to install and import the loader themselves.
• We've seen confusion regarding whether @loaders.gl/gltf has to be explicitly installed when using ScenegraphLayer, because it's pulled in by @loaders.gl/3d-tiles which is the dependency of a different submodule.
• MVTLoader and TerrainLoader by default run on webworkers, but the full parser code is still bundled with the main app.
• There is a pending feature request to make TerrainLayer support other loaders, e.g. quantized mesh terrain.
• Tiles3DLoader has a significant bundle footprint. It also includes GLTFLoader and DracoLoader. This makes it pointless to exclude GLTFLoader from the ScenegraphLayer from bundle size optimization perspective.
• Tile3DLayer allows the user to replace the default loader by specifying the loader prop, however the default loader will be bundled regardless.

Proposal

High-level principles

• There should be a consistent story regarding how deck.gl supports different data formats, with its own dedicated documentation page.
• When possible, *WorkerLoader should be used instead of the regular loader. This makes the bundle footprint negligible, without breaking the existing behavior.
• Loaders that have big footprints may not be bundled by default, if the layer equally supports multiple data formats.
• All layers should offer a way for the user to add to/replace the default loaders.

Roadmap

• v8.4 (non-breaking)
  • [ ] Remove loader-specific code from Tile3DLayer. It should be handled inside loaders.gl after a loader is selected.
  • [x] Add loaders prop to base Layer class
  • [x] MVTLayer and TerrainLayer should use props.loaders instead of hard-coded loaders
  • [x] Add GLTFLoader to ScenegraphLayer by default
  • [x] Create documentation page on using deck.gl with loaders
• v9.0 (breaking)
  • [ ] Drop Tiles3DLoader from Tile3DLayer defaults, mark loader prop deprecated for loaders
  • [x] Switch default loader to MVTWorkerLoader and TerrainWorkerLoader

[Pessimistress]

cc @kylebarron @ibgreen @alasarr

[ibgreen]

This sounds great. Approved from my side.

Happy to help write the docs on how to use loaders.gl with deck. Do you want to get a skeleton page in place so I see how it is intended to integrate with the rest of the docs?

Some comments:

This makes it pointless to exclude GLTFLoader from the ScenegraphLayer from bundle size optimization perspective.

Not completely true, as there is a use case where the ScenegraphLayer can be provided with a programmatically built up scenegraph from luma.gl. But we don't have any examples showing how to do this, and I doubt it is being used, so I agree that we might as well include the loader.

When possible, *WorkerLoader should be used instead of the regular loader. This makes the bundle footprint negligible, without breaking the existing behavior.

If the bundle size issue is particularly worrisome, we could certainly build the non-worker loaders as separately loadable scripts as well. This is not done today, but the DracoLoader loads all the Draco libraries dynamically even when using the non-worker version so we have a good starting point there.

Note that a couple of issues in loaders.gl recently have been around how to use the loaders in air-gapped/no-network environments. Not sure how important this is for deck (use of tiled layers would presumably be ruled out), but it is nice if the user can provide the bundled, non-worker loader as an override. This should be possible with the loaders prop in this proposal, so all good.

MVTLoader and TerrainLoader by default run on webworkers, but the full parser code is still bundled with the main app.

Yes... just curious as to whether we use the output in binary form? If not we are probably not getting optimal performance here.