Fallback to CDN imports when libraries needed by FileAttachment are not detected through static analysis

[mbostock]

The FileAttachment methods have implicit imports:

• FileAttachment.dsv, FileAttachment.csv, and FileAttachment.tsv use npm:d3-dsv.
• FileAttachment.arrow uses npm:apache-arrow.
• FileAttachment.parquet uses npm:apache-arrow and npm:parquet-wasm.
• FileAttachment.arquero uses npm:arquero, and maybe npm:apache-arrow and npm:parquet-wasm.
• FileAttachment.sqlite uses observablehq:stdlib/sqlite.
• FileAttachment.zip uses observablehq:stdlib/zip.
• FileAttachment.xlsx uses observablehq:stdlib/xlsx.

Normally we detect these through static analysis, but we can’t guarantee that we always detect which methods are used through static analysis. For example see https://talk.observablehq.com/t/404-errors-in-module-load/9623. A trivial example where static analysis fails is:

const file = FileAttachment("file.csv");
const data = await file.csv();

When static analysis fails, built sites currently break because they try to load the self-hosted copy of the library (which doesn’t exist because we only host copies of libraries that we know are used). Perhaps we should fallback to jsDelivr instead, so at least the site keeps working?

Alternatively, we could add a copy of these libraries even if we don’t detect that they’re used, but I don’t think people will like hosting copies of unused libraries.

Alternatively, we could make the static analysis heuristic more lax, for example looking for file.csv() calls and not just FileAttachment("name.csv").csv() calls, but that could have false positives.

Related #1574.

[Fil]

I think I'm in favor of option 2 "add a copy of these libraries even if we don’t detect that they’re used".

Self-hosting is a great feature, so option 1 is my least favorite (still better than option 0 = doing nothing, though).

Option 3 could be good too, but it feels like more work and would only cover "easy" cases; for example I don't think we would be able to cover the (contrived) case where you have a FileAttachment and a select dropdown that allows the user to choose what method they want to apply to read it. (Not that we should cover this particular case, but who knows what people will invent—a binary file reader app, maybe?)

But maybe I'm not seeing the drawback with option 2. Is your comment on "hosting copies of unused libraries" about the unnecessary file size, or about some other reason (like opening a door to some kind of mischievous access)?

[mbostock]

add a copy of these libraries even if we don’t detect that they’re used

We already rejected this option previously. (Yuri complained about it.) So I don’t think we’ll do that.

[yurivish]

I don't think I'd mind so much if the libraries were smaller, but some of them are quite large indeed. For example, parquet-wasm ships a 5.5mb .wasm file (1.7mb compressed). What would be the total additional file size added to dist if all default libraries were included?

Including these dependencies unconditionally without a way to turn that behavior off would also make it hard to post-process dist to delete unused libraries, since you couldn't easily tell if a dependency is included only due to the unconditional imports or not.

[mbostock]

DuckDB-Wasm is ~50MB which is even bigger, too. (Edit: Though it’s not in the current set of libraries that need implicit imports by FileAttachment. There was discussion about adding a duckdb method though.)

[Fil]

To clarify, these files would belong to the build, but they should not be loaded (nor bundled nor preloaded) by pages that don't use them. So for a viewer it should not change anything. The only additional costs are for the site owner: the time to upload the build, and the weight of hosting more bytes.