webcomponents / webcomponents.org

Home of the web components community
https://www.webcomponents.org
Apache License 2.0
359 stars 95 forks source link

[catalog-server] Manifests have a max size of 1MB #1358

Open justinfagnani opened 1 year ago

justinfagnani commented 1 year ago

Firestore has a limitation for the maximum field size of 1MB. Some manifests in the wild already exceed this size.

Options:

  1. Always use unpkg or similar CDN for manifests. This would add latency and network ingress for every page request, but unpkg is cached by Cloudflare and very fast.
  2. Store manifests in Cloud Storage. Cheap, fast, the preferred architecture for GCP. Requires making a blob storage interface for the Catalog class to use.
  3. Destructure the manifest into actual Firestore fields. JSON maps to Firestore's document model very well, and the manifest format is relatively shallow - each package contains a flat list of modules. Modules would exceed 1MB exceedingly rarely. This would allow for some more structured searching of manifest data, but we don't have use-cases for that yet.
  4. Compress the mainfest. Easy to do with current Firestore schema. We can serve the gzipped data directly, saving CPU.