New App: Charts

[SachaG]

Current Setup

1. We crawl every chart in results in every language and take a screenshot using Puppeteer.
2. We upload all those images to a static site.
3. For every chart in results, we generate a stand-alone Gatsby page that contains just that chart.
4. That stand-alone page contains a meta tag that points at the image generated in step 1.
5. That stand-alone page also redirects back to the chart in the context of the entire results site.

Issues:

• Because all images are pre-generated, the system cannot support dynamic preview images such as for filtered charts.
• Because there are dozens of charts and dozens of locales, the image generation process takes multiple hours, and the regular Gatsby builds also take 15-20 min.

Setup Idea 1 (Better)

A separate API that:

1. Takes in the name of a chart
2. Serves the appropriate meta tags and images (still using the static images generated with the current setup)
3. Redirects back to the chart in the context of the entire results site.

Splitting this out into a separate API would dramatically speed up the build of the results app; and the separate API responses could be dynamically generated instead of statically generated like the Gatsby app.

Setup Idea 2 (Best)

A separate API that:

1. Takes in parameters such as the chart name, filters, etc.
2. Makes the appropriate GraphQL query to get the data.
3. Generates the chart image and also adds all necessary branding, layout, etc.
4. Caches that image on AWS or similar
5. Generates a link that points to that image as a meta tag; and also redirects to the image in the context of the full survey results.

[eric-burel]

Just to confirm, we cannot stick to SVG: https://indieweb.org/The-Open-Graph-protocol#Does_not_support_SVG_images So Vercel Satori is probably useless, unless Nivo charts are using HTML instead of SVG. Edit: but they do use SVG it seems.

Nivo API generates SVG server-side too, but not images. https://nivo.rocks/bar/api/

First step would be to find an alternative to Puppeteer screenshotting, light enough to be usable in a more dynamic fashion (so with requests spread over time, but also with more of them because of the combination of filters). Perhaps JSDOM allows that but I am not sure if it handles SVG either. Or a separate non-JS lightweight application that can render SVG. It's tricky to install stuff on either Render or Vercel though, I had issues with mongodb tooling for exports for instance. You need a full Docker image in this case...

You'll say that I am a tad biased, but I think Next.js is a perfect fit here, as it bring all the API needed to fine-tune the rendering process (eg pre-render some critical chart, configure incremental rendering for the others, make some totally dynamic etc. etc.). Route handlers are tailored to generate non-HTML content like images.

[eric-burel]

@SachaG anything left here since you finalized this?