[x] fetch the largest data files perCountryData.json and perClusterData.json dynamically, instead of bundling them using Webpack
[x] initialize some of the recoil atoms using this JSON data asynchronously. The data fetching is done using the QueryClient from react-query, which ensures caching
[x] move directory web/data/ to web/public/data/, such that Next.js automatically copies data/ directory to web root, and that the files are made available on URLs like covariants.org/data/<filename>. The change of the data dir might require adjustment to various scripts. I did search & replace in all files in the repo, including python scripts, but this may require additional adjustment in external files if any.
Advantages:
Webpack no longer needs to process large data files, making app builds faster and consuming less memory
Data is fetched only when it's needed (but still in bulk)
Drawbacks:
data files are very large, so it might take some additional waiting
these files are now fetched in separate HTTP requests, which is additional possibility for failures
data is no longer type-checked at build time. If data does not match the expected format the app will crash at runtime
Further work:
Fetch the rest of JSON files dynamically as well
Subdivide JSON files such that
Don't store data on GitHub (the repo has become way too big)
In this version I:
perCountryData.json
andperClusterData.json
dynamically, instead of bundling them using WebpackQueryClient
fromreact-query
, which ensures cachingweb/data/
toweb/public/data/
, such that Next.js automatically copiesdata/
directory to web root, and that the files are made available on URLs likecovariants.org/data/<filename>
. The change of the data dir might require adjustment to various scripts. I did search & replace in all files in the repo, including python scripts, but this may require additional adjustment in external files if any.Advantages:
Drawbacks:
Further work: