Remove backend

[bmaupin]

Summary:

• Create a new repo to generate an SQLite data file as per https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/
• Update the UI to use the new data file
  • Hopefully we won't run into any cross-domain issues :crossed_fingers:; if so we could just put the data in the UI repo
• Use GitHub Actions to rebuild the new data file on a regular basis (daily? monthly?)
  • This might be helpful: https://octo.github.com/projects/flat-data
• Don't worry about cleaning up data; SQLite seems to be quite performant :tada:

We could probably remove the backend altogether (see further below for reasons):

• SQLite data file? https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/
  • Then in theory we wouldn't have to worry about the size of the data, cleaning up unused data points, having two separate data files, etc
• Use GitHub Actions or another CI to generate JSON data file
  • Daily? Monthly?
• Have two JSON data files?
  • One for all data, one for either all data from the last ~12 years or alternatively just the specific dates used by the frontend
  • e.g. monthly for the last 12 months, quarterly for the last 12 quarters, yearly for the last 12 years
• Put data into the same repo or separate?
  • Separate repos:: assuming we don't have cross-domain issues, I think keeping it separate makes more sense in order to keep the main repo from growing too large. It would mean we couldn't reuse the same logic in the frontend and backend, but I think that's a secondary concern.
  • We could possibly even re-use the existing API repo to create the JSON/SQLite file
  • Pros for putting data into the same repo
  • Can use the same logic where needed, e.g. can use the same dates to generate the smaller JSON file based on the dates to display
  • Wouldn't have to worry about any cross-domain issues
  • Should be more performant to use a local JSON file than have to do a web request I don't think this is true; the frontend would have to do a web request, either to the static site for the UI repo or the data repo either way
  • Pros for putting it into a separate repo
  • Main advantage: Having the data in a separate repo would keep the size of the UI repo down as the data grows (whether we decide to store it in JSON or SQLite)
  • Having it in a separate repo might be nice to keep the GitHub Actions separate between the UI code and the data

Reasons to remove backend:

• It was created as a POC for loopback, but in this case a simple JSON file would do just as well
• A JSON/SQLite data file should be more performant
• This would free up free Heroku hours for something else (e.g. Anki .apkg search backend, ...)
• Reduce complexity and possibly move all the code into one repo (this one)
• Shouldn't have to worry about loading screens and special messages/cases
• We might be able to remove the caching logic altogether

[bmaupin]

https://octo.github.com/projects/flat-data

[bmaupin]

Tried SQLite first but a quick POC made it clear that it would be more complex and end up being more data transferred: https://github.com/bmaupin/langtrends-data/tree/store-data-in-sqlite

So JSON it was :)