Open brian-norman opened 8 months ago
Thanks for the investigation @brian-norman! After taking a look at Neon, I think it's a great solution to our problem. We do have a complex enough structure now that a relational database would be useful. My only concern with Neon is the speed of the free tier. Based on my reading, it seems that the free tier is based on a shared compute, so we would be affected by high load on Neon servers themselves. I'm wondering if it would be worth running a quick prototype just to get a sense of the baseline free tier speed.
However, if we do plan on monetizing this product in the intermediate term, the free tier limits might not be a concern as we would have our own (semi) dedicated compute with the pro tier.
If we do go with the relational database (which I am also leaning towards now after learning about Neon), I have some suggestions for the proposed schema. Please let me know your thoughts:
As it sits right now, we have 4 different lists of works: Reviews, Watchlist, Backlog, and Nominations. I'm wondering if instead of having different schemas for each, we consolidate those into one schema with a type property? Something like:
CREATE TABLE lists (
list_id SERIAL PRIMARY KEY,
club_id INT REFERENCES clubs(club_id),
list_type VARCHAR(50), -- Types like 'watch', 'backlog', 'reviews', etc.
-- Add other list details as needed
);
That include items:
CREATE TABLE list_items (
item_id SERIAL PRIMARY KEY,
list_id INT REFERENCES lists(list_id),
work_id INT REFERENCES works(work_id),
created DATE
-- Add other item details as needed
);
I think this would give us more flexibility if we ever needed to add different lists like seasons or favourites etc.
With this model, the review_scores
table would likely have to stay, instead referencing a list item. There are ways of dynamic properties, but I think that just complicates matters.
As we start adding features for users to manage their own clubs, I see a situation where we might need to give the users more granularity in the permissions they're giving members. Because of this, I think it would make sense instead of having a is_admin
property in the club_members
table, have a role
property, which can take on various values like Admin
or Read Only
etc.
For the MVP, I think admin users are sufficient, I just want to make sure the schema is flexible in the future.
Just to make the schema more dynamic, I'm wondering if we should call the tmdb_id
column external_id
instead, in case we ever look at supporting other APIs for other types of works. As well, I'm just curious about the reasoning behind including release_year
in the works
table?
In a relational database paradigm, I'm not sure that the awards
table is necessary. In the document database, the awardYear
object was necessary as sort of a container, but I think in a relational database, we can ditch the awards
table and instead include club_id
and year
on the awards_categories
table. Then when we're querying for the categories in a given year, we can just index based on year
Let me know your thoughts on these suggestions.
Because all our data is based around the club, I'm wondering if it's possible to do an incremental migration starting with the club, then migrating each data type after that. Since there's not much for relationships between types below the club level, I don't think it would cause huge headaches, while lowering the risk of bugs as we'd be releasing smaller pieces of the migration at a time. I haven't thought it all the way through, however, so there could be a gotcha preventing us from moving incrementally.
On second thought, we will likely need a container Awards
object to keep track of the current active step of the awards ceremony, so disregard point 4 in my previous response.
Preface
We hate Fauna! The FQL language is confusing and not one we can get a lot of help from the wider community with (and therefore not ChatGPT friendly). We never even used it's big selling feature, the GraphQL support. Let's move away from it...
We can choose to go with another document based DB like MongoDB, or we can go with a relational DB like PostgresSQL. I'm personally leaning towards Postgres because I think it would have the most external support long term. There is a cool serverless platform called Neon that looks to have a good free tier, and a much more reasonable paid tier compared to Fauana. Both Neon and Postgres are open source, which I like :)
A Second Chance
Around two years ago, I made a bunch of commits straight to
main
, to merge the watch list, backlog, and reviews documents into a single club document. This was to support multiple clubs. That was a choice. @cole-adams claims that he'd never have let that merge if it was a PR, and he's right, I wish he stopped me in my tracks! In my defense, two years later, we have 4 actual clubs (+ 1 test club) using the site, and the schema has yet to fail us 😛If we move away from Fauna, we can use it as a second chance, to set ourselves up a little better for future work, and create less of a maintenance nightmare.
Looking Ahead
Here are a few things we'd want to have, if we started from scratch:
A New Schema
Here's one proposal, which I want to run by the gang this time, so we can actually come up with something together that sucks way less than our current schema. It's a Postgres SQL schema, which is a significant departure from our current model, but I think that it should theoretically set us up to do all the future improvements we desire.
Next Steps
I am not married to this solution, I wanna know what ya'll think, and if you agree that it's even worth doing. If so, we can come up with a project to do this migration, which I'm pretty sure will be no easy feat...
We'd want to minimize disruption to the other users, and not lose any data in the process. Ideally in this process we also further decouple our frontend and Netlify functions from our choice of database, so future migrations are less scary!