search

cpinitiative / ide

A realtime collaborative IDE with code execution, intellisense, mobile support, and built-in USACO submissions.
https://ide.usaco.guide/
Mozilla Public License 2.0
350 stars 36 forks source link

Urgently migrate away from Firebase #83

Closed thecodingwizard closed 1 year ago

thecodingwizard commented 2 years ago

12/01/2022 update: we are now paying $27/mo for November, forecasted to be $54/mo for December (if not more bc USACO contest is coming up). need to urgently get this resolved

I think we'll soon be paying $15/mo for Firebase database costs (currently it's 2.75 gigs in size). We can either:

We have over 48k files, which averages to a little over 50KB a file, which seems reasonable. There aren't any massive files -- not too many files are over 10MB in size, and the largest file is 66MB.

Assuming we store 3GB of data a month and 700MB of downloads a day, that's around $17/mo

If we add 6k files a month and each file is 70KB each, that's an additional 420mb/month. And since we store all the files forever, that's an additional $2.10/month forever every month!

mjh316 commented 1 year ago

Just curious, what's the status on this? I just became a web developer, but this seems like a pretty prominent issue especially looking forward to future USACO contests.

thecodingwizard commented 1 year ago

We have ~5.83gb of data stored right now. Will probably pay $40/mo for December. The USACO contest surge wasn't as bad as I thought it would be (though it was definitely noticeable).

I have a plan on what the architecture of the app should be like with YJS; hopefully I'll write it up and put it on Github at some point. However, I don't really know when I'll get around to actually being able to implement this.

mjh316 commented 1 year ago

Gotcha, sounds good. Additionally, I was wondering if CPI is eligible for/would it be worth the hassle to apply for something like Azure for Nonprofits? It might be overkill, though.

thecodingwizard commented 1 year ago

We actually already have Azure for Nonprofits. Unfortunately, Firebase is from Google Cloud and we don't have Google Cloud credits. Thanks for the suggestion though :)

thecodingwizard commented 1 year ago

Substantial progress has been made: https://ide-git-yjs-cpinitiative.vercel.app/

mjh316 commented 1 year ago

Awesome! I took a quick look through the code, but haven't wrapped my head around YJS yet. However, is the basic idea to keep the Monaco editor on the front-end in sync with the document on the server via WebSockets?

thecodingwizard commented 1 year ago

Yes! The only thing that changes is FirepadEditor.tsx (and other relevant files). YJS seems quite complicated (I spent a very long time reading through documentation / source code) 😰 so I am trying to minimize the amount of YJS we have to use.

If we really wanted to migrate away from Firebase entirely, I think the remaining data we are storing on firebase (user accounts, file lists, file names, etc) can be moved to an alternative database like PlanetScale. This would plausibly simplify a lot of code (since having to load data from Firebase on the client side makes the client side code quite messy, at least with the current implementation). However, doing this would take a lot of time and we would have to implement google / user authentication ourselves which would be annoying

mjh316 commented 1 year ago

Yeah, I tried reading a bit of the YJS docs a while back and didn't get too far in, but it seems to work nicely for now which is great!

Regarding authentication, especially since we're just using Google as a provider, we could probably get away with AuthJS. I'm not too familiar with how permissions work currently but storing perms along with the file information in something like PlanetScale could probably work (?) as far as authorization goes? If so, then it wouldn't be much of a stretch to move off of Firebase as a dependency.

Alternatively we switch over to Supabase and do this whole thing again