search

quantified-uncertainty / metaforecast

Fetch forecasts from prediction markets/forecasting platforms to make them searchable. Integrate these forecasts into other services.
https://metaforecast.org/
MIT License
57 stars 5 forks source link

Picking a platform #5

Closed berekuk closed 2 years ago

berekuk commented 2 years ago

If we do my proposal from #4, which platform should we pick for the main Next.js instance?

Top options for me are: Vercel, DO and Netlify.

My own preferences:

NunoSempere commented 2 years ago

Right now, we're using netlify. I think this was originally @OAGr 's choice, and so far I haven't had complaints. But setting up a vercel instance isn't difficult, and I've done that in the past when I wanted to double-check that a tricky bug wasn't netlify-specific (it wasn't).

Netlify also has deploy previews (though not built-in analytics). My guess is that it's fine, and not particularly urgent or necessary to move away from it.

OAGr commented 2 years ago

We already have a few things in Netlfiy, so I generally want to have stuff there. However, Netlify is really just for the static frontend, we'd need something else for the backend.

I don't have big preferences between Vercel and DO for the backend. I generally like Heroku for simplicity, but realize it might be expensive here.

berekuk commented 2 years ago

However, Netlify is really just for the static frontend, we'd need something else for the backend.

I'm sure Vercel can run Next.js backend code, and I think Netlify can too. See also: my comment on #4.

berekuk commented 2 years ago

Ok, I've thought about it some more and I've changed my mind about being able to fit all compute tasks on Netlify/Vercel.

I guess we need both:

  1. Netlify for frontend and API.
  2. VPS (or a container service, managed k8s or whatever) for background processing.

The key issue is the background jobs for fetching forecasts. While Netlify supports background functions, there are some limitations:

We could just get rid of Netlify and serve the website from DO, but then we'd lose some performance (CDNs are great) and deploy preview features.

(To be clear, I still believe that running REST/GraphQL API and synchronous code on Netlify is fine.)

Opinions on how to setup this background compute instance:

OAGr commented 2 years ago

Everything above sounds pretty reasonable to me. We probably want something fairly simple to oversee and maintain at this point. (It's possible future full-time hires will have strong opinions on this topic).

I agree Netlify probably won't work well for background jobs.

I'm nervous about using too many platforms. We already use Heroku, Netlify, and a bit of AWS. Maybe Vercel wouldn't be that bad. Using a lot of Digital Ocean makes me feel kind of uncomfortable, as it's a different platform, but at least it's one of the simpler ones.

I agree that kafka/rabbit/airflow/etc are overkill now

NunoSempere commented 2 years ago

What are your thoughts on keeping the background jobs for fetching forecasts on Heroku, and then moving to DO/AWS if the costs start being too high, or if the complexity gains start to be worth it, @berekuk?

berekuk commented 2 years ago

Oh, right. We could just stay on Heroku and Netlify. Then we just need to drop the new DO API server (API will be served from Netlify), and drop Mongo eventually. Let's go this route then?

Actually, the Postgres database could be moved to Heroku too. Is there a specific reason (costs & limits, I guess?) to keep it on DO?

AFAIK Netlify ≈ Vercel, so we can just stay on Netlify and migrate to Vercel if the need arises, but Vercel won't solve background jobs issue.

If we optimize for the number of platforms then we could just self-manage everything, but it'll be much costlier in terms of the time spent on devops tasks. (Though I have some experience with setting up k8s + terraform + CDN from scratch, so that everything could be fit on AWS or on DO + any CDN)

Two related questions:

  1. Where is twitter bot's source code? I couldn't find it so far
  2. Where is AWS used?
NunoSempere commented 2 years ago

Oh, right. We could just stay on Heroku and Netlify. Then we just need to drop the new DO API server (API will be served from Netlify), and drop Mongo eventually. Let's go this route then?

Sure, by default, let's. Ozzie is a big fan of Heroku's developer efficiency gains, though I have the feeling that we'll still want to leave Heroku eventually. Setting up new dynos on Heroku is just really annoying because it gets really expensive quickly. And changes take a bit to propagate, as opposed to on a server, where they are ~instantaneous. So I'm fairly on the fence here, and if you instead want to go with DO instead of Heroku that's also alright for me. But I don't see a need for it right now.

Actually, the Postgres database could be moved to Heroku too. Is there a specific reason (costs & limits, I guess?) to keep it on DO?

The cheapest usable postgres database on Heroku was $50/month, whereas a roughly equivalent product was $15/month on DO. This bothered me enough to switch, which was ~trivial (just spinning up a database on DO).

I don't think AWS is used for metaforecast in particular, but maybe for other QURI projects.

The twitter bot's source code is in https://github.com/QURIresearch/metaforecast-twitter-bot; I just shared it. It's also hosted on Heroku. Feel free to have a look, but working on the webpage first feels like the better course of action.

OAGr commented 2 years ago

I don't think we're using AWS for metaforecast, but are for some other sites; like Foretold.io (for the Simple Email Service)

berekuk commented 2 years ago

I thinks this is done, thanks.

(~70% that I'll still try to argue in favor of Vercel later, btw, mostly due to Vercel pricing & limits being more generous than Netlify, but I'll open a different issue for that if necessary)