search

osmbe / osm-welcome-tool

Website to help welcoming new OpenStreetMap contributors in your region
https://welcome.osm.be
ISC License
40 stars 87 forks source link

🧱 Better infrastructure #1413

Open jbelien opened 2 months ago

jbelien commented 2 months ago

Yes, this project is still maintained!

The current issue is that our (OpenStreetMap Belgium) server and the infrastructure (SQLite) worked perfectly fine for a few region but is undersized for the number of regions we have now!

I gave it some thought and here are a few suggestions/ideas:

We could also rewrite the whole application using other language(s) but that would require too much work.

You're more than welcome to contribute to this discussion if you have experience with one of those solutions.

jbelien commented 2 months ago

Platform.sh

The OSM Welcome Tool is developed using Symfony and Platform.sh is the hosting platform "linked" to Symfony: https://symfony.com/cloud/

The Symfony integration with Platform.sh has been created by the Symfony team and Fabien Potencier, Symfony project lead and founder, as a fully-managed platform optimized for you!

Rely on our Symfony expertise to offer a unique experience for anyone who wants to code and develop with Symfony. We conceived the Platform.sh's Symfony integration to cover all your needs while developing a project with Symfony without moving from your console.

The best way to deploy a Symfony project is Platform.sh. -- Source: https://symfony.com/cloud/features

Infrastructure

Provides all necessary features: app, PostgreSQL, queue/worker, cron job, ...

You don't seem to be have control on the hosting provider, this is handled for you based onnthe region you choose ; available regions (and providers): https://docs.platform.sh/development/regions.html

Pricing

https://platform.sh/pricing/

I can't immediately figure out how much it would cost us but I'm guessing something around 30 USD/month.

Laravel Forge

Infrastructure

Provides all necessary features: app, PostgreSQL, queue/worker, cron job, ...

You seem to be able to choose your hosting provider (which requires an account in that/those provider(s) ; providers: DigitalOcean, Akamai, Vultr, Amazon, Hetzner, ...

Pricing

https://forge.laravel.com/#pricing

Seems to NOT include server cost from provider (see above).

Laravel Vapor

Infrastructure

Provides all necessary features: app, PostgreSQL, queue/worker, cron job, ...

Provider: Amazon

Pricing

https://vapor.laravel.com/

39 USD/month + AWS usage cost (pay-as-you go)

jbelien commented 2 months ago

If we go for Laravel Forge or Vapor, might be worth migrating from Symfony to Laravel. It shouldn't be too much work and would allow us to enable features like Laravel Octane. This could be done in a second phase.

atomoil commented 2 months ago

I’m here cos @joostschouppe mentioned this to the local chapters & communities working group.

I’m very out of touch regarding modern PHP frameworks but it looks like you have a docker container so if that’s up to date then I wonder whether you’ve considered using a cloud provider (eg Amazon) and hosting via the container (eg https://aws.amazon.com/getting-started/hands-on/deploy-docker-containers/ ). Afaik all cloud providers offer somekind of docker/container option.

I think it should end up a bit cheaper but I appreciate it would be more/differently complex to get up and running in the first place.

xivk commented 2 months ago

I have quite some experience with hosting docker containers using Hetzner Cloud (https://www.hetzner.com/cloud/), probably even cheaper compared to Amazon and should be stable enough for this usecase. We host all our company stuff like that.

I would be willing to try this with @jbelien if he thinks this is a good enough option. But it's up to him, I also am very out of touch with PHP.

jbelien commented 2 months ago

Thank you both!

Docker containerization was indeed a plan I had a while back (see #766). I did deploy the OSM Welcome Tool (using Docker) on DigitalOcean App Platform because performances were really catastrophic (see https://github.com/osmbe/community_planning/issues/41#issuecomment-1286630486 for more details). I must admit I didn't try another provider because OSMBE uses DigitalOCean as hosting provider for the rest of our tools and I didn't want to have to manage 2 separate hosting providers.

I'm still convinced that Docker could be an option but looking at the time I could invest in this these last few months, I'm looking now for a "fully managed" solution.

I'm currently running tests on Plaform.sh and Laravel Forge that look promising (basically push update to main branch and wait for redeploy). After a few tests, there is already a clear difference between the 2:

I'm close to have a version of the app running on Platform.sh. I'll keep you updated.

jbelien commented 2 months ago

Test on Platform.sh

https://2-x-rai4ihq-b62c5e7ddypzg.fr-3.platformsh.site/

Daily automated update setup for Belgium, Berlin, and Brazil as test.

Price estimation: image

Test 1: Belgium + Brazil ✅ Test 2: Belgium + Brazil + 24 regios in Asia ✅ Test 3 all regions ✅

raphaelpe commented 2 months ago

Hello, I logged in to the website but the website is giving me an error.

I'm from Brazil and I started using this tool on the origin server and used it to invite new members of the community but it was no longer updated.

jbelien commented 2 months ago

@raphaelpe Ah yes, I also get an error... Most likely due to a dependency upgrade. I'll have a look 👍

raphaelpe commented 2 months ago

image

jbelien commented 2 months ago

Issue is fixed in https://github.com/osmbe/osm-welcome-tool/commit/102a13ba6b694f996e331dc544586388e7722ed8

jbelien commented 1 month ago

Since yesterday evening, https://welcome.osm.be/ is pointing to the new instance.

I'll copy the "historical" data from the previous instance to the new one in the next few days.

jbelien commented 1 month ago

image

Okay, it still needs a bit of work. Updates often go over the memory limit (screenshot show manual updates). Manual updates don't seem to be killed but cronjob well. I'll try to improve the process to reduce the memory usage.

jbelien commented 1 month ago

I've refactored the process (see adacee19751fbbcbfea164ca47091a306a668097) and it uses way less memory. Asia is being automatically updated since I've deployed the improvement. I'll keep an eye on it the next few days.

jbelien commented 3 weeks ago

Seems to work fine! 🎉

I've tried to restore the data from the "old" Welcome Tool. Unfortunately it's really a lot of data and we reach an Out Of Memory error when trying to open a region. I'll try to solve that issue when I have a bit more time.