search

webiny / webiny-js

Open-source serverless enterprise CMS. Includes a headless CMS, page builder, form builder, and file manager. Easy to customize and expand. Deploys to AWS.
https://www.webiny.com
Other
7.42k stars 613 forks source link

Multi-cloud support #1150

Closed Ashu96 closed 3 years ago

Ashu96 commented 4 years ago

Add support for GCP and Azure

Among all major cloud providers currently, we only support AWS out of the box. Support of other cloud providers like GCP and Azure is in our product roadmap.

The goal of this issue is to discuss the support of other cloud providers mentioned above.

gustavotr commented 4 years ago

I want to help with the GCP support, how do we start?

Pavel910 commented 4 years ago

@gustavotr First we need to figure out what alternatives do we have for CloudFront, API Gateway and S3 in the GCP world, as those are the main services Webiny depends on. Well, obviously Lambda as well, but for that we have functions.

So first we need a high level plan on how we will deploy Webiny as a whole (or partially?) to particular services.

gustavotr commented 4 years ago

For S3, Cloud Storage is an obvious choice. I know that GCP has an API service called API endpoints but I never used and don't know if if it has what we need. And for CDN we might be able to use Cloud Storage configuration.

Pavel910 commented 4 years ago

@gustavotr I'd be really interested in seeing a working PoC, not involving Webiny, just plain GCP setup with some simple code. We need to cover the following request flow:

CDN -> API Gateway -> Lambda -> S3 and back to the browser. Also it's important to be able to flush CDN cache via an API (it's used by our Page Builder, and is very important for SSR of pages).

Once we have that, we'll see what needs to be upgraded on the Webiny code side (we know of several things already, but it's not important until we know how GCP works).

gustavotr commented 4 years ago

I have managed to create a similar API Gateway -> Lambda -> S3 flow. I will add the CDN next but from what I have seen it's not very difficult. How can I share what I have done after I finish?

Pavel910 commented 4 years ago

Maybe you could share it as a github repo? Just create a repository on your account and post a link here. Thanks a lot for spending your time on this!! ❤️

gustavotr commented 4 years ago

I finally got some free time so I created a repo with my current progress. I still have to configure the CDN correctly. Probably I will have to configure with a real domain to make it work. Here is my repo -> https://github.com/gustavotr/serverless-api-over-cdn-on-gcp

gustavotr commented 4 years ago

Finally created the CDN and updated the repo. I had to use gcloud cli since the feature that creates a serverless network endpoint on terraform is on their master branch but did not get released yet. I believe that it will be available on the next one.

Pavel910 commented 4 years ago

@gustavotr Lovely, thanks a lot! I'll try to take a look at it one of these days, we're a bit overwhelmed with different tasks and calls, but I'm writing this down so it's not forgotten :)

ramonck commented 4 years ago

I'm looking forward to using Azure so looking at static web apps with Cosmos DB.

Pavel910 commented 4 years ago

@gustavotr I went through the code, thanks a lot for putting this together! 🚀

It looks pretty straightforward, but will will first need to introduce some improvements on the existing code base to be able to deploy to different cloud providers.

Main things to upgrade:

Just so you know, we'll be migrating to Pulumi for infrastructure deployments. They support a wide range of cloud providers, and make it really easy to define infrastructure via code (TS).

I'm going to create a separate issue for Function invocation plugin with some details on how to approach the problem. I'll tag you in the issue and if you're willing to give it a go, we'll be more than happy to help :)

gustavotr commented 4 years ago

Count me in @Pavel910 .

smartboyathome commented 4 years ago

I would love to see webiny running on Azure. It shouldn't be a hugely challenging migration, since it has a lot of equivalent services. Cosmos DB has Mongo support, Blob Storage is similar to S3, CDN and API definitions should be a pretty straight forward port as well. The largest challenge is likely to come from the porting of the Lambda to Azure Functions, since the latter is event-based, requiring you to define bindings and triggers.

ramonck commented 4 years ago

It's almost impossible to have all of these different APIs from PaaS of different companies working under one bucket. The closest you'll get to that is on the serverless product but it's focused only on the Functions part. It's either a "buy in" into those vendors or have a mesh Docker Swarm/Kubernetes setup, it's the easiest path. And depending on the case can be cheaper to sideload the nodes then to actually have the buy in into those flagship kubernetes, they're charging by the hour on those products it's insane to pay per hour for a controller specially depending on the size of the company it's ridiculous for smaller companies, just cheaper to do a Lightsail sideloading and done for the day.

Pavel910 commented 4 years ago

@ramonck the idea here is to have separate configuration templates for each vendor: AWS, Azure, whatever. When you create your Webiny project, we'll ask you which vendor you want to use, and will create infrastructure configuration specific for that vendor. From Webiny v5 we'll be using https://www.pulumi.com/ to handle all the infrastructure, and that will allow us to support different cloud providers.

So your project will not be deployable to any vendor at any time; instead, you will pick which vendor you want to work with, and that's what you'll get.

Other parts that remain to be improved on our side are storage interaction, database drivers, function invocation, but it's part of the challenge, and we're excited to tackle it :)

ramonck commented 4 years ago

@ramonck the idea here is to have separate configuration templates for each vendor: AWS, Azure, whatever. When you create your Webiny project, we'll ask you which vendor you want to use, and will create infrastructure configuration specific for that vendor. From Webiny v5 we'll be using https://www.pulumi.com/ to handle all the infrastructure, and that will allow us to support different cloud providers.

So your project will not be deployable to any vendor at any time; instead, you will pick which vendor you want to work with, and that's what you'll get.

Other parts that remain to be improved on our side are storage interaction, database drivers, function invocation, but it's part of the challenge, and we're excited to tackle it :)

@Pavel910 If you get it working with the main vendors that's huge, it's like the alternative to docker to me at least. Yes it will not work simultaneously but that's is fine because each project you want to focus on a vendor anyways is not like I want to use CosmosDB with AWS Lambda.. not for now at least :).

But thinking about the storage portion why not try to make it like they do with the docker approach concept where the default is the internal SQLite and then you can use some other storage if you actually configure it for the production scenario? Make it optional is that possible? Will that make life easier? The YAML approach to things.

Pavel910 commented 4 years ago

@ramonck by storage I meant file storage (sorry for not being clear). As for DBs, we're now working on supporting DynamoDB, so by default, we'll always setup a project for DynamoDB. Mongo and SQL will also be supported, if you need it. But again, if you're on AWS, Dynamo is an obvious choice.

Also, by supporting VPCs (also coming in v5), we're opening the gates for DocumentDB, Aurora, etc. so there's a lot of cool stuff coming soon :)

adrians5j commented 4 years ago

I had the need to respond to this question as well, just didn't have the time.

@ramonck

The closest you'll get to that is on the serverless product but it's focused only on the Functions part.

Actually, to setup Webiny, you "just" need services like API Gateway, CDN, Storage, Functions. And these kinds of services are provided by every major cloud provider, which means we should be able to deploy Webiny there.

Once a user installs Webiny on his provider of choice, it's up to him to pick more complex services, if needed.

As for databases, just to add to @Pavel910's answer, we did experiment with databases a lot, and for V5, we'll be relying on DynamoDb, which, as @Pavel910 mentioned, is a no-brainer, when it comes to AWS. But we are constantly thinking of multi-database support, because, for example, Azure doesn't have DynamoDb. So, we are trying to model the data in a way that will be acceptable for other databases as well. We will hopefully do some experiments and testing with DocumentDb / MySQL in the following weeks, so we'll see how it all works together.

adrians5j commented 4 years ago

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

smartboyathome commented 4 years ago

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

I know I am too late on this particular piece, but while I like the pulumi offering, I have hesitations due to self hosting being restricted to their enterprise offering. Hopefully you guys can document the setup process so that it can be replicated on something more friendly to this like Terraform.

Pavel910 commented 4 years ago

@smartboyathome what do you mean by self hosting being restricted to their enterprise offering ? Pulumi is open-source and you are deploying to your own cloud account (AWS, Azure). We're doing that for months already, deploying our stacks to our own AWS accounts.

adrians5j commented 4 years ago

https://www.pulumi.com/docs/troubleshooting/faq/#can-i-use-pulumi-without-depending-on-pulumi-com @smartboyathome

smartboyathome commented 4 years ago

I see that my info was out of date. I was referring to the state storage. Apologies.

ramonck commented 4 years ago

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

@doitadrian I'm still learning about webiny but I understand the main ideas I believe no clue about the code, if you want some coding help just send me what you can and I will be glad to help and publish in whichever branch so we can make it work. If you can tell me were the AWS part is I can start from there that's fine too. Not sure how pulumi works but if it's NodeJS I will figure out.

adrians5j commented 4 years ago

Cool @ramonck, I'll try to polish up a couple of things over the weekend and in the beginning of the next week, and then provide some instructions.

BTW which cloud provider would you be interested in trying to tackle? And which database?

Thanks a lot! 🍻 🍻

ramonck commented 4 years ago

Cool @ramonck, I'll try to polish up a couple of things over the weekend and in the beginning of the next week, and then provide some instructions.

BTW which cloud provider would you be interested in trying to tackle? And which database?

Thanks a lot! 🍻 🍻

Azure and CosmosDB.

adrians5j commented 4 years ago

Just a short update @ramonck, please hold on for a couple of more days (I've been busy with some other stuff). I will start working on preparing the necessary parts for you tomorrow, with full attention to it.

Will provide updates ASAP.

ramonck commented 4 years ago

Ok no worries @doitadrian , would be nice to maybe put it on the README/Wiki so others can also jump in and help as they can and you can have more people helping the library and growing the features around it. The more you share on this piece the more plugins others can help you guys out there in this perspective so maybe we can be an army of a couple and not only us.

alanreynosov commented 4 years ago

I think this link could be useful to know what are the equivalencies of resources required by webiny in GCP and Azure https://www.datamation.com/cloud-computing/aws-vs-azure-vs-google-cloud-comparison.html

adrians5j commented 4 years ago

Very nice @alanreynosov!

BTW just a short update from my side...

Still doing some refactors and testing. Actually, this is my main focus this week, and will probably be in the next one as well.

More time was needed for abstracting a couple of places and make them multi-cloud compatible. And there is still a couple of places to check.

But it's going well I'd say and we'll probably have something ready next week.

I will update you as soon as possible.

P.S. The branch I'm working is feat/v5/pulumi, so you can see the activity / changes there.

ramonck commented 4 years ago

Hey no problem, I've been working late so no worries no rush. Crazy times.. lots of demand.

webiny-bot commented 4 years ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

gustavotr commented 4 years ago

Any updates? Can I try integrating GCP with pulumi branching from feat/v5/pulumi, @doitadrian?

adrians5j commented 4 years ago

Hey @gustavotr,

sorry for the delay.

We didn't manage to do the thorough multi-cloud support research, that's why this issue hasn't received any updates.

We're extremely busy with wrapping Webiny v5 up, and I'm sure we won't have the time to concentrate on this until the end of the year.

I think it might be just better to pause this until we officially release v5.

That being said - if you still want to play with it on your own, and slooowly work on a solution, feel free to do so, and we can check it out once you have something. The branch from which you can start is the v5 branch.

Who knows, maybe once we finally reach the multi-cloud, we'll be able to use what you've created as an official solution. But just have in mind that we won't be officially doing multi-cloud for now, and we won't be able to focus a lot of our time on it.

Here's 🍺 to having the Webiny v5 out as soon as possible. 🙂

gustavotr commented 4 years ago

🍺

webiny-bot commented 3 years ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

itpropro commented 3 years ago

I would like to run webiny on Azure Functions, with the possibility to use durable-functions, it seems like a much better solution than Lambda. As v5 is out, are there any update on this @doitadrian?

adrians5j commented 3 years ago

Sorry @itpropro, no updates here, unfortunately. We just had a lot of other things to deal with along the way.

But it's still something we want to revisit in the near future.

itpropro commented 3 years ago

Thanks for the update @doitadrian! Let me know if you need any help or testing with Azure Functions

ajcwebdev commented 3 years ago

I've been researching this general problem space with @t-eckert. Can confirm this is wicked complicated and resists any sensible time estimation. I think the two most fundamental things you need to understand is how they each handle events and how they integrate HTTP.

Handling events

AWS Lambda uses a straightforward event-driven architecture that does not depend on event sources. A function:

The event type defines the schema of those objects, which are documented and defined in language SDKs. Each service that integrates with Lambda sends data to your function in JSON as an event. The structure of the event document is different for each event type, and contains data about the resource or request that triggered the function. Lambda runtimes convert the event into an object and pass it to your function.

With Azure Functions, a trigger is an event that the function listens to. The function may have any number of input and output bindings at the time of processing. It can read a document from Cosmos DB and send a queue message via binding configuration. If you want event sourcing with Lambda you need to grant the Lambda permission.

Event source mapping

For services that generate a queue or data stream in Lambda, you create an event source mapping and grant Lambda permission to access the other service in the execution role. An event source mapping is an AWS Lambda resource that reads from an event source and invokes a Lambda function. Lambda reads data from the other service, creates an event, and invokes your function. You can use event source mappings to process items from a stream or queue in services that don't invoke Lambda functions directly.

HTTP Integration with Lambda

AWS Lambda listens to HTTP traffic many different ways:

HTTP triggers with Azure

Azure Functions, on the other hand, come with HTTP triggers for integrating HTTP traffic. By default when you create a function for an HTTP trigger, the function is addressable with a route. The trigger input type is declared as either HttpRequest or a custom type. If you choose HttpRequest, you get full access to the request object.

For a custom type, the runtime tries to parse the JSON request body to set the object properties. The HTTP output binding responds to the HTTP request sender. This binding requires an HTTP trigger and allows you to customize the response associated with the trigger's request.