Online demo hosting - Githubissues

apedroferreira commented 1 year ago

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Summary 💡

From https://github.com/mui/mui-toolpad/issues/592.

Host online demo of Toolpad - with error reporting, analytics and captcha enabled.

Some questions to solve:

- Where should we host it? render.com as that's what we've been using for the Toolpad app.

- How should deployments be made? Render can deploy based on certain branches, should we try to use it or for example have a completely manual process for deploying the demo?

- What domain should we host on? From the discussion so far it seems like we're leaning towards hosting as a subdomain of toolpad.io - maybe demo.toolpad.io? Another suggestion was toolpad-demo.mui.com

Examples 🌈

No response

Motivation 🔦

By hosting the online demo users can finally access it and use it.

apedroferreira commented 1 year ago

@prakhargupta1 we may want to move this into the board to give it more visibility?

prakhargupta1 commented 1 year ago

What domain should we host on?

I am assuming we can't use mui.com/toolpad/demo for some reason? Maybe we should use: demo.toolpad.io

bytasv commented 1 year ago

I think we should be able to host even on mui.com/toolpad/demo we would just to make sure that there are no issues because of what's the base url is 🤔

apedroferreira commented 1 year ago

Probably mui.com/toolpad/demo would be too much trouble for what it's worth? This is a completely separate app so I think it would be good to host it in its own subdomain. I think demo.toolpad.io sounds great too if we can use that new domain.

prakhargupta1 commented 1 year ago

@oliviertassinari What do you suggest?

oliviertassinari commented 1 year ago

Render can deploy based on certain branches, should we try to use it or for example have a completely manual process for deploying the demo?

@apedroferreira I think that a new branch here would be more than enough. I see you used demo, sounds great.

What do you suggest?

Using a subfolder of the TLD (https://mui.com/) might lead to domain reputation "leaks". I would be afraid of:

SEO content that gets indexed by the community
Community member that writes spamming logic, running on the backend, that gets our domain blacklisted
maybe there are more risks?

But even then, from what I understand, when self-hosted, and in the cloud, it will always be / so I don't think that it would make business sense to invest resources to support a base path different from /. So, I think that we should remove this option from the list.

maybe demo.toolpad.io

Why not. Strapi uses https://api-iax04.strapidemo.com/. We also have toolpad.app and toolpad.dev available for us.

Regarding the configuration. Jan, Matt, and I are admins on these DNS, we can add a new entry as I did for https://master--toolpad.mui.com/, and we will be done;

https://app.netlify.com/teams/mui/dns/mui.com

and

https://dashboard.render.com/web/srv-c8uo8nn6d9klern4gk00/settings#custom-domains

Done: https://demo.toolpad.io/, feel free to change it.

Janpot commented 1 year ago

so I don't think that it would make business sense to invest resources to support a base path different from /.

Agree 100%

Render can deploy based on certain branches, should we try to use it or for example have a completely manual process for deploying the demo?

For me, an important criteria would be that the demo version can be deployed independently of the master branch. Secondary, I think from a dogfooding perspective, it could be interesting to treat it as self-hosting an instance of toolpad? Based on past experience, my recommendation would be to create a separate repository that holds our infrastructure as code. It contains a render.yml referencing the docker image just like any self-hoster would do. We can pin the docker image to a certain tag and in the future expand this blueprint with other instances. (e.g. our own MUI production instance, or a demo specifically for one client, or whatever). Technically this can be done within the same repo as well, but I feel like it would seriously clutter the commit log (since it would deploy from docker images which are only built after merging to master, a second commit will be required to update the image tag in render.yml)

edit

Another important aspect of using the docker image is that we don't have to overprovision the render.com web server like we currently do. We currently have increased the plan to accomodate for the memory using during the build command. Which is only executed the first time it has to start up. Afterwards, the app runs at a relative stable memory footprint, well under what's provisioned. Running from the docker image means we don't have to run the build command, and can choose an instance based on nominal required resources during normal operation.

oliviertassinari commented 1 year ago

Running from the docker image

@Janpot This sounds great.

my recommendation would be to create a separate repository that holds our infrastructure as code.

Ok, why not. Personally, I don't think that the noisy commit makes a difference. The main value of having a different repository for the demo seems to be able to feel more closely the experience that the developers that self-host have.

How should we call the new repository mui/mui-toolpad-demo? Or should we organize it more like having a private mui/mui-toolpad-infra repository that will host demos, future cloud, and future payment systems? I wonder how the other open-source projects handle this.

Regarding the version that it would use, I wonder if using latest wouldn't be better. This might help to keep it always up to date.

Regarding the option of using a branch, I assume that the idea would be for the demo to live inside this repository under a /demo folder and only deploy when a new commit is pushed to a specific branch, like demo-production. It also feels not too far from what developers will experience, maybe closer for those that have a mono-repository.

Janpot commented 1 year ago

Regarding the version that it would use, I wonder if using latest wouldn't be better. This might help to keep it always up to date.

It would still need to be deployed when the muicom/toolpad:latest is updated. In general I'm ok with this, as long as we have the option to choose a tag when necessary it's fine for me.

Regarding the option of using a branch, I assume that the idea would be for the demo to live inside this repository under a /demo folder and only deploy when a new commit is pushed to a specific branch

Ah, that may be possible now with their new monorepo support. If I recall correctly when we started using them only top level render.yml was possible

apedroferreira commented 1 year ago

How should we call the new repository mui/mui-toolpad-demo? Or should we organize it more like having a private mui/mui-toolpad-infra repository that will host demos, future cloud, and future payment systems?

Either sounds good to me. Maybe let's go with infra as it seems more flexible and future-proof?

Regarding the version that it would use, I wonder if using latest wouldn't be better. This might help to keep it always up to date.

I'm ok with using latest too, even though of course we need to deploy the app for it to be up to date. We can always make changes to the Dockerfile to deploy other versions.

Regarding the option of using a branch, I assume that the idea would be for the demo to live inside this repository under a /demo folder and only deploy when a new commit is pushed to a specific branch, like demo-production. It also feels close to what developers will experience, maybe even closer for those that have a mono-repository.

Could be, we would just need to give Render a path for the Dockerfile we want to use. But I guess we'll go with a separate repo anyway?

oliviertassinari commented 1 year ago

But I guess we'll go with a separate repo anyway?

@apedroferreira I only have a light inclination to use the same repository with buildFilter https://render.com/docs/monorepo-support#build-filters, a /demo folder, and a demo-production branch. The argument would be kind of dogmatic: a single repository is usually simpler to work with.

Feel free to overrule this. If we go with a new repository, I would vote for mui/mui-toolpad-demo on the name, to keep it simpler until we know more about how we will handle the cloud: will we host the source of the cloud offering in a public repository or in a private one? If public, will it be in the same repository as the rest of the code?

apedroferreira commented 1 year ago

@apedroferreira I only have a light inclination to use the same repository with buildFilter https://render.com/docs/monorepo-support#build-filters, a /demo folder, and a demo-production branch. The argument would be kind of dogmatic: a single repository is usually simpler to work with.

So this only deploys if there are any changes in a certain folder, that's pretty cool…

How about a /demo folder with the Dockerfile and deploying from the master branch? It would only deploy when we change the Dockerfile, but it would do so automatically without us having to do anything separately. On the other hand, this would only work with specific version numbers - if we use latest there would never be any changes inside the demo folder.
If we want to use latest we can push to a demo-production branch but we shouldn't use build filters, as there would usually not be changes to the Dockerfile. Build filters would not be needed, as in that branch only the demo web service would deploy.

If you think either of these works I'll go with it.

Feel free to overrule this. If we go with a new repository, I would vote for mui/mui-toolpad-demo on the name, to keep is simpler, until we know more about how we will handle the cloud: will we host the source of the cloud offering in a public repository or in a private one? If public, will it be in the same repository as the rest of the code?

Ok, if we go with a separate repo let's just use a repo for the demo for now. About the cloud, I guess we would have to discuss and decide later, but I assume our normal deployments from the main repo would be for our cloud version? So in that sense maybe there's no point in creating new repositories, at least not yet.

Janpot commented 1 year ago

@oliviertassinari

a /demo folder, and a demo-production branch.

I'm not sure I understand this setup

What would be in the demo folder? As far as I understand, to self host Toolpad on render.com, all we need is a render.yml with one docker service and one database. Nothing else from the repository is needed. But as far as I understand only one render.yml can go in the repository, at the root.
Why do we need a branch? The docker images are tagged, why wouldn't we use the tags to control which version is running in production?

Or are you proposing to create a branch that has a completely different render.yml from the master branch? And then we'd have to create and manage such a branch for each environment we add?

oliviertassinari commented 1 year ago

If you think either of these works I'll go with it.

@apedroferreira I think that both of the options that you have listed can work. Option 2. makes it easier to work with the demo without pushing to production straightaway. If always releasing latest is an issue, e.g. for reverting, we could not use latest.

What would be in the demo folder?

@Janpot I guess one Dockerfile like in https://github.com/render-examples/metabase and an environment variables to enable the "demo mode" and that's it? #1092

Why do we need a branch? The docker images are tagged, why wouldn't we use the tags to control which version is running in production?

I assumed it would help with not pushing to production as soon as we merge a PR on master, maybe it's not needed.

Regardless of how we host the demo, we might actually want to have repositories like https://github.com/render-examples, say mui/mui-toolpad-example-render that wouldn't be in the "Demo" mode (I assume the demo mode has limits that a true self-hosting doesn't have) and that we wouldn't maintain live.

Janpot commented 1 year ago

I guess a Dockerfile like in https://github.com/render-examples/metabase and an environment variables to enable the "demo mode" and that's it? https://github.com/mui/mui-toolpad/pull/1092

I don't think we can have render.yml in a repository subfolder

I assumed it would help with not pushing to production as soon as we merge a PR on master

At the time we are pushing to master, the docker image hasn't been built yet. So a deploy can only really be done say 20 minutes after that. If we want a deploy from docker image to happen automatically on pushes to master we'll have to trigger one at the end of CI https://render.com/docs/deploy-hooks.

bytasv commented 1 year ago

If we want a deploy from docker image to happen automatically on pushes to master we'll have to trigger one at the end of CI

Could we configure CI so that it adds one more job specifically when we push to master that would deploy it automatically?

oliviertassinari commented 1 year ago

I don't think we can have render.yml in a repository subfolder

@Janpot Agree, this wouldn't be in the demo folder, only the Dockerfile.

the docker image hasn't been built yet.

Agree, but then there is no staging environment, as soon as the new Docker image is used -> production. Why not, it might be overkill to have an other env 👍 .

Janpot commented 1 year ago

Agree, but then there is no staging environment.

I don't really see how a branch creates a staging environment. Basically, what I propose is a separate repository with:

# ./demo/Dockerfile
FROM muicom/toolpad:v0.0.22

# ./render.yml
services:
  - type: web
    name: toolpad-demo
    env: docker
    dockerfilePath: ./demo/Dockerfile

Scenario: You want to promote the next version to demo environment Solution: Change the tag to :v0.0.23 in the dockerfile, render.com will deploy when you push that

Scenario: You want to add a staging environment Solution: add another dockerfile and service and configure CI to call the deployhook of this blueprint at the end of build in the main repo.

# ./staging/Dockerfile
# NOTE: :latest tag
FROM muicom/toolpad:latest

# ./render.yml
services:
  - type: web
    name: toolpad-demo
    env: docker
    dockerfilePath: ./demo/Dockerfile
  - type: web
    name: toolpad-demo
    env: docker
    dockerfilePath: ./staging/Dockerfile

Scenario: You want to add a custom instance for customer ABC Solution: Add another service as per previous Scenario, but set the version to a fixed version to avoid automatic deployment without staging

Scenario: Something accidentally broke on master, it's 03:25 on a Sunday morning Solution: Set the version to the previous version in the dockerfile, push and go back to sleep

Question: Why not do this in the main repository in render.yml?

privacy: We may keep some parts of our infra private. especially when we do custom instances for customers
commit log: You want a deployment history? git log
usability: We may treat this repository differently, e.g. use it as a control panel of our infrastructure and allow pushing straight to master for admins (this is fine in a IAC repository if all you do is update the image version)
synchronisation: render.com build starts in the main repo, the image doesn't exist yet. Instead treat that render.yml as a staging environment and periodically deploy in the infra repo
dogfooding: you're 100% mimicking self-hosting scenario, no incentives to cheat

apedroferreira commented 1 year ago

Let's go with separate repo then? (mui/mui-toolpad-demo) I guess the branch deployments could be a bit of a weird flow, and deploying from master with build filters also would be weird for using latest or trying to deploy the latest image.

oliviertassinari commented 1 year ago

I don't really see how a branch creates a staging environment.

@Janpot I had in mind the following. In Netlify for MUI X, the HEAD branch is for staging: https://next--material-ui-x.netlify.app/ and the docs-v5 branch is for production: https://mui.com/x/. It's the exact same code that powers the two. We could do the same with Render: create one render app for each branch. It seems to be how it's configured right now: https://dashboard.render.com/web/srv-ccrjirarrk08u0m3dlmg/settings. The advantage is that the code different staging and production is smaller, what changes are env variables.

You want a deployment history?

With Netlify, I usually open the UI to see this information. Maybe it would be the same with Render, to be closer to the source of truth?

Privacy

So a new private repository then? With our current GitHub plan, we can't do branch protections on private repositories. We might need to change to upgrade our plan at some point.

Let's go with separate repo then? (mui/mui-toolpad-demo)

@apedroferreira I think that Jan meant to create a private infra repository: mui/mui-toolpad-infra.

Janpot commented 1 year ago

It's the exact same code that powers the two.

We can do that as well, but then we can't deploy from the docker image.

apedroferreira commented 1 year ago

We can do that as well, but then we can't deploy from the docker image.

That's an old deployment, we can do the same using the Docker image, like this PR would do: https://github.com/mui/mui-toolpad/pull/1092

I'm fine with any approach we've discussed, whether a separate repo or a separate branch.

oliviertassinari commented 1 year ago

I'm fine with any approach we've discussed, whether a separate repo or a separate branch.

Same for me. @Janpot Ping me on Slack if you need to create a repository but you don't have access rights.

Janpot commented 1 year ago

👍 Let's start with the branch approach first then and see where we get. we can always change if it doesn't scale.

apedroferreira commented 1 year ago

Reopened just to fix a few more issues in live demo:

[x] Disable basic auth in demo
[x] ~~Live demo thinks that I'm a bot and I can't pass the captcha - we might have to change the minimum score or use some alternative~~ (just needed to whitelist the custom domain)

oliviertassinari commented 1 year ago

I can't pass the captcha

Same for me

mui / mui-toolpad

Online demo hosting #1056

Duplicates

Latest version

Summary 💡

Examples 🌈

Motivation 🔦