Create a development deployment

[abarciauskas-bgse]

It is desirable to have a second deployment for development so that changes proposed in pull requests can be tested in a deployed environment.

There are a few options for doing this:

1. Deploy feature branches to the development deployment directly using a pull_request trigger (ie whenever there is a PR to main, that branch gets deployed to dev). Pretty sure we do not want to do this as it could easily result in many deployments ($$ in github actions) and confusion about what is currently deployed to the development environment.
2. Have a "dev" branch and upon pushes to that branch, code on the dev branch gets deployed to the development environment.
3. Use a tool like https://github.com/github/branch-deploy to deploy feature branches to development.

All the options above make it easy to deploy to the development environment, which I think is desirable (easy to test changes). I think I'm not too concerned about how easy it is with only a few developers on the project but if there are more it could be that multiple people want to use the development deployment at once.

I have tried branch-deploy and didn't have success but I may try it again today.

Let me know your thoughts @vincentsarago

[vincentsarago]

@abarciauskas-bgse 🙏

I'm not sure we really want to get each new PR to create a brand new stack so I think having a dev branch that get's deployed on all push is fine.

We would have 2 stacks: production (deployed on tags in main) and staging

The staging stack could be deployed on two events:

• merge to a dev branch (meaning that we have to sync main from dev when we want to do a release)
• all push to main branch

I guess I have a more general question, what changes do we want to tests in a deployed stack that can't be tested in a pytest application?

[abarciauskas-bgse]

I'm not sure we really want to get each new PR to create a brand new stack so I think having a dev branch that get's deployed on all push is fine.

Sorry if my issue description was unclear - creating a new stack for each PR was definitely not my intention!

We would have 2 stacks: production (deployed on tags in main) and staging

I've always wondered why we use staging as the development stack name and not just development. But I could see a case for either name so if you prefer staging we can go with that.

The staging stack could be deployed on two events:

merge to a dev branch (meaning that we have to sync main from dev when we want to do a release) all push to main branch

presumably push to main would trigger a deployment to the production stack. Assuming the staging stack is already on the commit from the dev branch and the dev branch is then synced to the main branch upon release, do we need to redeploy to the staging stack?

I guess I have a more general question, what changes do we want to tests in a deployed stack that can't be tested in a pytest application?

I think it's useful to have at least one development (aka staging) stack to test out changes in the cloud environment. Specifically I think this is practical because the pytest tests do not replicate the docker image and AWS stack used to run the application in AWS. I definitely think we should include pytests for all new features and I will try and include new ones for the histogram method in #31

[abarciauskas-bgse]

Oh, I understand. We don't deploy to production until we tag and push a release. So the staging stack is always on the latest push to main or dev? 🤔 I think that makes sense.

[vincentsarago]

I've always wondered why we use staging as the development stack name and not just development. But I could see a case for either name so if you prefer staging we can go with that.

staging or development, up to you!

Oh, I understand. We don't deploy to production until we tag and push a release. So the staging stack is always on the latest push to main or dev? 🤔 I think that makes sense.

Yeah we could have a production stack should only be deployed on tags and all other push to either a dev branch or main be deployed to development stack.

let's create a dev branch, and then make it the default branch to this repo. once we are ready to do new release we will merge the changes to main and tags to release to production.

OR we only have one stack that get's automatically deployed on every new commits to main. We don't really need two stacks anyway.

[abarciauskas-bgse]

We are planning to use a deployment in the VEDA dashboard so I would like to try having 2 stacks for now.

To summarize:

1. We will have 2 stacks - one production and one development.
2. The production stack will be deployed when main is tagged (aka "released"). The production stack will have the domain prod-titiler-xarray.delta-backend.com/
3. The development stack will be deployed upon pushes to the dev and main branches. The development stack will have the domain dev-titiler-xarray.delta-backend.com/
   4. We will push dev to main whenever we have verified dev is working as expected.

I think I will revert the changes in #31 in ci.yml to implement the above in a separate PR.

Changes we need to make:

• [x] make dev branch the default branch
• [x] create a PR to deploy to the development stack upon pushes to main and dev and to production when main is tagged
• [ ] tag main and verify the deployment.
• [ ] create the prod-titiler-xarray.delta-backend.com/ and use it in veda-ui
• [x] update the dev-titiler-xarray.delta-backend.com/ to point to the development stack
• [ ] Add documentation for the above
• [ ] Verify alarm + notification are working

I will start making the above changes so let me know if you would suggest anything different.

[abarciauskas-bgse]

One question I have about this and similar setups in the past is it means there must be 2 PRs whenever a change needs to make it to main (i.e. one PR for the feature branch to dev and then a PR from dev to main). Is there a way to streamline this?

Case in point: After this PR is ✅ https://github.com/developmentseed/titiler-xarray/pull/36 I would still need to create a PR from dev to main before I can tag and deploy main to production.

What do you think @vincentsarago

[vincentsarago]

Ideally we don't have to make PR to directly pull change from dev to main to make a new release

[abarciauskas-bgse]

So we just merge dev into main after we have verified that the dev deployment is working as expected? I think that makes sense as then we should be confident that whenever we decide to tag main and deploy to prod that the code in main should be working.

[vincentsarago]

Absolutely

[abarciauskas-bgse]

@vincentsarago I would like to merge the dev branch into main and tag it so that we have production environment like dev and we can update the VEDA integration to use the prod-titiler-xarray deployment and not the dev-titiler-xarray.delta-backend.com/. This will also be useful to serve as a comparison as I intend to implement some caching and we can compare performance.

Questions:

1. Should there be a PR to merge dev into main?
2. I would guess the tag should indicate pre-release, so either 0.1.0 or 1.0.0-alpha, wdyt

[vincentsarago]

Should there be a PR to merge dev into main?

I think no PR is fine

I would guess the tag should indicate pre-release, so either 0.1.0 or 1.0.0-alpha, wdyt

🤷 Versioning do not really matter because we don't publish any packages, I'm fine using 0.X.0 version for now as long as we add a changelog 👍

[abarciauskas-bgse]

This is complete