aws / aws-proton-public-roadmap

This is the public roadmap for AWS Proton
https://aws.amazon.com/proton
Other
199 stars 13 forks source link

[Design] Service Sync high level design ✨ #65

Closed kohidave closed 1 year ago

kohidave commented 2 years ago

πŸ’• Howdy ya'll! We (the Proton team) have started work on a feature we're pretty excited about (GitOps management of services #11 ) and wanted to get your feedback on how we're thinking about designing our manifest for Service Sync! πŸ’•

What's Service Sync?

So one feature we've been really excited about for a while is enabling folks to sync their service specs directly from their git repos. Right now, service owners can set up a Pipeline that pulls a particular branch, packages up the code, and automatically updates the service spec file for a particular instance.

Service Sync, a new feature we're working on, let's folks store their spec files directly in their git repo. Proton will listen to your repo, and any time you update the spec file, automatically sync that change to a particular service instance.

How will it work?

The way we're thinking about this feature is you'll register your git repository with Proton (just like you do with Template Sync today), and then associate that repository with a Service.

Proton will then look for a special, magical file called the .proton-ops (we'll probably come up with a better name) which will instruct Proton on what to sync where.

What does the Proton Ops file look like?

So the .proton-ops file will contain all the information Proton needs to understand where a particular service spec is in your repository, and which service instance it should be synced to. Let's take a look at an example of a (very early, and very draft) version of this file:

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main: # When a commit happens on this branch
    sync: 
        # Sync the service-spec.yaml file on the main branch to the service instance daves-svc-beta
        to: daves-svc-beta 
  prod: # When a commit happens on this branch
    sync:
        # Sync the service-spec.yaml file on the prod branch to the service instance daves-svc-prod
      to: daves-svc-prod

In the above example, we're using a branch per environment strategy. We map the main branch to the daves-svc-beta service instance. Whenever we update the service spec on main, the daves-svc-beta service instance will be updated. Whenever we update the service spec on the prod branch, the daves-svc-prod service instance will be updated.

Multiple Services per Repo

One possibility that we want to support is letting folks sync multiple services from one repository. Let's see what that might look like:

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main: # When a commit happens on this branch
    sync: 
        - to: daves-svc-beta 
        - 
          to: bob-svc-beta
          spec: bob-svc/service-spec.yaml 
  prod: # When a commit happens on this branch
    sync:
      - to: daves-svc-prod
      -
        to: bob-svc-prod
        spec: bob-svc/service-spec.yaml 

As you can see - when syncing to a service spec, you can specify the location of the spec to use. If you don't provide an explicit spec location - Proton will default to ./service-spec.yaml.

Dependencies between service instances

Another feature we want to help folks with is ordering syncs. Within a particular branch we want to let folks tell us if there are dependencies between two service instances. Taking our example from above, let's have bob-svc depend on daves-svc:

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main: # When a commit happens on this branch
    sync: 
        - to: daves-svc-beta 
        - 
         to: bob-svc-beta
         spec: bob-svc/service-spec.yaml 
         depends-on: daves-svc-beta
  prod: # When a commit happens on this branch
    sync:
      - to: daves-svc-prod
      -
        to: bob-svc-prod
        spec: bob-svc/service-spec.yaml 
        depends-on: daves-svc-prod

Publishing

Another feature we're planning on supporting is the ability to publish after a commit is synced. This will let customers automatically issue pull requests to (for example), another branch. Let's see how this might look:

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main:
    sync: 
        to: daves-svc-beta 
   # Once all syncs for this branch are complete, Proton will create a PR from the main branch to the prod branch.
    publish:
        to: prod
  prod: 
    sync:
      to: daves-svc-prod

In this example, once all syncs from the main branch are complete, Proton will create a pull request to prod with the contents of the main branch. We realize that sometimes you might want more control over your publish (what files, what content within a file, etc). We'll save the design of what that might look like for another design issue.

Feedback πŸ‘‚

So I hope this helps keep ya'll up to speed with what we're trying to build! We're still early on in the design of this feature, so we'd LOVE your feedback. Specifically around the format of the proton-ops file. Is the experience intuitive? Do you have any recommendations? In a follow up post I'll show some other proton-ops file layouts we're considering and we'd love your help choosing one which works best.

kohidave commented 2 years ago

Manifest Designs

We're thinking about other ways to design our manifest file and would love your thoughts:

Option 1️⃣

Simple Case: Syncing a single service from multiple branches.

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main: # When a commit happens on this branch
    sync: 
        # Sync the service-spec.yaml file on the main branch to the service instance daves-svc-beta
        to: daves-svc-beta 
  prod: # When a commit happens on this branch
    sync:
        # Sync the service-spec.yaml file on the prod branch to the service instance daves-svc-prod
      to: daves-svc-prod

Advanced Case: Syncing multiple services from multiple branches.

# This is the contents of the .proton-ops.yaml file
service-sync: 
  main: # When a commit happens on this branch
    sync: 
        - to: daves-svc-beta 
        - 
          to: bob-svc-beta
          spec: bob-svc/service-spec.yaml 
          depends-on: daves-svc-beta
    publish:
        to: prod
  prod: # When a commit happens on this branch
    sync:
      - to: daves-svc-prod
      -
        to: bob-svc-prod
        spec: bob-svc/service-spec.yaml 
        depends-on: daves-svc-prod
kohidave commented 2 years ago

Option 2️⃣

Simple Case: Syncing a single service from multiple branches.

# This is the contents of the .proton-ops.yaml file
sync: 
  services:
    daves-svc:
      daves-svc-beta:
        branch: main
        spec: daves-svc/service-spec.yaml
      daves-svc-prod:
        branch: prod
        spec: daves-svc/service-spec.yaml

Advanced Case: Syncing multiple services from multiple branches.

# This is the contents of the .proton-ops.yaml file
sync:
  services: 
    front-end:
      depends_on: back-end
      beta:
        branch: main
        spec: daves-svc/service-spec.yaml
      prod:
        branch: prod
        spec: daves-svc/service-spec.yaml
    back-end:
      beta:
        branch: main
        spec: scnd-svc/service-spec.yaml
      prod:
        branch: prod
        spec: scnd-svc/service-spec.yaml
  publish:
    from: main
    to: prod
kohidave commented 1 year ago

Howdy ya'll! Just wanted to mention that we're moving forward with option 2! https://github.com/aws/aws-proton-public-roadmap/issues/65#issuecomment-1133335244

We figure it provides clarity and flexibility when syncing. Let us know what ya'll think!

haakond commented 1 year ago

Hi, this sounds great, would be very interested in trying it out.

kohidave commented 1 year ago

https://aws.amazon.com/blogs/containers/announcing-git-based-service-deployments-with-service-sync-for-aws-proton/

We launched :D Feel free to try it out and provide feedback! Thanks @haakond :D