aws / aws-cdk-rfcs

RFCs for the AWS CDK
Apache License 2.0
517 stars 81 forks source link

Amazon CloudWatch Evidently L2 Constructs #428

Closed ljuti closed 5 months ago

ljuti commented 2 years ago

Amazon CloudWatch Evidently L2 Construct Library

Amazon CloudWatch Evidently lets application developers conduct experiments and identify unintended consequences of new features before rolling them out for general use, thereby reducing risk related to new feature roll-out. Evidently allows you to validate new features across the full application stack before release, which makes for a safer release. When launching new features, you can expose them to a small user base, monitor key metrics such as page load times or conversions, and then dial up traffic. Evidently also allows you to try different designs, collect user data, and release the most effective design in production.

Description

I propose an L2 construct library that cloud engineers can use to conduct Evidently launches experiments with CDK. The construct library would bring the Evidently capabilities closer to the development workflow, enabling quicker and smoother iterations and experimentation.

Core Constructs

Project

A Project is an AWS Evidently project where you group and track features, launches and experiments.

Feature

A Feature is an AWS Evidently feature, a set of behaviour in your application that you wish to launch or test.

Launch

A Launch is an AWS Evidently launch which you use to schedule or incrementally release a feature.

Experiment

An Experiment is an AWS Evidently experiment that helps you make feature design decisions based on evidence and data.

API Walkthrough with a Use Case Example

In an AWS News Blog article, author @sebsto demonstrates how to use AWS Evidently with a web application example.

Let's see how we would use the construct library to replace AWS Console tasks of the example.

1. Create a Project

import * as evidently from 'aws-cdk-lib/aws_evidently';

const project = new evidently.Project(this, 'GuestbookWebApp', {
  projectName: 'AWSNewsBlog',
  description: 'AWSNewsBlogDemo',
});

2. Add a Feature

const editable = new evidently.Variation({
  variationName: 'editable',
  valueType: VariationObjectValueType.BOOLEAN,
  value: true,
});

const readonly = new evidently.Variation({
  variationName: 'readonly',
  valueType: VariationObjectValueType.BOOLEAN,
  value: false,
});

const feature = new evidently.Feature(this, 'EditableGuestbook', {
  project: project,
  featureName: 'Editing',
  variations: [
    editable,
    readonly,
  ],
  entityOverrides: [
    new EntityOverride({
      entityId: 'seb',
      variation: editable,
    });
  ],
});

3. Create a Launch

const editableGroup = new evidently.LaunchGroup({
  groupName: 'Editable',
  feature: feature,
  variation: editable,
});
const readonlyGroup = new evidently.LaunchGroup({
  groupName: 'Readonly',
  feature: feature,
  variation: readonly,
});

const launchConfiguration = new StepConfig({
  groupsToWeight: [
    {
      group: editableGroup,
      splitWeight: SplitWeight.50_PCT,
    }
  ],
  startTime: <timestamp>,
});

const launch = new evidently.Launch(this, 'EditableGuestbook', {
  launchName: 'EditableGuestbook',
  description: 'Launch the editable guest book feature',
  groups: [
    editableGroup,
    readonlyGroup,
  ],
  scheduledSplitsConfig: [launchConfiguration],
});

4. Start an Experiment

const goal = new evidently.MetricGoal({
  desiredChange: MetricGoalDesiredChange.INCREASE,
  entityIdKey: 'user',
  eventPattern: 'eventbridge-pattern',
  metricName: 'Edits',
  valueKey: 'edits',
});

const treatmentOne = new evidently.Treatment({
  feature: feature,
  treatmentName: 'editing',
  variation: editable,
});

const treatmentTwo = new evidently.Treatment({
  feature: feature,
  treatmentName: 'readonly',
  variation: readonly,
});

const config = new evidently.OnlineAbConfig({
  controlTreatmentName: treatmentOne.treatmentName,
  treatmentWeights: [
    new evidently.TreatmentToWeight({
      splitWeight: 20000,
      treatment: treatmentOne,
    }),
    new evidently.TreatmentToWeight({
      splitWeight: 80000,
      treatment: treatmentTwo,
    }),
  ],
});

const experiment = new evidently.Experiment({
  project: project,
  experimentName: 'EditableGuestBook',
  metricGoals: [goal],
  onlineAbConfig: config,
  treatments: [
    treatmentOne,
    treatmentTwo,
  ],
});

Roles

Role User
Proposed by @ljuti
Author(s) @ljuti
API Bar Raiser @alias
Stakeholders @alias, @alias, @alias

See RFC Process for details

Workflow


Author is responsible to progress the RFC according to this checklist, and apply the relevant labels to this issue so that the RFC table in README gets updated.

serithemage commented 1 year ago

Is there a timeline for this proposal?

ljuti commented 1 year ago

@serithemage I haven't been assigned an API bar raiser, so this isn't likely moving forward unless there are sufficient backers for the RFC.

My PoC code is available here.

serithemage commented 1 year ago

@ljuti Thank you!

awsmjs commented 5 months ago

Closing this ticket. We believe the functionality is beneficial, but does not intersect with the core framework and should be vended and maintained separately.

flochaz commented 1 week ago

FYI, I published @ljuti work (with a small fix on projectName returned value) on npm: https://www.npmjs.com/package/aws-evidently-l2

hoegertn commented 1 week ago

Would that be something you consider contributing to https://www.npmjs.com/package/@open-constructs/aws-cdk

flochaz commented 1 week ago

Would that be something you consider contributing to https://www.npmjs.com/package/@open-constructs/aws-cdk

Nice, I didn't know about this initiative. Sure happy to open a PR there