aws-amplify / amplify-backend

Home to all tools related to Amplify's code-first DX (Gen 2) for building fullstack apps on AWS
Apache License 2.0
185 stars 62 forks source link

Split some of auth and functions e2e tests #2211

Closed sobolk closed 1 week ago

sobolk commented 2 weeks ago

Problem

The data-storage-auth-with-triggers-ts got crowded with rather advanced use cases.

This impacts:

  1. overal test runtime. (this PR improves that)
  2. hotswap runtime stability, i.e. hotswap often goes over threshold we define. (this PR is potentially improving that)

Changes

Extract functions and auth scenarios that can be standalone project.

Out of scope

I was thinking about extracting entirety of storage and functions scenarios but I noticed that we also use data-storage-auth-with-triggers-ts project in backend_outputs test. Therefore I decided to go slower and extract only scenarios that wouldn't impact that coverage.

Validation

PR checks.

before Overal runtime: 23m 26s image

after Overal runtime: 14m 30s image

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

changeset-bot[bot] commented 2 weeks ago

🦋 Changeset detected

Latest commit: 2e6418c1dc418bc4e3e0f450dfae8353ff861312

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

sobolk commented 1 week ago

The change is in the right direction.

I fear the new advanced e2e test will become a dumping ground to add unrelated concepts for testing. Ideally we should split this further and name them as specific as possible such that each test on average covers one customer use case.

@Amplifiyer That would be ideal. However, there are two constraints that are in the way (or will be at some point):

  1. Deployment/deletion of sandbox/cfn stacks overhead. Grouping couple of scenarios into single project let them share that burden.
  2. There's hard limit of 256 of jobs per workflow. We'll hit that eventually. We have a mechanism to run multiple tests on single worker, but that will eventually saturate too. Which will eventually drive us towards what I mention in point 1.

Given that. I think that the important aspect is to make sure (maybe find some abstractions?) that independent scenarios that are part of single project being deployed are not being coupled. (an example of that is using separate group of resources to test scheduled functions vs testing custom email sender, both use sqs queues, but their resources are separate from each other).