search

aws-quickstart / quickstart-illumina-dragen

AWS Quick Start Team
Apache License 2.0
24 stars 26 forks source link

Add additional test cases for taskcat #20

Closed ajfriedman18 closed 5 years ago

ajfriedman18 commented 6 years ago

Currently, we have tested happy path for e2e deployment for DRAGEN exome, f1.2xl, in us-east-1.

@vsnyc how many test cases would you like us to build for taskcat? The main variables are:

  1. VPC creation vs deploy in existing VPC
  2. EXOME vs GENOME
  3. us-east-1 vs us-west-2 vs eu-west-1
  4. f1.2xlarge vs f1.16xlarge

Combinatorially, that would be 24 different use cases, which seems a bit much. Here are my thoughts on testing - @mmmmmike would love for you to weigh in.

This tests all of the above but is non-exhaustive. Thoughts?

Also would want to test e2e job submission, which creates dependencies on #8 and #10.

vsnyc commented 6 years ago

The tests that run in our CI system are stateless so we don't run existing VPC scenarios. Also, it is not necessary to explicitly test that as testing the master template also tests the existing VPC template due to nesting.

We'll test all the supported regions, are these 3 the only ones? So we could do the test configurations as two or four tests:

  1. dragen-master, EXOME, f1.2xlarge in us-east-1, us-west-2, eu-west-1
  2. dragen-master, GENOME, f1.2xlarge in us-east-1, us-west-2, eu-west-1

If testing instance types is important:

  1. dragen-master, EXOME, f1.16xlarge in us-east-1, us-west-2, eu-west-1
  2. dragen-master, GENOME, f1.16xlarge in us-east-1, us-west-2, eu-west-1

If GENOME testing is not important, then tests count could be reduced to 2 (# 1 & # 3) or 1 (# 1)

ajfriedman18 commented 6 years ago

Thanks, @vsnyc. Testing all 4 is probably a good idea.

With TaskCat, is there a way to disable stack deletion? Full e2e testing will mean a subsequent submitJob call in AWS Batch.

vsnyc commented 6 years ago

Yes, you can pass a '-n' flag.

On Tue, Apr 3, 2018, 07:53 ajfriedman18 notifications@github.com wrote:

Thanks, Vinod. Testing all 4 is probably a good idea.

With TaskCat, is there a way to disable stack deletion? Full e2e testing will mean a subsequent submitJob call in AWS Batch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws-quickstart/quickstart-edico-genome/issues/20#issuecomment-378223731, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmGqWV2duQBJY8nMGiviGkJhIAaGgM8ks5tk2LPgaJpZM4TCmIj .