163719 Create templates for AWS and Azure resources

[robe070]

Resourcing: 2 devs for creating templates in AWS and Azure. 1 dev for writing DB backup and restore scripts 1 dev for writing stack management script

CREATOR:

Symptoms/Effect*

The starting environment will be setup manually in both AWS and Azure. The VM will be saved as an image already and a new VM instantiated from the image (part of CCS 163728). So the VM has been 'backed up'. This task is to automate the creation of all the resources, including the controlling VM, including taking database snapshots/backups and restoring those when re-creating the stack.

All resources created by the templates must be new resources leaving the existing resources in place until its proven that the templates are re-creating the manually created environment exactly.

So the new environment must also be tested thoroughly as part of this task.

Refer here for the Architecture: https://github.com/lansa/db-regression/wiki

A script will control the creation and initialisation of the stack. A stack is created for a LANSA Version and all resources are tagged with LansaVersion=[version number].

The creation of the stacks from the templates must be automated into a PowerShell script which must first backup the existing databases in the 'other stacks' and restore them to the new stacks. e.g. backup GREEN and restore to BLUE.

This script is the basis of the pipeline start up script. It needs to be idempotent. That is, its run to create the resources and also to ensure the resources still exist and are running, for example to be run by the pipeline before running the tests.

Backing up the database is a separate script thats run manually as part of the LASNA Upgrade process when creating a new stack for a new version of LANSA. This is the first script to be written.

The best way to get a summary of what creating a stack is required to do, refer here: https://github.com/lansa/db-regression/wiki#upgrade

[robe070]

Note that rather than a script it will be a serious of Azure DevOps Stages, Jobs and Tasks, some of which will use scripts.

Note that inline Powershell scripts are not permitted in final code. They must only be used for developing scripts, if at all. Often its better just to develop the script on the VM or on your own PC, and then once its completed, call it from the Pipeline.

[robe070]

The stack currently consists of the following resources: All resources are tagged Stack:db-regression-green Azure SQL Database AWS Oracle RDS AWS mysql RDS All the rest of the databases are in the VM image. So any need to restore those is a matter of re-creating the VM from the image. And the EC2 VM - Instance: i-0651ebadbbcd45823 (Database Regression Test)

[robe070]

N.B. All the existing resources must be left as they are. Team members developing the pipeline and scripts will need to be using the existing resources whilst the template is being constructed.

[robe070]

The AMI to use for the VM MUST BE DERIVED. It will change. So query the AMIs in us-east-1 for tag LansaVersion=[version #] There must be only 1 AMI that meets this criteria. If there are 0, or more than 1, it is a fatal error.

[robe070]

@MohitArora001 a thought has come to mind that the stack management script will be used in two ways. Its implied by what I have written and maybe its not clear.

1. Firstly by the architect to create a new stack from an existing set of database backups, Say TEST to 150050. Or 150050 to 150060. In this case the database state to restore is TEST when creating stack 150050. Or 150050 when creating stack 150060. This could be considered a 'Clone'.
2. Secondly, by the pipeline to ensure that a particular stack is ready to be used. e.g. Stack 150050. In this case, the backup to restore for stack 150050 is backup 150050.

So the stack management script will require an optional input for Version to 'Clone' from.

And when cloning, its also the image thats to be used.

[MohitArora001]

@robe070 Yes we can add this optional parameter to clone the database.

[robe070]

@MohitArora001 There are 2 AMIs labelled with the 150050 version tag. There should only be one. Why have your tests not shown this? Oh. And I've just created a third one! Maybe you've been using an old VM tagged with "123"? Why are you not using 150050?

[MohitArora001]

@robe070 I have tested with version 150050 and my script says Found more than 1 AMI for the tag 150050, below is the snippet for your reference and I have been using tag 123 as I have created new AMI from existing one and tagged it 123 and also created RDS and tagged them with version 123 So that i can do my testing.

Pipeline URL : https://dev.azure.com/VisualLansa/Database%20Regression%20Test/_releaseProgress?releaseId=151&_a=release-pipeline-progress

[robe070]

@MohitArora001 Ran .\aws_stack_provision.ps1 150050 test

It soon fataled. You are accessing a null object. You can't count using this method because count of 0 will fatal. There will be a number of places in your code that need to change.

Searching for VM with Lansa Version tag = 150050
C:\Users\Robert.SYD\Documents\GitHub\db-regression\Scripts\aws_stack_provision.ps1 : The property 'Instances' cannot be found on this object. Verify that the property exists.
At line:1 char:1
+ .\aws_stack_provision.ps1 150050 test
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [aws_stack_provision.ps1], PropertyNotFoundException
    + FullyQualifiedErrorId : PropertyNotFoundStrict,aws_stack_provision.ps1

[robe070]

@MohitArora001

1. Name of VM needs to have LansaVersion appended as in DB Regression Test 150050

2. Snapshot not found:

   Searching for Oracle RDS with Lansa Version tag = 150050
   No existing oracle database exist with lansa version tag = 150050
   Looking for snapshot database identifier oratest...
   C:\Users\Robert.SYD\Documents\GitHub\db-regression\Scripts\aws_stack_provision.ps1 : The property 'DBSnapshotArn' cannot be found on this object. Verify that the property exists.
   At line:1 char:1
   + .\aws_stack_provision.ps1 150050 test
   + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : NotSpecified: (:) [aws_stack_provision.ps1], PropertyNotFoundException
   + FullyQualifiedErrorId : PropertyNotFoundStrict,aws_stack_provision.ps1

   This is because its looking for the DB Identifier e.g. ora19cdb, not the Snapshot ID 'oratest'. The code should use -DBSnapshotIdentifier

3. MySQL Similarly

[MohitArora001]

@robe070 If count is 0, it never goes to fatal it outputs 0 only, Below is the snippet for your reference where I am trying to count the no of instances(It would be 0 as it dont exist) for some random LansaVersion tag that dont exist in the system.

[MohitArora001]

@MohitArora001

1. Name of VM needs to have LansaVersion appended as in DB Regression Test 150050
2. Snapshot not found:

Searching for Oracle RDS with Lansa Version tag = 150050
No existing oracle database exist with lansa version tag = 150050
Looking for snapshot database identifier oratest...
C:\Users\Robert.SYD\Documents\GitHub\db-regression\Scripts\aws_stack_provision.ps1 : The property 'DBSnapshotArn' cannot be found on this object. Verify that the property exists.
At line:1 char:1
+ .\aws_stack_provision.ps1 150050 test
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [aws_stack_provision.ps1], PropertyNotFoundException
    + FullyQualifiedErrorId : PropertyNotFoundStrict,aws_stack_provision.ps1

This is because its looking for the DB Identifier e.g. ora19cdb, not the Snapshot ID 'oratest'. The code should use -DBSnapshotIdentifier

3. MySQL Similarly

1. I will append lansa version to the name of the VM.
2. We are using the Database Identifier purposely to make sure our snapshots are derived from the right database. There is also a chance that more than 1 manual snapshot exist for the database but only 1 database snapshot is tagged as a oratest and mysqltest So using the database identifier gives you all the snapshot created from oratest and mysqltest.

[robe070]

The snapshot is what is important, not the database it comes from. Clearly. Its not working with Clone_version = 'test'

When a Command returns a NULL object, and then you reference a property - .Instances - PowerShell throws an error. This is happening. I'm not code reading. Its a runtime fatal error.

[robe070]

Now I have an instance this code is failing:

$EXISTING_INSTANCE_COUNT = (((Get-EC2Instance -Filter @{ Name="tag:LansaVersion"; Values=$lansa_version }).Instances).InstanceId).count

The property 'count' cannot be found on this object. Verify that the property exists.
At line:1 char:1
+ (((Get-EC2Instance -Filter @{ Name="tag:LansaVersion"; Values=$lansa_ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : PropertyNotFoundStrict

Without .Count it returns i-0584eb71308b471ca

[MohitArora001]

Could you please help me to reproduce the Fatal error when Instance count is 0 because I have tested it locally and from pipeline and its working fine. As I have attached snippet for your reference. Please help me with the tags that you are testing

For snapshot If I understood right you executed .\AWS.ps1 150050 test here 150050 stack needs to be created and test would be used to clone the 150050 stack. So we need to make sure oratest and mysqltest database snapshot exist in the system.

[robe070]

I'll remove the tag from the 150050 instance and try again.

[MohitArora001]

It found count 1 as 150050 VM exist

[MohitArora001]

Now after you have removed the tag 150050, count is 0

[MohitArora001]

It found for tag 150050x

[robe070]

Then it must be because I have an older version of the AWS CLI. I'll upgrade it.

[robe070]

What if you put in a value that does not exist? Does that return 0?

[MohitArora001]

Yes it returns 0

[robe070]

Updating AWS Powershell has fixed the .Count issues

[MohitArora001]

Great!! and please suggest if I should replace Db Identifier with snapshot identifier to search for snapshots. I would recommend using Db Identifier as it would help us to find all the manual snapshot of a database.

[robe070]

I get your point but thats not what this kind of snapshot is about. Its a singular snapshot. THE one to use. It does not change. Not for a particular version, unless there is a defect found by the regression test. Then the snapshot needs to be overwritten. There should only be one.

[robe070]

Also note that you have put tabs into the source code. This mucks up the formatting when the editor uses different tab settings. Some of the Write-Host lines you put in last. They need to be spaces.

[MohitArora001]

I have updated the code to use Snapshot Identifier instead of DB Identifier. Also removed the tab spaces.