Consolidate working directories of Jenkins and Testgrid

[kasunbg]

Description: The working directory need to be within the testgrid_home under its own directory. We came up with a build directory for each infrastructure combination. Everything need to happen in there.

Right now, things are scattered all over. Some are in Jenkins workspace, some are directly inside the testgrid_home//, and some are in the build workspace of the given infrastructure combination (test-run.log).

Steps to reproduce: Run generate-test-plan and study the behavior.

Related Issues: N/A

[pasindujw]

The initial structure after the consolidation will look as below example:

(Directory: <TESTGRID_HOME>/wso2is-5.4.1-LTS)
├── aws-cloudformation-templates
│   ├── cloudformation-templates
│   │   ├── pattern-1
│   │   │   ├── pattern-1-with-puppet-cloudformation.template.yml
│   │   │   └── wso2-is-with-alb-cf-template-parameters.json
│   │   └── pattern-2
│   │       └── pattern-2-with-puppet-cloudformation.template.yml
│   ├── infrastructure-build-plans
│   │   └── single-node-infra.yaml
│   ├── product-configs
│   │   
│   └── scripts
│   
├── identity-test-integration
│   ├── scenario01
│   ├── scenario02
│   ├── scenario03
│   ├── 
│   ├── 
│
├── job-config.yaml
├── testgrid
├── test-plans
│   ├── test-plan-01.yaml
│   └── test-plan-02.yaml
└── WSO2-TestGrid
    ├── lib
    ├── log4j2.xml
    ├── testgrid
    └── web-app

[kasunbg]

+1 for the overall structure. Few suggestions:

• Change the Jobs directory to: <TESTGRID_HOME>/jobs/<job-name>
• Move Testgrid distribution out of testgrid-home. Say, to /testgrid/testgrid-dist
• Have a separate workspace directory to keep build related files: say: <TESTGRID_HOME>/jobs/<job-name>/workspace/

[pasindujw]

The structure is updated to the following:

<TESTGRID_HOME>
│ 
│   
├── testgrid-dist
│     ├── WSO2-TestGrid.zip
│     └── WSO2-TestGrid
│
└── jobs
      ├── wso2is-5.4.1-LTS
      │       │              
      │       ├── workspace      
      │       │       ├── aws-cloudformation-templates
      │       │       ├── identity-test-integration
      │       │       └── <DEPLOYMENT_REPOSITORY>
      │       │
      │       ├── job-config.yaml
      │       │ 
      │       └── test-plans
      │           │
      │           ├── test-plan-01.yaml
      │           ├── test-plan-02.yaml
      │           ├── test-plan-03.yaml
      │
      ├── <PRODUCT_NAME> (= <JOB_NAME>)
      │       │
          ├─     

We can close this if not further modifications needed :) @kasunbg

[kasunbg]

+1. We need to think about the location of test-plans as well since it is a generated one. But, we can come back to that later.