Create Jenkins pipeline jobs for all DB types

[harshanL]

Description:

Currently, there are build jobs only for MySQL database. We need to add MariaDB, PostgreSQL, SQLServer and Oracle database build jobs to the pipeline stages.

[kasunbg]

Shall we have a scripted pipeline step where we can iterate and call run-testplan of all the test-plans generated by generate-testplans command?

[pasindujw]

+1

[kasunbg]

Cool.. We can may be add parallel step inside the for-loop.. but need to verify that.

[pasindujw]

Parallel step is tested, and works fine (See link ). But we can not add parallel steps ATM since AWS VPC limits may cause problems.

[kasunbg]

Parallel step is tested, and works fine (See link https://testgrid-live-dev.private.wso2.com/blue/organizations/jenkins/wso2is-5.4.1-LTS/detail/wso2is-5.4.1-LTS/154/pipeline But we can not add parallel steps ATM since AWS VPC limits may cause problems.

I see.. Can we control the parallelism? For example, can we request jenkins pipeline to only have three/four concurrent builds? Basically, we want to run these tasks in a size-controlled thread-pool.

Do you have a sample Jenkinsfile where u run parallel tasks?

[pasindujw]

Yes we can control the amount of concurrent builds. Yes. I just used something like below for testing (but we need to re-structure it to add to testgrid flow :) )

steps {
    script {
    parallel ( 
        "stream1" : {
                        try {
                            echo 'Running Test Plan for infra-combination 1'
                            sh "java -version"

                            unstash name: "${JOB_CONFIG_YAML}"

                            sh """
                            cd ${TESTGRID_HOME}/testgrid-dist/${TESTGRID_NAME}
                            ./testgrid run-testplan \
                                --product ${PRODUCT} \
                                --file "${TESTGRID_HOME}/jobs/${PRODUCT}/test-plans/test-plan-01.yaml"
                            """
                            } catch (Exception err) {
                                echo "Error : ${err}"
                                currentBuild.result = 'FAILURE'
                            }
        },
        "stream2" : {

                        try {
                            echo 'Running Test Plan for infra-combination 2'
                            sh "java -version"

                            unstash name: "${JOB_CONFIG_YAML}"

                            sh """
                            cd ${TESTGRID_HOME}/testgrid-dist/${TESTGRID_NAME}
                            ./testgrid run-testplan \
                                --product ${PRODUCT} \
                                --file "${TESTGRID_HOME}/jobs/${PRODUCT}/test-plans/test-plan-02.yaml"
                            """
                            } catch (Exception err) {
                                echo "Error : ${err}"
                                currentBuild.result = 'FAILURE'
                            }

        }
        )
    }
    echo "RESULT: ${currentBuild.result}"
}

[pasindujw]

We can even control and name nodes for each concurrent build as well. Haven't tested though! :) So each build can run parallel in different nodes (machines).

[wso2-jenkins-bot]

Cool.. this is of course possible. :) I was wondering how we can get the same working when we don't really know the total number of test-plans.. Here, we have two static (stream1, stream2) parallel steps as I see.

[pasindujw]

@kasunbg check this out! Its dynamic steps in parallel and working! :D

[kasunbg]

whoa.. that's really nice!!

[kasunbg]

For reference, this is the jenkine pipeline stage where the parallel steps are executed where the # of steps is dynamic.

        stage('parallel-run') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/test-plans/*.yaml')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       

[pasindujw]

Final flow including TestGrid test-plan execution code in each parallel step is --> here

Code ATM

stage('TestGrid Test Execution') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/test-plans/*.yaml')) {
                        tests["${f}"] = {
                            node {
                                stage("Combination of ${f}") {
                                        script {
                                                try {
                                                    echo 'Running Test Plan for infra-combination ${f}'
                                                    sh "java -version"

                                                    unstash name: "${JOB_CONFIG_YAML}"

                                                    sh """
                                                    cd ${TESTGRID_HOME}/testgrid-dist/${TESTGRID_NAME}
                                                    ./testgrid run-testplan \
                                                        --product ${PRODUCT} \
                                                        --file "${TESTGRID_HOME}/jobs/${PRODUCT}/${f}"
                                                    """
                                                } catch (Exception err) {
                                                    echo "Error : ${err}"
                                                    currentBuild.result = 'FAILURE'
                                                }
                                            }
                                        echo "RESULT: ${currentBuild.result}"
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       

[kasunbg]

Cool.. We may need to bring the 'post' processing tasks into the above stage now?

And, what's your suggestion to address the VPC limit? Is there a way to limit the parallelism via a thread pool or similar?

[harshanL]

I think we need a resource tracking and allocation mechanism inside the TestGrid. There can be other resources which also have certain limitations. If we do that, we can increase the level of parallelism.

[kasunbg]

Yea.. shall we raise a github ticket and a mail thread on that? At the same time, we need a solution to limit the parallelism.

[pasindujw]

+1 for having a resource tracking in TestGrid side.. and once we have that, TestGrid is aware of live stats of the available, reserved resources like VPC and therefore I think we can handle parallelism from TestGrid side in efficient way, instead of letting jenkins do.