search

saagie / gradle-saagie-dataops-plugin

Saagie Gradle Plugin for the new version (2.0) of Saagie Dataops Orchestrator
Apache License 2.0
2 stars 0 forks source link

Import Pipeline and Job - Inconsistencies and using wrong configuraitons #325

Open sgokaram-saagie opened 3 years ago

sgokaram-saagie commented 3 years ago

When running a integration test suite noticed that the ImportJob was using a wrong configuration and importing old artifacts.

See below output refer to importpython , instead of importing a single python job its importing pipeline. This is a similar issue to caching issue we faced before.

saagies-mbp:Gradle shivakumar$ gradle projectTestAll

Task :projectExportDocker {"status":"success","exportfile":"./job/docker.zip"}

Task :projectImportDocker {status=success, job=[{id=22712, name=jupyter/scipy-notebook}], pipeline=[], variable=[]}

Task :projectExportJava {"status":"success","exportfile":"./job/java.zip"}

Task :projectImportJava {status=success, job=[{id=21754, name=RetrieveCustomerData}], pipeline=[], variable=[]}

Task :projectExportPipelineWithJobandVersion {"status":"success","exportfile":"./pipeline/pipelinejobver.zip"}

Task :projectImportPipelineWithJobandVersion {status=success, job=[{id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=21754, name=RetrieveCustomerData}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=10}], variable=[]}

Task :projectExportPython {"status":"success","exportfile":"./job/python.zip"}

Task :projectImportPython {status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportR {"status":"success","exportfile":"./job/r.zip"}

Task :projectImportR {status=success, job=[{id=20983, name=Demo Projects 2}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportSpark {"status":"success","exportfile":"./job/spark.zip"}

Task :projectImportSpark {status=success, job=[{id=2777, name=Build new Model, versions=2}, {id=2779, name=ERP Data Cleansing, versions=10}, {id=2778, name=Import ERP Data, versions=4}, {id=9726, name=Build model (sales / ERP), versions=3}, {id=4708, name=pm Extraction batch PLCs, versions=7}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportSqoop {"status":"success","exportfile":"./job/sqoop.zip"}

Task :projectImportSqoop {status=success, job=[{id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}, {id=22186, name=test_koalas}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportTalend {"status":"success","exportfile":"./job/talend.zip"}

Task :projectImportTalend {status=success, job=[{id=21439, name=talend job}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportV1Var FAILED

FAILURE: Build failed with an exception.

BUILD FAILED in 7m 19s 17 actionable tasks: 17 executed

medamineziraoui commented 3 years ago

HOW TO TEST WIP:

The release : 2.1.11

import io.saagie.plugin.dataops.tasks.projects.artifact.ProjectsExportJobV1Task
import io.saagie.plugin.dataops.tasks.projects.artifact.ProjectsImportJobTask
import groovy.json.JsonSlurper;
plugins {
    id 'groovy'
    id 'base'
    id 'io.saagie.gradle-saagie-dataops-plugin' version '2.1.11'
}

dependencies {
    implementation group: 'gradle.plugin.io.saagie', name: 'gradle-saagie-dataops-plugin', version: '2.1.11'
}

def exportLocation = '/test_gradle/' // <=== edit this with the export path
def tmpLocation = './tmp' // <=== edit this with the tmp file path 

task projectExportSqoop(type: ProjectsExportJobV1Task) { // <====== Create the first job
    configuration = saagie {}
    doFirst {
        projectExportSqoop.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['22186']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/sqoop.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportSqoop'
    group = 'Saagie'
    description= 'export v1 sqoop job for saagie'
}

task projectExportSpark(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportSpark.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['4708']
                include_all_versions=true
            }
            exportArtifacts {
                export_file = exportLocation + '/job/spark.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportSpark'
    group = 'Saagie'
    description= 'export v1 spark job for saagie'
}

task projectExportJava(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportJava.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['21754']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/java.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportJava'
    group = 'Saagie'
    description= 'export v1 Java job for saagie'
}

task projectExportR(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportR.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['20983']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/r.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportR'
    group = 'Saagie'
    description= 'export v1 R job for saagie'
}

task projectExportTalend(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportTalend.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['21439']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/talend.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportTalend'
    group = 'Saagie'
    description= 'export v1 talend job for saagie'
}
task projectExportPython(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportPython.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['22702']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/python.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportPython'
    group = 'Saagie'
    description= 'export v1 python job for saagie'
}
task projectExportDocker(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportDocker.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            job {
                ids = ['22712']
            }
            exportArtifacts {
                export_file = exportLocation + '/job/docker.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportDocker'
    group = 'Saagie'
    description= 'export v1 Docker job for saagie'
}
task projectExportPipelineWithJobandVersion(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportPipelineWithJobandVersion.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            pipeline {
                ids = ['484']
                include_job=true
                include_all_versions=true
            }
            exportArtifacts {
                export_file = exportLocation + '/pipeline/pipelinejobver.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportPipelineWithJobandVersion'
    group = 'Saagie'
    description= 'export Pipleline with job and version dependencies'
}
task projectExportPipeline(type: ProjectsExportJobV1Task) { // <====== Create the second job
    configuration = saagie {}
    doFirst {
        projectExportPipeline.configuration = saagie {
            newTaskContext {}
            server {
                url = saagieurl
                login = saagieuserid
                password = saagiepassword
                environment = saagieplatformid
                jwt = true
                acceptSelfSigned = true
            }
            pipeline {
                ids = ['36']
                include_job=false
                include_all_versions=false
            }
            exportArtifacts {
                export_file = exportLocation + '/pipeline/pipeline.zip'
                overwrite=true
                temporary_directory= tmpLocation
            }
        }
    }
    taskName = 'projectExportPipeline'
    group = 'Saagie'
    description= 'export Pipleline with out dependency'
}
task projectImportSqoop( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportSqoop'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportSqoop.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/sqoop.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import sqoop job for saagie'
    taskName = 'projectImportSqoop'
}
task projectImportSpark( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportSpark'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportSpark.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/spark.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import sqoop job for saagie'
    taskName = 'projectImportSpark'
}
task projectImportJava( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportJava'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportJava.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/java.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import java job for saagie'
    taskName = 'projectImportJava'
}
task projectImportR( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportR'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportR.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/r.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import R job for saagie'
    taskName = 'projectImportR'
}
task projectImportTalend( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportTalend'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportTalend.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/talend.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import Talend job for saagie'
    taskName = 'projectImportTalend'
}
task projectImportPython( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportPython'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportPython.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/python.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import Python job for saagie'
    taskName = 'projectImportPython'
}
task projectImportDocker( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportDocker'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportDocker.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/job/docker.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'import Docker job for saagie'
    taskName = 'projectImportDocker'
}
task projectImportPipelineWithJobandVersion( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportPipelineWithJobandVersion'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportPipelineWithJobandVersion.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/pipeline/pipelinejobver.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'Import Pipeline with Dependency Job and all versions'
    taskName = 'projectImportPipelineWithJobandVersion'
}
task projectImportPipeline( // <======== Import artifacts, see that we use the gradle dependOn
        type: ProjectsImportJobTask) {
    dependsOn(['projectExportPipeline'])
    configuration = saagie {
        server {
            url = saagieurl
            login = saagieuserid
            password = saagiepassword
            environment = saagieplatformid
            jwt = true
        }
    }
    doFirst {
        projectImportPipeline.configuration = saagie {
            newTaskContext {}
            project {
                id = saagieprojectid
            }
            importArtifacts {
                import_file = exportLocation + '/pipeline/pipeline.zip'
                temporary_directory= tmpLocation
            }
        }
    }
    description= 'Import Pipeline with Dependency Job and all versions'
    taskName = 'projectImportPipeline'
}
task projectTestAll( // <======== Import artifacts, see that we use the gradle dependOn
        type:Exec) {
    dependsOn(['projectImportSqoop','projectImportSpark','projectImportJava','projectImportR','projectImportTalend','projectImportPython','projectImportDocker', 'projectImportPipeline', 'projectImportPipelineWithJobandVersion' ])
    commandLine 'echo','Testing'
}

If you got this response then It worked fine

❯ gradle -b build.tasks.gradle projectTestAll

> Task :projectExportDocker
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/docker.zip"}

> Task :projectImportDocker
{status=success, job=[{id=22712, name=jupyter/scipy-notebook}], pipeline=[], variable=[]}

> Task :projectExportJava
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/java.zip"}

> Task :projectImportJava
{status=success, job=[{id=21754, name=RetrieveCustomerData}], pipeline=[], variable=[]}

> Task :projectExportPipeline
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/pipeline/pipeline.zip"}

> Task :projectImportPipeline
{status=success, job=[], pipeline=[{id=36, name=Wine Recognition}], variable=[]}

> Task :projectExportPipelineWithJobandVersion
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/pipeline/pipelinejobver.zip"}

> Task :projectImportPipelineWithJobandVersion
{status=success, job=[{id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}, {id=2777, name=Build new Model}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

> Task :projectExportPython
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/python.zip"}

> Task :projectImportPython
{status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}], pipeline=[], variable=[]}

> Task :projectExportR
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/r.zip"}

> Task :projectImportR
{status=success, job=[{id=20983, name=Demo Projects 2}], pipeline=[], variable=[]}

> Task :projectExportSpark
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/spark.zip"}

> Task :projectImportSpark
{status=success, job=[{id=4708, name=pm Extraction batch PLCs, versions=8}], pipeline=[], variable=[]}

> Task :projectExportSqoop
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/sqoop.zip"}

> Task :projectImportSqoop
{status=success, job=[{id=22186, name=test_koalas}], pipeline=[], variable=[]}

> Task :projectExportTalend
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/talend.zip"}

> Task :projectImportTalend
{status=success, job=[{id=21439, name=talend job}], pipeline=[], variable=[]}

> Task :projectTestAll
Testing

BUILD SUCCESSFUL in 2m 43s
19 actionable tasks: 19 executed

For every export / import tasks with corresponding task then the build script did succeed.

You then need to check in the saagie platform that the all the job / pipeline that is exported, and the versions for the job or the pipeline is added like it s shown in the job.json/pipeline.json for every job or pipeline in every exported artifact

Tested on version : 2.1.11. Worked fine

Bug Behavior : Python import with have multiple jobs and pipelines from previous tasks:

Task :projectImportPython
{status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}
medamineziraoui commented 3 years ago

Tested on version : 2.1.11 OK

image

image

ZouhairBear commented 3 years ago

Tested on version : 2.1.12 OK

image

image image