projectsExportV1

[sgokaram-saagie]

Spec - https://github.com/saagie/gradle-saagie-dataops-plugin/wiki/projectsExportV1

This task is used to migrate the jobs, pipelines from V1 into V2.

Logic

1. Using the task artifacts connect to Saagie tenant using Rest interfaces ( V1 is only accessible via Rest).
2. Download the artifacts
3. Using the attached document of property mappings maps each field from V1 to V2 and then store the file in the same JSON format as that of projectExport format.
4. Validate by Use the export file created and import into V2 via projectImport task.

Approach

1. Mappings - no streaming // no support. It will be, btu right now just don't map the value. docker with longrunning : if an API or UI shown ==> Apps docker with long running : if a processing ==> Jobs version missing : defualt value to setup.

2. Scheduling : Create a generic reusable module to map the scheduling from V1 to V2.

Gradle V1 to V2 Mappings - Sqoop.pdf

[medamineziraoui]

How to test :

Version to test : 2.0.22

The Export phase:

Create a build file: Example of build.projectsExportV1.gradle:

plugins {
  id "io.saagie.gradle-saagie-dataops-plugin" version "2.0.22"
}
saagie {
    server {
        url = "REPLACE_SAAGIEURL"
        login = "REPLACE_SAAGIELOGIN"
    password = "REPLACE_SAAGIEPASSWORD"
        environment = "REPLACE_SAAGIE"
    jwt = true
    }
    job {
          ids =  [ "21754" ]
        }
    exportArtifacts {
          export_file = "./job/export.zip" // <=== url "./job/" should exist  
          overwrite=true
      temporary_directory='./tmp'    // <=== temporary_directory ./tmp'  should exist        
    }
}

Export job using this commande : The command line : gradle -b build.projectsExportV1.gradle projectsExportV1

The import phase:

Create a new build file: Example : build.projectsImport.gradle:

plugins {
    id 'groovy'
    id 'io.saagie.gradle-saagie-dataops-plugin' version '2.0.22'
}

saagie {
    server {
        url = "REPLACE_SAAGIEURL"
        login = "REPLACE_SAAGIELOGIN"
    password = "REPLACE_SAAGIEPASSWORD"
        environment = "REPLACE_SAAGIE"
    jwt = true
        acceptSelfSigned = true
    }
    project {
        id = "REPLACE_SAAGIE_PROJECT_ID"
    }
    importArtifacts {
        import_file = "./job/export.zip"
        temporary_directory='./tmp'
    }
}

Then use this command line: The command line : gradle -b build.projectsImport.gradle projectsImport Bug behvior You should get

Task :projectsImport FAILED
Something went wrong when creating project job: {"errors":[{"message":"Job not valid","extensions":{"commandLine":"not allowed","package":"not allowed","classification":"ValidationError"}}],"data":null}

Expected behvior

> Task :projectsImport
{status=success, job=[{id=21754, name=RetrieveCustomerData}], pipeline=[]}

BUILD SUCCESSFUL in 5s
1 actionable task: 1 executed

[sgokaram-saagie]

This is the main story or epic and will be closed after 2.1 is complete