[ Code integration in progress ] Add option version_dedup to Job and Pipeline to filter redundant versions from export

[medamineziraoui]

Why ?

We will add an option version_dedup to job and pipeline so it will allow us to control if we will get unique version from versions or keep all the versions with redundant data

Links / Ressources

https://github.com/saagie/gradle-saagie-dataops-plugin/issues/256

Actual status / Investigations

if we set include_all_versions=true to pipeline or to job and use projectsExportV1 it will export the job/pipeline concerned and export all versions found in the v1. but it will only export different versions automaticaly,we want to control that

How?

• [ ] Add property version_dedup to job with false default value
• [ ] Add property version_dedup to pipeline with false default value
• [ ] wrapp exportPipeline.versions.unique() inside the code with a condition on if version_dedup = true fromgetPipelineAndPipelineVersionDetailToExportV1
• [ ] wrapp exportJob.versions.unique() inside the code with a condition on if version_dedup = true fromgetJobAndJobVersionDetailToExportV1
• [ ] A case that wasn't took care of in current release, is that it will compare files by name and not by size(). but then if 2 versions have the same name but diffent contains then our tasks will consider it the same. so we need the adress that.
• [ ] I will add a new http method similar to the one from this stackoverflow post https://stackoverflow.com/questions/12800588/how-to-calculate-a-file-size-from-url-in-java
• [ ] the previous method will allow us to get size of files without getting the binary so i will perform well
• [ ] I will add new property fileSize to the JobVersionDTO class
• [ ] I will affect the size of each artifact inside fileSize
• [ ] I will update the compareTo method of JobVersionDTO from

  !o.packageInfo.equals(packageInfo) ? 1 : 0

  To

  !o.packageInfo.equals(packageInfo) ? 1 : 
                              !o.size.equals(size)? 1 : 0

• [ ] Fix the bug that happens when removing current remove duplication logic. ( one of the version is missing )
• [ ] Fix the tests that will fails becauses of the changes