opendatamesh-initiative / odm-specification-dpdescriptor

The Data Product Descriptor Specification (DPDS) Repository
https://dpds.opendatamesh.org/
Apache License 2.0
65 stars 3 forks source link

Add `lifecycleInfo` configuration block to `InternalComponents` Object #26

Closed andrea-gioia closed 2 months ago

andrea-gioia commented 10 months ago

In order to manage the dp lifecycle we want to add lifecycleInfo configuration block to InternalComponents object and remove the previously defined ProvisionInfo, BuildInfo and DeployInfo objects. This is how the new configuration block should look like:

"internalComponents": {
    "lifecycleInfo": {
      "test": [{
        "x-prop":"x-prop-value", 
        "service": {
          "$href": "{azure-devops}"
        },
        "template": {
          "name": "testPipeline",
          "version": "1.0.0",
          "specification": "azure-devops",
          "specificationVersion": "1.0.0",
          "definition": {
            "organization": "andreagioia",
            "project": "opendatamesh",
            "pipelineId": "3",
            "branch": "main"
          }
        },
        "configurations": {
          "stagesToSkip": ["Deploy"]
        }
      }],
      "prod": [{
        "x-prop":"x-prop-value", 
        "service": {
          "$href": "{azure-devops}"
        },
        "template": {
          "name": "testPipeline",
          "version": "1.0.0",
          "specification": "azure-devops",
          "specificationVersion": "1.0.0",
          "definition": {
            "organization": "andreagioia",
            "project": "opendatamesh",
            "pipelineId": "3",
            "branch": "main"
          }
        },
        "configurations": {
          "stagesToSkip": []
        }
      }],
      "deprecated": [{ }]
    }
}
dorvak commented 7 months ago

Is the intention to completely remove ProvisionInfo etc. from the Specification and if yes, what's the reasoning behind this? I think Provision, Build and Deploy are very generic, well defined concepts, whereas "lifecycle" (Maybe "Environment" is a better term?) is somewhat arbitrary and can be treated as internals of the respective Build & Deployment-Pipelines?

Another note/question: How you handle environments in the Ports? Do you create multiple ports (one for Test, Integration and Prod-Envs for example), or even dedicated DPSC and separate them per env?

andrea-gioia commented 4 months ago

Sorry for the delay in my response.

The lifecycle block combines the Provision, Build, and Deploy blocks with the concept of an environment. However, these blocks don't currently allow you to specify the environment for an activity. You could do this using parameters in the configuration object, but the descriptor wouldn't contain a list of valid environments for a data product at the metadata level.

We haven't explicitly used the term "environment" because a stage in the lifecycle can be an environment, but it doesn't have to be. For example, in your case, the stages could be build, provision, and deploy, which would bring you back to the current situation.

I hope this is helpful! Let me know if you have any other questions.