search

datacontract / datacontract-specification

The Data Contract Specification Repository
https://datacontract.com/
MIT License
278 stars 41 forks source link

Allow lists of quality specifications #12

Closed stefannegele closed 7 months ago

stefannegele commented 1 year ago

Users of the CLI would find it very helpful if various quality specifications were supported. 

quality:
  - basic:
      weight: 0.2
      type: SodaCL
      specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-basic.yaml"
  - premium1:
      weight: 0.4
      type: SodaCL
      specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-premium1.yaml"
  - premium2:
      weight: 0.4
      type: SodaCL
      specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-premium2.yaml"

I suggest we support both - individual specifications or a list.

See also: https://github.com/datacontract/cli/pull/3#discussion_r1358205380

jochenchrist commented 1 year ago

If we decide to support multiple quality definitions, I would prefer this structure to be consistent with the examples section:

quality:
  - name: basic
    type: SodaCL
    specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-basic.yaml"
  - name: premium1
    type: SodaCL
    specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-premium1.yaml"
  - name: premium2
    type: SodaCL
    specification: "$ref: https://github.com/data-engineering-helpers/data-contracts/blob/123b1310662557c469e091b646f64e47b938836f/datacontract.com/contracts/data-contract-flight-route-quality-premium2.yaml"
da115115 commented 1 year ago

If we decide to support multiple quality definitions, I would prefer this structure to be consistent with the examples section.

Sure, it makes a lot of sense!

jochenchrist commented 7 months ago

Closing this issue as won't fix, as this currently has no priority.