search

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.57k stars 516 forks source link

[Feature][CustomizePlugin] Data extraction 'rawDataParams' does not support wild cards #7571

Closed riskpeep closed 3 months ago

riskpeep commented 4 months ago

Search before asking

Use case

As a DevOps engineer, I would like to be able to easily configure data extraction into custom fields using the Customize plugin data extraction feature so I can add custom data fields.

Description

The Customize plugin supports the extraction of data from raw data using a data extraction feature. The current implementation allows configuration of data extraction using an array of transformation rules that look like this:

"transformationRules":[
          {
            "table":"issues", 
            "rawDataTable":"_raw_jira_api_issues", 
            "rawDataParams":"{\"ConnectionId\":1,\"BoardId\":8}", 
            "mapping":{
              "x_test":"fields.status.name" 
            }
          }
        ]

In a transformation rule the rawDataParams field is used as a filter on the source data to identify source data to pull from. Currently the rawDataParams field requires an exact match on the source data. This means that DevOps engineers must configure a transformation rule for EACH board from which data should be extracted. For even moderately complex devlake installations, this can mean 10s of transformation rules to capture the issues associated with multiple boards. This also means that if additional sources are configured, the customize plugin must also be configured to include the new source board id.

It would be much more desirable if the 'rawDataParams' supported wildcards or other mechanisms to allow one transformation rule to match on more than one board.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct