AutoIDM / tap-clickup

tap-clickup , singer compliant tap for pulling clickup data
MIT License
12 stars 19 forks source link

Feature to add a filtering options on the API call #145

Open Nevsksar opened 11 months ago

Nevsksar commented 11 months ago

Description

When dealing with enterprise workspaces in ClickUp, as ClickUp admins, we often find ourselves having access to multiple workspaces, lists, tasks, and other elements that are not needed in our pipeline. To save resources, processing time, and improve efficiency, it is beneficial for us to have the ability to filter specific elements that we want to fetch.

Noteworthy ClickUp API Bugs

During the development of this project, I encountered some inherent bugs in ClickUp's API. Primarily, these bugs prevented the passing of a single space or list as a parameter. However, a workaround was discovered: by passing the same ID twice, the API would function correctly, and there are not any duplicate values in the API response.

Issues

  1. The following changes have been implemented and work, but they are currently coded to only fetch data from the general settings in the meltano.yml file. Consequently, it is not possible to configure different pipelines for various environments because they are not being correctly read.

For example, when using the command meltano --environment=dev run tap-clickup ..., the configurations are not being applied as expected from the dev envv because it reads only from the common configurations.

Taking this meltano.yml sample, the code always reads "spaces_id" as '90100424497' even when we run it in prod (which in this case should be '90070255202'

version: 1
default_environment: prod
project_id: 8f483ca8-e13c-4abb-8d24-666666666
environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-clickup
        variant: autoidm
        pip_url: tap-clickup
        config:
          workspace_id: 30979640
          spaces_id: '90070255202'
          list_ids: ''
        select:
        - space.name
        - space.id
        - task.*
      loaders:...
plugins:
  extractors:
  - name: tap-clickup
    variant: autoidm
    pip_url: tap-clickup
    config:
      workspace_id: 30979640
      spaces_id: '90100424497'
      list_ids: ''
    select:
    - space.name
    - space.id
    - task.*
  loaders:...
  1. Could not figure out how to correctly add the new setting in the meltano.yml file. At first I was just hardcoding it to the file, then I found out about tap-clickup--autoidm.lock file and in this file I added the new configuration settings to be added when using meltano config tap-clickup -set. But I cannot figure out how to edit this into the projects code, i basically added these configs to the og file:
    . . .
    {
      "name": "stream_map_config",
      "kind": "object",
      "label": "Stream Map Config",
      "description": "User-defined config values to be used within map expressions."
    },
    {
      "name": "stream_maps",
      "kind": "object",
      "label": "Stream Maps",
      "description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
    },
        {
      "name": "workspace_id",
      "kind": "integer",
      "label": "Workspace Id",
      "description": "To limit from which workspace to fetch data pass it's id(https://dev-doc.clickup.com/333/d/h/ad-455845/c464293970a6906)."
    },
        {
      "name": "spaces_id",
      "kind": "string",
      "label": "Spaces Ids",
      "description": "To limit from which Spaces to fetch data from, pass their ids as a list string Example: spaceid1,spaceid2,ispaced3."
    },
        {
      "name": "list_ids",
      "kind": "string",
      "label": "Lists Ids",
      "description": "To limit from which Lists to fetch data from, pass their ids as a list string Example: listid1,listid2,listid3."
    }
    ]
    }