[source-clickup] Pagination doesn't work

[makz81]

Connector Name

source-clickup-api

Connector Version

0.1.1

What step the error happened?

During the sync

Relevant information

The pagination in the task stream is not working. I can only retrieve 100 tasks through the stream, even though more are available.

The pagination in the task stream is configured in the clickup_api.yaml as follows:

paginator:
  type: DefaultPaginator
  page_size_option:
    inject_into: "request_parameter"
    field_name: "page_size"
  pagination_strategy:
    type: "OffsetIncrement"
    page_size: 1
  page_token_option:
    field_name: "page"
    inject_into: "request_parameter"

Without having the complete overview of the pagination configuration possibilities in the config.yaml, field_name: "page_size" seems to be incorrect, as according to the ClickUp API documentation, the field name should be "page".

Relevant log output

2024-03-26 21:00:44 platform > sync summary: {
  "status" : "completed",
  "recordsSynced" : 0,
  "bytesSynced" : 0,
  "startTime" : 1711486839771,
  "endTime" : 1711486844173,
  "totalStats" : {
    "bytesCommitted" : 96820,
    "bytesEmitted" : 96820,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 1711486844172,
    "destinationWriteStartTime" : 1711486839771,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 100,
    "recordsCommitted" : 100,
    "replicationEndTime" : 1711486844173,
    "replicationStartTime" : 1711486839771,
    "sourceReadEndTime" : 1711486843431,
    "sourceReadStartTime" : 1711486839771,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ {
    "streamName" : "task",
    "stats" : {
      "bytesCommitted" : 96820,
      "bytesEmitted" : 96820,
      "recordsEmitted" : 100,
      "recordsCommitted" : 100
    }
  } ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 2261425,
      "executionCount" : 100,
      "avgExecTimeInNanos" : 22614.25
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 3633597650,
      "executionCount" : 407,
      "avgExecTimeInNanos" : 8927758.353808355
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 1234341,
      "executionCount" : 100,
      "avgExecTimeInNanos" : 12343.41
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 4375556286,
      "executionCount" : 863,
      "avgExecTimeInNanos" : 5070169.508690614
    }
  }
}

Contribute

• [X] Yes, I want to contribute

[marcosmarxm]

Thanks for reporting the issue @makz81 ClickUp is a community connector and it isn't in the current roadmap for improvements. If you want to contribute fixing the issue please reach me out in Slack so I can provide you instructions to make the contribution 🎖️