gflewis / sndml3

ServiceNow Data Mart Loader: an application to load SQL databases from ServiceNow
MIT License
30 stars 19 forks source link

Action sync does not respect pagesize #41

Closed ecg0030 closed 7 months ago

ecg0030 commented 2 years ago

I'm using this config:

tables:
- source: table_name
  action: sync
  partition: hour
  pagesize: 50

However, I see this request being issued with a 10k limit:

20:09:43 WARN JsonRequest [main] table_name RESPONSE: GET https://<servicnow-url>/api/now/table/<table_name>?sysp
arm_limit=10000&sysparm_exclude_reference_link=true&sysparm_display_value=false&sysparm_fields=sys_id%2Csys_created_on%2Csys_updated
_on&sysparm_query=.....

I see in Synchronizer.java that this might be hardcoded and not respecting the pagesize option:

https://github.com/gflewis/sndml3/blob/8b38980d1fac5dcd0f8aef2a5d413f38c5092a38/src/main/java/sndml/datamart/Synchronizer.java#L98

I did not see this same issue when using since: last

ecg0030 commented 2 years ago

This appears to be resolved with the following pull request https://github.com/gflewis/sndml3/pull/42