search

NewPath-Consulting / warm

Pro Reports for Wild Apricot is a Google Data Studio connector that helps Wild Apricot administrators understand your organization’s membership engagement, renewal rates, and demographics. Trend your revenue and get expense analyses. Whatever your question, enjoy all the organized data you need at your fingertips to make the business decisions want.
https://newpathconsulting.com/warm
GNU General Public License v3.0
5 stars 4 forks source link

Optimize data gathering algorithm for more than 2,000 contacts #17

Closed asirota closed 4 years ago

asirota commented 4 years ago

Need to add paging ($top, $skip) support for larger than 2,000 contacts in the /Contacts end point.

There should be a configuration setting to allow end user to configure the # of rows to page through (could be a way to optimize performance by limiting the numbers of row to page through). Currently the connector breaks at more than 5,000 contacts.

This is a parameter for the API call for /contacts - it allows you grab a few records at a time (I think 1,000 would be ok but we should experiment)

$skip integer (query) 'Specifies the number of records to skip (not include in a result set). For example if you expect about 300 records in result set and want to get them in small portions, you could make do so with 3 calls:'

...&$top=100 - will return records from 0 to 99 - ...?$skip=100&top=100' - will return records from 100 to 199 - ...?$skip=200' - will return records from 199 to the end

$top integer (query) this parameter specifies the maximum number of entries to be returned

edtsoi430 commented 4 years ago

Paging support functionality added as a parameter in the configBuilder. Right now you can select either the top 500, 1000, 2000, 3000, 4000 or 4500 records to be retrieved.

asirota commented 4 years ago

The Select the top number of records you would like to retrieve should only show for Contacts and Contacts custom fields (it's not applicable to other endpoints).

The function label should be reworded as

Select the maximum number of records to retrieve per API call.

Provide the following options and set 500 as default:

100 500 1000 1500 2000 2500 3000 3500 4000

The page size should be set to this value and should stop retrieving records when there are no more records left. Remember you can use $count to count the total number of records to retrieve given the other 2 checkboxes (isArchive and Membership Enabled).

edtsoi430 commented 4 years ago

Resolved. Paging Mechanics is changed to desired behavior, with options as listed above (100, 500, 1000, 1500 etc.).