search

tzmfreedom / rustforce

Salesforce REST API Client by Rust
MIT License
41 stars 19 forks source link

Add pagination for SOQL queries #31

Closed djrodgerspryor closed 2 days ago

djrodgerspryor commented 1 year ago

Problem

There is an underlying response size limit for SOQL query responses; salesforce will return truncated results and return a nextRecordsUrl on the response if that limit is reached. Previously, handling this case wasn't possible since rustforce didn't capture or expose the nextRecordsUrl.

Solution

Now, it captures the URL and automatically consumes the remaining pages of results so that callers don't need to think about it.

I've suggested in a comment that the existing done field should be removed from QueryResponse since it's now always true, but I haven't done that because it would break compatibility for existing users: maybe that should be saved for a 1.0 release?

Future Work

If there was demand for it, we could also expose the underlying get-a-single-page-of-results function, but I suspect a streaming API would be more useful as a next step beyond this.

Misc

I also fixed up the various clippy warnings while in the area.

tzmfreedom commented 2 days ago

@djrodgerspryor Thank you for your PullRequest ! Sorry for late reply and I've closed your PR by mistake.

I think that querying result all at once is memory heavy workload. So iterable querying(paging) is better than it.

I'll think this approach and implementation.