The general ledger stream uses the stream partition feature to iterate the list of modules. Theres a desire to avoid waiting until an entire module is completed before moving on to the next one. An approach could be to have a maximum number of records to paginate before pausing and starting on the next partition, ultimately looping until all partitions are completed. This would have 2 benefits:
the user gets the a variety of data while backfilling so they can start working with the data before the full sync completes (days in some cases)
the offset size decreases and eventually resets which could potentially have performance benefits depending on how the intacct backend implements pagination
The general ledger stream uses the stream partition feature to iterate the list of modules. Theres a desire to avoid waiting until an entire module is completed before moving on to the next one. An approach could be to have a maximum number of records to paginate before pausing and starting on the next partition, ultimately looping until all partitions are completed. This would have 2 benefits: