Working with a Large Number of Records and a Large Number of DLRS Rollups

[benchastain]

I've been working on a project recently for an org with something like 150,000 contact and account records and over 300,000 opportunity records. Additionally, they have a number of rollup needs for summarizing opportunity records on the account object. I've configured about 100 rollups with DLRS.

As you can imagine, the volume of records and rollups demands a great deal of processing. On import, the triggers had to be deactivated because it was causing a timeout issue (for obvious reasons).

I have all the rollups configured to calculate on a schedule, but in order to get initial values for all of these rollups I am manually clicking the "Calculate" button for each of these records.

Can you provide any tips or guidance for how to deal with such a large number of records?

I know that I'm probably really pushing DLRS to the limit here, and there's a good chance the answer is "DLRS wasn't designed for this," but I thought maybe there's something that I could do to customize the apex to change the batch size for quicker processing.

I don't imagine you'll have an answer for me by the time I've completed this, but perhaps it can save me some time in the future.

[rogeramitchell]

@benchastain I encountered a similar case, and as part of implementing DLRS within an org with larger data volumes, it does take a bit of time to set up the initial calculation for the rollups. Your manual "Calculate" is the approach that I used as well.

[benchastain]

@rogeramitchell Did you encounter any subsequent difficulties with the scheduled jobs?

[rogeramitchell]

@benchastain I have not, though there is relatively low transactions with respect to the data volumes (under 10% of records are recalculated nightly).

[afawcett]

A few thoughts...

• Make sure to review the schedule modes on the wiki, ensure you have your scheduled rollups set up correctly.
• For that amount you do want to use scheduled modes to avoid impacting performance. The tool will try to combine aggregate queries if possible, but if you have a lot of variation on your criteria this will not be as effect.
• Other thing is indexes..... these can be a little hard to spot, but basically make sure any fields your including in your criteria are reviewed for custom index requirements. https://developer.salesforce.com/blogs/engineering/2015/06/know-thy-salesforce-field-indexes-fast-reports-list-views-soql.html

[afawcett]

• You also want to review carefully if you can use the incremental scheduled mode rather than the full recalc schedule. If you have to use the later, have a think if you can scope the full recalc with the criteria, e.g. past x years records only?

[benchastain]

Thanks @afawcett. I will look further into indexing. There may be some value there.

I had configured the rollups in the incremental scheduled mode, rather than the full recalc schedule. My plan is to have them most of the fields recalculated nightly. Many of these rollups -- about 40 -- are for historic information (so total amount of opportunities 15, 16, 17 years ago, etc.). My thinking is that these are unlikely to need recalculation and I can probably schedule them to only recalculate once a week.

I think a fair number of the rollups should be able to be aggregated because they draw upon the same 4 or 5 fields. The criteria for many of my operations are very similar (I'm mainly just counting, summing, and taking the min/max of the amount and closedate fields).

[benchastain]

Hey @afawcett I've run in to a bit of a problem and I'm not sure from the documentation if I'm misunderstanding something or not.

If I am running the rollups in incremental scheduled mode, do I need to schedule each individual rollup to run using the "Schedule Calculate" button or will scheduling a job for the apex class "RollupJob" fire all rollups set to "Scheduled" mode.

I am asking because I attempted to go through and schedule each of my rollups individually but I have hit the limit of 100 scheduled apex jobs (and still have like 50 more rollups).

Should I only be using the "Schedule Calculate" button and programming a date/time for those jobs which I want to run at specific times? Will the remainder of the rollups be run by the RollupJob class even if they do not have their own scheduled apex class?

I am sorry if my question is unclear or if your documentation clearly answers this--I have been reviewing it and still don't exactly understand the intended design of the incremental calculate mode.

[afawcett]

"If I am running the rollups in incremental scheduled mode, do I need to schedule each individual rollup to run using the "Schedule Calculate" button or will scheduling a job for the apex class "RollupJob" fire all rollups set to "Scheduled" mode." - The latter.

Please review this wiki page, let me know if it still does not make sense.

https://github.com/afawcett/declarative-lookup-rollup-summaries/wiki/What-you-need-to-know-about-Scheduling-Rollups

[benchastain]

Thanks for clarifying. I'd read the wiki several times and it made sense to me initially but I started to overthink it--I was also looking at the batch apex jobs log to verify that it was running and it was seemed like the RollupJob wasn't being run with as many batches as I was expecting to see so I was doubting my understanding.

Sent from my iPhone

On Aug 20, 2017, at 2:45 PM, Andrew Fawcett notifications@github.com wrote:

"If I am running the rollups in incremental scheduled mode, do I need to schedule each individual rollup to run using the "Schedule Calculate" button or will scheduling a job for the apex class "RollupJob" fire all rollups set to "Scheduled" mode." - The latter.

Please review this wiki page, let me know if it still does not make sense.

https://github.com/afawcett/declarative-lookup-rollup-summaries/wiki/What-you-need-to-know-about-Scheduling-Rollups

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.