Reoptimize PoolOutputModule settings for multi-threading

[Dr15Jones]

The settings we used for PoolOutputModule for RECO and AOD were done for Run 1 software. Now with these modules being the major factor for lower throughput for multi-threading we should redo the study to factor

• time spent in the module (which affects multi-threaded scaling)
• memory used (which we have more of relative to Run 1)
• size on disk
• speed at which we can read the created file

[cmsbuild]

A new Issue was created by @Dr15Jones Chris Jones.

@davidlange6, @smuzaffar, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

@dan131riley could you look into this? This is our limiting factor for how many threads we can efficiently use.

[Dr15Jones]

@davidlange6 do you have a suggestion for what workflow we should use to stress test reconstruction for output performance?

[Dr15Jones]

@bbockelm I think it might be useful to redo Brian's measurement of the per branch performance to see if there is any problematic data format.

[Dr15Jones]

@dan131riley In light of the recent re-RECO measurements, this looks to be more important than I at first thought. I think what we need to do is measure: write timing, read time, and file size for

• different compression algorithms
• different compression setting
• different buffer sizes

we will need to pay close attention for how this would affect stall timing as the number of threads is increased.

[bbockelm]

If possible, I'd be interested in actually increasing the buffer sizes: that should make the compression ratio better (and increase the amount of stalling going on, unfortunately).

What "balance of resources" would we like to achieve here?

[davidlange6]

FYI- When last checked this didn't help the compressed size... But it's been a few years. Of course if bigger means more stalls maybe reducing can also be considered?

On Oct 7, 2016, at 12:36 AM, Brian Bockelman notifications@github.com<mailto:notifications@github.com> wrote:

If possible, I'd be interested in actually increasing the buffer sizes: that should make the compression ratio better (and increase the amount of stalling going on, unfortunately).

What "balance of resources" would we like to achieve here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/cms-sw/cmssw/issues/16035#issuecomment-252080850, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEzyw3Zm2CdVHHuOfKCi_f36_Vt32KhJks5qxVu9gaJpZM4KKByL.

[smuzaffar]

@Dr15Jones , @makortel have we every re-optimized the PoolOutputModule ?

[dan131riley]

This was only partially completed, and probably needs to be revisited anyway to account for the changes since the last round of measurements. Will promote it my (much neglected) todo list.

[smuzaffar]

any update on this issue?

[makortel]

Maybe Chris' studies in https://indico.cern.ch/event/1131806/#17-compression-algorithm-compa would be close-enough to close this issue?